Go PHP 5, Go!

[Larry Garfield]

This is a follow-up to the PHP 5 thread from a week or two ago. It looks like
some momentum is building. Ken Rickard, Robert Douglas, and I have been
talking with some of the Jooma folks, and have a working draft of the "core
statement and justification". That is, what the goal is and why it is.
Joomla's development team is discussing the matter and is leaning yes. Based
on the earlier thread here I am hoping that there isn't much objection to
Drupal participating in the "Go PHP5" effort. :-) So far Joomla is leaning
yes, CakePHP is interested, and I had a positive first response from Typo3.
Robert Douglas has volunteered himself to setup a web site for it.

I'm not sure how Dries wants to handle the question of Drupal's participation
(by vote, by consensus, or by fiat). Dries?

Anyway, here's the working statement. Consider this an official
recommendation that Drupal commit to participating in this effort.

------------------------------------
PHP 4 has served the web developer community for seven years now, and served
it well.  However, it also shows its age.  Most of PHP 4's shortcomings have
been addressed by PHP 5, released three years ago, but the transition from
PHP 4 to PHP 5 has been slow for a number of reasons.

PHP developers cannot leverage PHP 5's full potential without dropping support
for PHP 4, but PHP 4 is still installed on a majority of shared web hosts and
users would then be forced to switch to a different application.  Web hosts
cannot upgrade their servers to PHP 5 without making it impossible for their
users to run PHP 4-targeted web apps, and have no incentive to go to the
effort of testing and deploying PHP 5 while most web apps are still
compatible with PHP 4 and the PHP development team still provides maintenance
support for PHP 4.  The PHP development team, of course, can't drop
maintenance support for PHP 4 while most web hosts still run PHP 4.  

It is a dangerous cycle, and one that needs to be broken.  The open source PHP
developer community has decided that it is indeed now time to move forward,
together.  Therefore, the listed open source PHP projects have all agreed
that effective 5 February 2008, any new feature release will have a minimum
required PHP version no older than PHP 5.2.0.  It is our believe that this
will allow web hosts a reason to upgrade and the PHP development team the
ability to retire PHP 4 and focus efforts on PHP 5 and the forthcoming PHP 6,
all without penalizing any existing project for being "first out of the
gate".
------------------------------------

Notes:
- I chose the date because I figure that will be 7-8 months after we
officially announce this thing, which I believe should be sufficient time for
web hosts.  It also comes out to 5/2/2008 (using European convention), and I
just like inside references like that. :-)
- This does not preclude any project from moving before the deadline date, or
from supporting older versions for however long they wish to.  That's up to
each project.  
- PHP 5.2 is already the most widely installed version of PHP 5, based on the
latest published stats. I know at least two web hosts I work with that
either have jumped or are in the process of jumping from PHP 4 straight to
PHP 5.2. By the target date it will have been out for nearly a year and a
half. It also adds a number of new and useful core features (filter_input,
json, a stable PDO, etc.). It's a good version to target.

Thoughts? Comments? Support? Rotten tomatoes?
--
Larry Garfield AIM: LOLG42
***@garfieldtech.com ICQ: 6817012

"If nature has made any one thing less susceptible than all others of
exclusive property, it is the action of the thinking power called an idea,
which an individual may exclusively possess as long as he keeps it to
himself; but the moment it is divulged, it forces itself into the possession
of every one, and the receiver cannot dispossess himself of it." -- Thomas
Jefferson

[Robert Douglass]

I just wanted to rehash what Larry said (because saying it many
different ways best avoids confusion). We are preparing a public
statement of intent. The statement is to be supported by PHP open source
projects and it is to reflect their intentions. The entire statement is
that by the 5th of February, 2008, the software projects involved will
no longer see the necessity of writing software that avoids language
features and constructs that are available in PHP5 but not PHP4.

Dries has written very clearly that PHP adoption is a real source for
concern:
http://buytaert.net/php-is-dead-long-live-php

Many people on this list have also voiced frustration with PHP4 shackles.

The main goals of this public statement are to a) help set the direction
of development across many projects, and b) send a loud and clear
message to companies who offer web hosting services.


-Robert

[James Walker]

Post by Robert Douglass
I just wanted to rehash what Larry said (because saying it many
different ways best avoids confusion). We are preparing a public
statement of intent. The statement is to be supported by PHP open source
projects and it is to reflect their intentions. The entire statement is
that by the 5th of February, 2008, the software projects involved will
no longer see the necessity of writing software that avoids language
features and constructs that are available in PHP5 but not PHP4.
Dries has written very clearly that PHP adoption is a real source for
http://buytaert.net/php-is-dead-long-live-php
Many people on this list have also voiced frustration with PHP4 shackles.
The main goals of this public statement are to a) help set the direction
of development across many projects, and b) send a loud and clear
message to companies who offer web hosting services.

great initiative, guys! Rasmus kinda threw down the gauntlet at OSCMS
about how to get php5 adoption going. Nice to see the various
communities coming together and unifying on this (that was one of the
initial reasons for OSCMS after all).

Go Robert and Larry Go! +1 Rah rah! ;)

[Bill Fitzgerald]

Adding to the chorus here -- at OSCMS, Rasmus seemed to imply that he
would look kindly on initiatives that sped the adoption of PHP5 --

This is a great idea/effort, both in the way that it brings different
open source communities together and in how it will eventually result in
better functionality for end users.

+1 , rah, etc.

Cheers,

Bill

Post by James Walker
great initiative, guys! Rasmus kinda threw down the gauntlet at OSCMS
about how to get php5 adoption going. Nice to see the various
communities coming together and unifying on this (that was one of the
initial reasons for OSCMS after all).
Go Robert and Larry Go! +1 Rah rah! ;)

[Michelle Cox]

It is our believe that this will allow web hosts a reason to upgrade and
the

I don't know if you're looking for feedback as mundane as typos, but that
should be "belief". While I'm on the sentence, "allow web hosts" sounds kind
of funny. I think "give" would be a better word.

Michelle

[Larry Garfield]

Post by Michelle Cox

It is our believe that this will allow web hosts a reason to upgrade and
the

I don't know if you're looking for feedback as mundane as typos, but that
should be "belief". While I'm on the sentence, "allow web hosts" sounds
kind of funny. I think "give" would be a better word.
Michelle

Thanks for the catch. I wasn't sure which sounded less pushy toward hosts.
At least some of them, I think/hope, will be just as in favor of breaking the
deadlock as we are. We can tweak the exact wording as needed.
--
Larry Garfield AIM: LOLG42
***@garfieldtech.com ICQ: 6817012

"If nature has made any one thing less susceptible than all others of
exclusive property, it is the action of the thinking power called an idea,
which an individual may exclusively possess as long as he keeps it to
himself; but the moment it is divulged, it forces itself into the possession
of every one, and the receiver cannot dispossess himself of it." -- Thomas
Jefferson

[Sean Robertson]

Are there currently that many PHP4 apps that won't run on PHP5? I
thought Drupal already does. Why can't ISPs upgrade without breaking
older apps?

I am a huge fan of this idea overall, BTW. I think migrating apps to
PHP5 will start forcing ISPs to at least provide the option to upgrade
since they'll start losing customers if they don't. At my company, we
own our own servers so it'd be relatively easy for us to upgrade.

Post by Larry Garfield
This is a follow-up to the PHP 5 thread from a week or two ago. It looks like
some momentum is building. Ken Rickard, Robert Douglas, and I have been
talking with some of the Jooma folks, and have a working draft of the "core
statement and justification". That is, what the goal is and why it is.
Joomla's development team is discussing the matter and is leaning yes. Based
on the earlier thread here I am hoping that there isn't much objection to
Drupal participating in the "Go PHP5" effort. :-) So far Joomla is leaning
yes, CakePHP is interested, and I had a positive first response from Typo3.
Robert Douglas has volunteered himself to setup a web site for it.
I'm not sure how Dries wants to handle the question of Drupal's participation
(by vote, by consensus, or by fiat). Dries?
Anyway, here's the working statement. Consider this an official
recommendation that Drupal commit to participating in this effort.
------------------------------------
PHP 4 has served the web developer community for seven years now, and served
it well. However, it also shows its age. Most of PHP 4's shortcomings have
been addressed by PHP 5, released three years ago, but the transition from
PHP 4 to PHP 5 has been slow for a number of reasons.
PHP developers cannot leverage PHP 5's full potential without dropping support
for PHP 4, but PHP 4 is still installed on a majority of shared web hosts and
users would then be forced to switch to a different application. Web hosts
cannot upgrade their servers to PHP 5 without making it impossible for their
users to run PHP 4-targeted web apps, and have no incentive to go to the
effort of testing and deploying PHP 5 while most web apps are still
compatible with PHP 4 and the PHP development team still provides maintenance
support for PHP 4. The PHP development team, of course, can't drop
maintenance support for PHP 4 while most web hosts still run PHP 4.
It is a dangerous cycle, and one that needs to be broken. The open source PHP
developer community has decided that it is indeed now time to move forward,
together. Therefore, the listed open source PHP projects have all agreed
that effective 5 February 2008, any new feature release will have a minimum
required PHP version no older than PHP 5.2.0. It is our believe that this
will allow web hosts a reason to upgrade and the PHP development team the
ability to retire PHP 4 and focus efforts on PHP 5 and the forthcoming PHP 6,
all without penalizing any existing project for being "first out of the
gate".
------------------------------------
- I chose the date because I figure that will be 7-8 months after we
officially announce this thing, which I believe should be sufficient time for
web hosts. It also comes out to 5/2/2008 (using European convention), and I
just like inside references like that. :-)
- This does not preclude any project from moving before the deadline date, or
from supporting older versions for however long they wish to. That's up to
each project.
- PHP 5.2 is already the most widely installed version of PHP 5, based on the
latest published stats. I know at least two web hosts I work with that
either have jumped or are in the process of jumping from PHP 4 straight to
PHP 5.2. By the target date it will have been out for nearly a year and a
half. It also adds a number of new and useful core features (filter_input,
json, a stable PDO, etc.). It's a good version to target.
Thoughts? Comments? Support? Rotten tomatoes?

[Robert Douglass]

Post by Sean Robertson
Are there currently that many PHP4 apps that won't run on PHP5? I
thought Drupal already does.

That's not the problem. The problem is that PHP5 has new features which
we can't use.

Post by Sean Robertson
Why can't ISPs upgrade without breaking older apps?

They can, and we want them to. They're just lazy, afaik.

Post by Sean Robertson
I am a huge fan of this idea overall, BTW. I think migrating apps to
PHP5 will start forcing ISPs to at least provide the option to upgrade
since they'll start losing customers if they don't.

That's the point.

[Jonathan Lambert]

Post by Robert Douglass

Post by Sean Robertson
Why can't ISPs upgrade without breaking older apps?

They can, and we want them to. They're just lazy, afaik.

Indeed, if I might interject a point here to support this idea.

While it probably is lazyness to some degree, what I've found to be
true with the vast major of hosting companies is that they are
running off of the stock Control Panel applications, and many of them
are running on SW-Soft's products (Plesk and Virtuozzo). As a former
Gold customer (recovering, we're now using mostly XEN, VMWARE, and
some other higher end systems for everything), I know that the
support policy for SW-Soft with regards to PHP is the OS Support
Upgrade Policy (that's what they called it, I've never seen it
actually). They will provide support for only whatever the vendor
has released on the platform, including any extended applications
like PHP and MySQL.

For hosting companies with any kind of integrated system, or those in
need of second tier support from SW-Soft, doing an upgrade can
invalidate their support agreements (you're always riding the line on
this one if you need "advanced upgrades," aka something that's
relatively new) and break their setups. In fact, doing an upgrade
with SW-Soft's popular HSP Complete system of php or mysql will
almost certainly functionally break the entire control panel. In
addition, it's extremely difficult to do a full OS upgrade with many
contol panel solutions, as they're using a lot of custom rpms and the
like, so things tend to stay in place and get patched. For example,
with HSP Complete (again, just an example), it's not actually
possible to do an upgrade - you have to delete the customer virtuozzo
environment and re-create it since it tracks rpms granularly.

So, we're moving at the speed of RedHat. Actually, most times,
RedHat running through CentOS. And since so many of the upgrades are
actually attached to the OS Upgrades, and since it's difficult to do
an OS Upgrade, things tend to stagnate across the entire industry.

Occasionally some brave sysadmin will stand up and start releasing
packages. For big hosting customers with a lot of purchase power
(think GoDaddy, 1&1, BlueHost, etc), this is absolutely an option.
But for the long tail, obstacles remain. But most big companies will
only patch when they have to, and leave the support up to companies
like SW-Soft.

So, it's not as much the hosting companies that need convincing -
it's the vendors that write the Control Panels that have the real
authority - they're the ones that need convincing. And I don't think
they're planning to fix some of the upgrade os problems anytime soon
(just ask me the difference between an insert and overlay control
panel - my own terms - sometime).

I think it would be worth rallying around the flag on this one, and
you'll certainly get link juice from me about it, but I don't thin
it's a question of hosting company lazyness exclusively.

Post by Robert Douglass

Post by Sean Robertson
I am a huge fan of this idea overall, BTW. I think migrating apps
to PHP5 will start forcing ISPs to at least provide the option to
upgrade since they'll start losing customers if they don't.

That's the point.

Indeed.

Jonathan

[Larry Garfield]

Post by Jonathan Lambert
So, it's not as much the hosting companies that need convincing -
it's the vendors that write the Control Panels that have the real
authority - they're the ones that need convincing. And I don't think
they're planning to fix some of the upgrade os problems anytime soon
(just ask me the difference between an insert and overlay control
panel - my own terms - sometime).
I think it would be worth rallying around the flag on this one, and
you'll certainly get link juice from me about it, but I don't thin
it's a question of hosting company lazyness exclusively.

Post by Robert Douglass

Post by Sean Robertson
I am a huge fan of this idea overall, BTW. I think migrating apps
to PHP5 will start forcing ISPs to at least provide the option to
upgrade since they'll start losing customers if they don't.

That's the point.

Indeed.
Jonathan

If the knock-on result is that hosts put pressure on lazy Control Panel companies to get with the 21st century lest the Control Panel companies start losing business, I am perfectly happy with that outcome. :-) The stagnation needs to break somewhere.

--Larry Garfield

[Karoly Negyesi]

Post by Jonathan Lambert
So, we're moving at the speed of RedHat. Actually, most times,
RedHat running through CentOS. And since so many of the upgrades are
actually attached to the OS Upgrades, and since it's difficult to do
an OS Upgrade, things tend to stagnate across the entire industry.

The CentOS team is pleased to announce the availability of CentOS 5.0. Major changes in CentOS 5 compared to CentOS 4 include:

These updated software versions: Apache-2.2, php-5.1.6

... RHEL 5 has moved all the way up to PHP 5.1.6 from RHEL 4's mere PHP 4.3.9

We are getting there.

[Khalid Baheyeldin]

I think the whole idea here is to break that catch-22 situation.
If the largest PHP application say they will move off PHP4, then
the users/hosts/distros will each pressure/nudge the other entities
to move faster, rather than the chicken-egg we are in today.

So, +1 on this initiative. What is the worst case? Come February
and we see that things have not moved. We evaluate it them and see
what other projects did, and decide either to stay the course, or go
back to PHP4.

So, the risk is not high in such a move.

Unless we (all CMS's that is) try, we will never know if it can succeed.

[Konstantin Käfer]

Post by Khalid Baheyeldin
So, +1 on this initiative. What is the worst case? Come February
and we see that things have not moved. We evaluate it them and see
what other projects did, and decide either to stay the course, or go
back to PHP4.

I think this is a bad idea. If we go into this process with this
attitude, we won't honestly try to get providers to PHP 5 because
there is no need to for them – they will know that software projects
won't move to 5 when the majority of providers still run 4. When we
really make Drupal (and all the other projects) PHP 5 only starting
from this date, providers won't have a choice but to upgrade.

Konstantin Käfer – http://kkaefer.com/

[Khalid Baheyeldin]

Post by Konstantin Käfer

Post by Khalid Baheyeldin
So, +1 on this initiative. What is the worst case? Come February
and we see that things have not moved. We evaluate it them and see
what other projects did, and decide either to stay the course, or go
back to PHP4.

I think this is a bad idea. If we go into this process with this
attitude, we won't honestly try to get providers to PHP 5 because
there is no need to for them – they will know that software projects
won't move to 5 when the majority of providers still run 4. When we
really make Drupal (and all the other projects) PHP 5 only starting
from this date, providers won't have a choice but to upgrade.

We don't need to publicize that there is a possibility that we can go
back in our decision.

This is a reality check that leaves a fallback option open. If February
comes and nothing changed as far as distros and hosting companies
are concerned. What are we going to do then? Just keep forging ahead
as if PHP4 does not exist, and corner ourselves?

We can say that we reevaluate this in February and that is it. We don't
need to make any decisions now (apart from pushing hard for hosts
and distros to go PHP5).

[Konstantin Käfer]

Post by Khalid Baheyeldin
We don't need to publicize that there is a possibility that we can go
back in our decision.

We just said that. I think it is necessary to *force* all
participating projects to switch to PHP 5 only, in all circumstances.
If we just give in when nothing has changed in february 2008, we end
up in the same state as we are now.

Post by Khalid Baheyeldin
Just keep forging ahead as if PHP4 does not exist, and corner
ourselves?

If all participating projects (most major ones showed interest) use
PHP 5 from 2008 on, we are not cornering ourselves as Drupal's
competitors are also PHP5 only.

Post by Khalid Baheyeldin
We can say that we reevaluate this in February and that is it. We don't
need to make any decisions now (apart from pushing hard for hosts
and distros to go PHP5).

Again, if hosters don't have a reason (= major PHP software is PHP 5
only) to upgrade, they won't. I think it is necessary to decide *now*
to give everyone half a year's notice.


Konstantin Käfer – http://kkaefer.com/

[Chris Kennedy]

This discussion brings to mind Thomas Schelling's landmark game theory
book, "The Strategy of Conflict." What we have here is a game of
coordinated bargaining. Two related quotations from the first chapter:

"We have learned that a threat has to be credible to be efficacious, and
that its credibility may depend on the costs and risks associated with
fulfillment for the party making the threat."

"While prudence suggests leaving open a way of escape when one threatens
an adversary with mutually painful reprisal, any visible means of escape
may make the threat less credible."

So, a coordinated shift to PHP5 is more persuasive if there is no
possibility for reevaluation.

This brings to mind some possible tactics for Drupal to pursue:
* Provide a chance for distributions and hosting companies to give
feedback on the proposal, prior to the final commitment.
* Provide adequate time for distributions and hosting companies to
upgrade (this is where the input process will also help).
* Recruit as many participating projects as possible.
* Secure concrete, irreversible agreements from all participating projects.
* Clearly communicate the rationale and benefits of the decision,
citing external, legitimate authorities if possible (e.g. Zend).
* Secure internal commitment by adding code, or reimplementing old
code, in ways that eliminate the possibility of reverting to PHP 4.
* Measure the extent to which its community is affected (e.g. by
including a question on php4 vs. php5 in the drupal.org user survey).
* Reduce migration costs by providing documentation or even automated
testing and conversion (coder.module, unit test automation, etc.).
* Promote (communicating to its users) distributions and hosting
companies that provide PHP 5, and penalize or deemphasize those that do not.

.ck

Post by Khalid Baheyeldin
We don't need to publicize that there is a possibility that we can go
back in our decision.

We just said that. I think it is necessary to *force* all participating
projects to switch to PHP 5 only, in all circumstances. If we just give
in when nothing has changed in february 2008, we end up in the same
state as we are now.

Post by Khalid Baheyeldin
Just keep forging ahead as if PHP4 does not exist, and corner ourselves?

If all participating projects (most major ones showed interest) use PHP
5 from 2008 on, we are not cornering ourselves as Drupal's competitors
are also PHP5 only.

Post by Khalid Baheyeldin
We can say that we reevaluate this in February and that is it. We don't
need to make any decisions now (apart from pushing hard for hosts
and distros to go PHP5).

Again, if hosters don't have a reason (= major PHP software is PHP 5
only) to upgrade, they won't. I think it is necessary to decide *now* to
give everyone half a year's notice.
Konstantin Käfer – http://kkaefer.com/

[Larry Garfield]

Post by Chris Kennedy
So, a coordinated shift to PHP5 is more persuasive if there is no
possibility for reevaluation.
* Provide a chance for distributions and hosting companies to give
feedback on the proposal, prior to the final commitment.

I have been thinking in the past few days if contacting selected hosts to get
their feedback as well would be a good idea. Allie mentioned[1] that she
runs a web hosting company and supported this idea because it would make her
life simpler, too. Allie, your input here would be hugely welcome. What's a
good way to go about getting your company and other web hosts on board?

[1] http://lists.drupal.org/archives/development/2007-05/msg00465.html

Post by Chris Kennedy
* Provide adequate time for distributions and hosting companies to
upgrade (this is where the input process will also help).
* Recruit as many participating projects as possible.
* Secure concrete, irreversible agreements from all participating projects.

I am actively working on these points, which is what this thread is about. :-)
If anyone has suggestions for projects that we should contact, let me know.
So far my hitlist consists of Drupal, Joomla, CakePHP, Symfony (they say
they're in), Symfony's partner projects, Gallery, and WordPress (Dries said
he talked to Matt and he's being stubborn, but we need to keep on it).

Post by Chris Kennedy
* Clearly communicate the rationale and benefits of the decision,
citing external, legitimate authorities if possible (e.g. Zend).

We should see if we can get a quote or comment out of Rasmus, too, especially
giving how much he was haranguing on us about this at DrupalCon. :-)

Post by Chris Kennedy
* Secure internal commitment by adding code, or reimplementing old
code, in ways that eliminate the possibility of reverting to PHP 4.

Agreed. I think by just explicitly allowing devs to use PHP 5 features this
will happen naturally for Drupal.

Post by Chris Kennedy
* Measure the extent to which its community is affected (e.g. by
including a question on php4 vs. php5 in the drupal.org user survey).
* Reduce migration costs by providing documentation or even automated
testing and conversion (coder.module, unit test automation, etc.).

The GoPHP5.org site can probably include some guides and tips for making sure
code is PHP 5-friendly.

Post by Chris Kennedy
* Promote (communicating to its users) distributions and hosting
companies that provide PHP 5, and penalize or deemphasize those that do not.

One of the plans for the web site is a listing of web hosts that offer PHP 5.2
as their default setup for new sites (with appropriate "we aren't endorsing
or saying these work with every project, just that they're playing nice" text
to CYA).

Agreed on all points down the line, Chris!
--
Larry Garfield AIM: LOLG42
***@garfieldtech.com ICQ: 6817012

"If nature has made any one thing less susceptible than all others of
exclusive property, it is the action of the thinking power called an idea,
which an individual may exclusively possess as long as he keeps it to
himself; but the moment it is divulged, it forces itself into the possession
of every one, and the receiver cannot dispossess himself of it." -- Thomas
Jefferson

[sime]

Post by Larry Garfield

Post by Chris Kennedy
So, a coordinated shift to PHP5 is more persuasive if there is no
possibility for reevaluation.
* Provide a chance for distributions and hosting companies to give
feedback on the proposal, prior to the final commitment.

I have been thinking in the past few days if contacting selected hosts to get
their feedback as well would be a good idea. Allie mentioned[1] that she
runs a web hosting company and supported this idea because it would make her
life simpler, too. Allie, your input here would be hugely welcome. What's a
good way to go about getting your company and other web hosts on board?

As soon as there is some sort of public announcement, I've got a well
place contact who straddles marketing/support of a largish hosting
company. I *could* test the waters with these guys... (no, can we just
start?) ... well, at your service either way. :)

[FGM]

FWIW, Zend Framework already required PHP 5.1.4 today (1.0RC1).

Given Zend's role in PHP overall, especially towards isolated developers, it
might be interesting to suggest they make the move at the same time as
community projects.

[Frederik 'Freso' S. Olesen]

Post by Larry Garfield
If anyone has suggestions for projects that we should contact, let me know.

How about purely administrative projects that aren't meant to face as
much towards the visitors of a site as to the people behind it? I'm
thinking phpMyAdmin, phpPgAdmin, php(other databases)Admin, web-based
system administration applications (I know they're out there, but
can't remember the names), and similar.

Also, webmails clients such as SquirrelMail and groupware applications
like Horde should also be contacted.

Finally, going through
http://en.wikipedia.org/wiki/Category:PHP_programming_language
probably wouldn't hurt either. ;)

[Larry Garfield]

Post by Khalid Baheyeldin
We don't need to publicize that there is a possibility that we can go
back in our decision.
This is a reality check that leaves a fallback option open. If February
comes and nothing changed as far as distros and hosting companies
are concerned. What are we going to do then? Just keep forging ahead
as if PHP4 does not exist, and corner ourselves?
We can say that we reevaluate this in February and that is it. We don't
need to make any decisions now (apart from pushing hard for hosts
and distros to go PHP5).

It's really disingenuous to plan for optionally changing our minds later. If
all we do come February is say "we only support PHP 5 now" but we've still
made sure the code works on PHP 4, then that's frankly dishonest. Rather,
whatever version we expect to ship next after that date we should permit
ourselves to use PHP 5-only features. There's plenty of places in core where
that would greatly simplify things, but using PHP 5-specific features,
naturally, means breaking PHP 4. (That's the evil cycle that we're all
caught in right now.) And there's no reasonable way to then "undo" that if
we decide "oh well, it wasn't quite as successful as we hoped, I guess we'll
back out".

It would be completely infeasible, not to mention destroying any credibility
Drupal has in the OSS world. "Wait and see" means "don't participate at
all". That would be really unfortunate, as this effort cannot be successful
without big names like Drupal.
--
Larry Garfield AIM: LOLG42
***@garfieldtech.com ICQ: 6817012

"If nature has made any one thing less susceptible than all others of
exclusive property, it is the action of the thinking power called an idea,
which an individual may exclusively possess as long as he keeps it to
himself; but the moment it is divulged, it forces itself into the possession
of every one, and the receiver cannot dispossess himself of it." -- Thomas
Jefferson

[Gerhard Killesreiter]

Post by Khalid Baheyeldin

Post by Konstantin Käfer

Post by Khalid Baheyeldin
So, +1 on this initiative. What is the worst case? Come February
and we see that things have not moved. We evaluate it them and see
what other projects did, and decide either to stay the course, or go
back to PHP4.

I think this is a bad idea. If we go into this process with this
attitude, we won't honestly try to get providers to PHP 5 because
there is no need to for them – they will know that software projects
won't move to 5 when the majority of providers still run 4. When we
really make Drupal (and all the other projects) PHP 5 only starting
from this date, providers won't have a choice but to upgrade.

We don't need to publicize that there is a possibility that we can go
back in our decision.

Well, you already kind of did.

I think that the first patch that goes in after HEAD will be open for
development again should be one that cleans up some corners of the code
where specific allowance for PHO 4 has been made. (I only know of
drupal_clone, though.)

Then, we could rewrite our RSS generator to use SimpleXML.

If we proceed like this, there will be no way we could revert to support
PHP 4 in February 2008.

I think that the code freeze for Drupal 7 will be some time early next
year. D7 would then be the first release that would make PHP 5.2
mandatory. D8 would then be released late in 2008 or early in 2009. Only
when we release D8, D6 will become unsupported and everybody who wants a
supported Drupal version will actually _have_ to upgrade to PHP 5.2.

I think that having 1.5 years advance warning is plenty and hosters who
dont heed it well deserve to go out of business.

Cheers,
Gerhard

[Dries Buytaert]

Post by Gerhard Killesreiter
If we proceed like this, there will be no way we could revert to support
PHP 4 in February 2008.

If I can believe my own projections (http://buytaert.net/php-is-dead-
long-live-php), PHP5 will have a 30% adoption rate by February 2008,
rather than the current 20%. Maybe February 2008 is a little too early?

--
Dries Buytaert :: http://www.buytaert.net/

[Johan Forngren]

Post by Dries Buytaert

Post by Gerhard Killesreiter
If we proceed like this, there will be no way we could revert to support
PHP 4 in February 2008.

If I can believe my own projections (http://buytaert.net/php-is-dead-
long-live-php), PHP5 will have a 30% adoption rate by February 2008,
rather than the current 20%. Maybe February 2008 is a little too early?

Not if we can help push it ;)

--

Post by Dries Buytaert
Dries Buytaert :: http://www.buytaert.net/

[Karoly Negyesi]

Post by Johan Forngren
Not if we can help push it ;)

Right. We want to help PHP 5 adoption not follow it. We are now big enough to be the leader. We are so deep in OO territory already that we might as well begin using the class construct... in due time, we can retire JonBob's excellent how Drupal is OOP essay. I think Barcelona would be a great time to draft what we can gain with PHP5 OOP constructs... even I have ideas...

[sime]

I was having a little chat about PHP5 support with a guy at
midphase.com. Off the record, at this stage, they generally support the
movement, but the main problem for them with *pushing* PHP5 is Fantastico.

Had a little look around. Here is a list of scripts supported by
Fantastico, along with their PHP5 status. Hope this is useful.
http://tdknights.com/fantastico.htm

(It was obtained from this thread (Fantastico forum):
http://www.netenberg.com/forum/viewtopic.php?t=3933)

.s

[Larry Garfield]

I'll add those projects to my contact list, thanks. :-)

Most of them it looks like already run in PHP 5. For the few that don't, excuse me for being blunt but as far as I'm concerned those projects are abandoned. Not even running on PHP 5 now, after 3 years, is simply inexcusable. (Although I know that at least one or two of those are working on new PHP 5-friendly or PHP 5-only versions.)

Does anyone have any contacts at Fantastico or cPanel to see how we could get them on board (which would probably mean not including PHP 5-incompatible apps in their next releases)?

--Larry Garfield

Post by sime
I was having a little chat about PHP5 support with a guy at
midphase.com. Off the record, at this stage, they generally support the
movement, but the main problem for them with *pushing* PHP5 is Fantastico.
Had a little look around. Here is a list of scripts supported by
Fantastico, along with their PHP5 status. Hope this is useful.
http://tdknights.com/fantastico.htm
http://www.netenberg.com/forum/viewtopic.php?t=3933)
.s

[Gerhard Killesreiter]

Post by sime
I was having a little chat about PHP5 support with a guy at
midphase.com. Off the record, at this stage, they generally support the
movement, but the main problem for them with *pushing* PHP5 is Fantastico.
Had a little look around. Here is a list of scripts supported by
Fantastico, along with their PHP5 status. Hope this is useful.
http://tdknights.com/fantastico.htm

This list does not seem to be up to date. I've checked phpwcms and their
latest release is 1.3.3 and according to this url:

http://phpmagazin.de/cms/index.php?site=159&start=158&show=twolist

php 5 is supported. phpwcms was the only cms that was listed as not
supporting php5.

Cheers,
Gerhard

[sime]

Post by Gerhard Killesreiter

Post by sime
Had a little look around. Here is a list of scripts supported by
Fantastico, along with their PHP5 status. Hope this is useful.
http://tdknights.com/fantastico.htm

This list does not seem to be up to date. I've checked phpwcms and their
http://phpmagazin.de/cms/index.php?site=159&start=158&show=twolist

Thanks Gerhard

I've updated the thread, assuming that the author of the list will update.
http://netenberg.com/forum/viewtopic.php?p=27362#27362

[Larry Garfield]

Post by Johan Forngren

Post by Dries Buytaert

Post by Gerhard Killesreiter
If we proceed like this, there will be no way we could revert to support
PHP 4 in February 2008.

If I can believe my own projections (http://buytaert.net/php-is-dead-
long-live-php), PHP5 will have a 30% adoption rate by February 2008,
rather than the current 20%. Maybe February 2008 is a little too early?

Not if we can help push it ;)

That is precisely the intent of the GoPHP5 effort. :-) Change the curve so that it's not 20%-30%, but 50%+. That pushes past the tipping point, and PHP 4 can more quickly go the direction of PHP 3. To do that, though, requires market pressure. We're trying to be that market pressure.

--Larry Garfield

[Dries Buytaert]

Post by Larry Garfield

Post by Johan Forngren

Post by Dries Buytaert
If I can believe my own projections (http://buytaert.net/php-is-
dead-
long-live-php), PHP5 will have a 30% adoption rate by February 2008,
rather than the current 20%. Maybe February 2008 is a little too early?

Not if we can help push it ;)

That is precisely the intent of the GoPHP5 effort. :-) Change the
curve so that it's not 20%-30%, but 50%+. That pushes past the
tipping point, and PHP 4 can more quickly go the direction of PHP
3. To do that, though, requires market pressure. We're trying to
be that market pressure.

Sure, I understand that much. The question is: what if we fail to
bend the curve?

My guess is that all Drupal installations take up at most 0.5% of
PHP's total install base. That doesn't make for a lot of "bending
power", does it? Add a dozen of other systems, and it might add up
to 10% of PHP's total install base?

Are we willing to take this bet and to leave most of our users behind
when we failed to significantly bend the curve 6 months down the
road? Clearly, being able to use PHP5 would help us help our users.
Not providing an upgrade path for 70-80% of our install base seems to
be at odd with that. It certainly has something kamikaze-like.

Will people remember gophp5.org two weeks from now after it fell of
the Digg main page? Just asking.

I'm all for pushing PHP5, but I'm not sure I want to take such an
extreme stance. As I mentioned in my blog post, let's start by
making some features PHP5 only. Like, let's rewrite the aggregator
module with PHP5's simplified XML parser API. That's disruptive too,
but at least we'd be shooting ourselves in the foot, rather than
through the head.

--
Dries Buytaert :: http://www.buytaert.net/

[Morbus Iff]

Post by Dries Buytaert
I'm all for pushing PHP5, but I'm not sure I want to take such an
extreme stance. As I mentioned in my blog post, let's start by
making some features PHP5 only. Like, let's rewrite the aggregator
module with PHP5's simplified XML parser API. That's disruptive too,
but at least we'd be shooting ourselves in the foot, rather than

I'm against this, personally - I'd much rather see us standardize around
an external library like SimplePie. RSS and Atom parsers are a very very
complicated thing, and our parsing issue queue is quite indicative of
that. I'd much rather use a library that is /dedicated/ to that task
then the parser we currently have.

What if, instead of rewriting aggregator, we simply shove it out of core
(ala archive.module) and recommend people use something like Feed Parser
or SimpleFeed, both of which use and require SimplePie, both require a
healthy dose of loving, and both offer a much stronger API?

[Boris Mann]

Post by Morbus Iff

Post by Dries Buytaert
I'm all for pushing PHP5, but I'm not sure I want to take such an
extreme stance. As I mentioned in my blog post, let's start by
making some features PHP5 only. Like, let's rewrite the aggregator
module with PHP5's simplified XML parser API. That's disruptive too,
but at least we'd be shooting ourselves in the foot, rather than

Dries *was* just using that as an example....but you're right, it's probably
about time to bang the drum about this...again.

I'm against this, personally - I'd much rather see us standardize around

Post by Morbus Iff
an external library like SimplePie. RSS and Atom parsers are a very very
complicated thing, and our parsing issue queue is quite indicative of
that. I'd much rather use a library that is /dedicated/ to that task
then the parser we currently have.

+1. And lo, the discussion from many years continues.

What if, instead of rewriting aggregator, we simply shove it out of core

Post by Morbus Iff
(ala archive.module) and recommend people use something like Feed Parser
or SimpleFeed, both of which use and require SimplePie, both require a
healthy dose of loving, and both offer a much stronger API?

Neither one of those modules is a suitable replacement for aggregator in
core (both do feeds-as-nodes and feed-items-as-nodes ... which is great /
necessary for some use cases, but bad for others).

See http://groups.drupal.org/node/4624 --> Aggregation API as a GSOC
project.

Side note: core needs love. It needs your love, and everyone's love. Views
in core, more of CCK in core, improved / re-factored / API-ized aggregator
-- etc. etc. etc. Oh yes, it also needs testing love, issue queue review
love, documentation love. Where's the love?

[Morbus Iff]

Post by Boris Mann
Neither one of those modules is a suitable replacement for aggregator in
core (both do feeds-as-nodes and feed-items-as-nodes ... which is great
/ necessary for some use cases, but bad for others).

Not entirely true - the Feed Parser module does have an "items as
aggregator.module" module, but it has fallen into disrepair. I plan to
fix that up for my own needs.

Post by Boris Mann
Side note: core needs love. It needs your love, and everyone's love.
Views in core, more of CCK in core, improved / re-factored / API-ized
aggregator -- etc. etc. etc. Oh yes, it also needs testing love, issue
queue review love, documentation love. Where's the love?

Is this intended at me? I can scratch only my own itches nowadays.

Post by Boris Mann
+1. And lo, the discussion from many years continues.

Unfortunately, yes. Can you sum up why it's never settled?

[Boris Mann]

Post by Morbus Iff

Post by Boris Mann
Neither one of those modules is a suitable replacement for aggregator in
core (both do feeds-as-nodes and feed-items-as-nodes ... which is great
/ necessary for some use cases, but bad for others).

Not entirely true - the Feed Parser module does have an "items as
aggregator.module" module, but it has fallen into disrepair. I plan to
fix that up for my own needs.

Urgh. Double Urgh. The Feed Parser stuck in 4.7 for which there are now
better options :( -- /me laments for the non-convergence of forces around
aggregation.

Post by Morbus Iff
Side note: core needs love. It needs your love, and everyone's love.

Post by Boris Mann
Views in core, more of CCK in core, improved / re-factored / API-ized
aggregator -- etc. etc. etc. Oh yes, it also needs testing love, issue
queue review love, documentation love. Where's the love?

Is this intended at me? I can scratch only my own itches nowadays.

No. it's intended at everyone. Just a "feeling" I'm getting.

Post by Morbus Iff
+1. And lo, the discussion from many years continues.
Unfortunately, yes. Can you sum up why it's never settled?

Dries wrote the parser functions originally to spec, and loves a challenge
:P

Stuff like Magpie was not really very pretty code in the past, so the whole
"we can likely do this better" as well the great XMLRPC external library
scare. These days, SimplePie is pretty solid.

Other than that....no one has tackled it with actual code.

[Morbus Iff]

Post by Boris Mann

Post by Morbus Iff
Not entirely true - the Feed Parser module does have an "items as
aggregator.module" module, but it has fallen into disrepair. I plan to
fix that up for my own needs.

Urgh. Double Urgh. The Feed Parser stuck in 4.7 for which there are now
better options :( -- /me laments for the non-convergence of forces
around aggregation.

Eh? I don't know what you're saying.

[Boris Mann]

Post by Morbus Iff

Post by Boris Mann

Post by Morbus Iff
Not entirely true - the Feed Parser module does have an "items as
aggregator.module" module, but it has fallen into disrepair. I plan to
fix that up for my own needs.

Urgh. Double Urgh. The Feed Parser stuck in 4.7 for which there are now
better options :( -- /me laments for the non-convergence of forces
around aggregation.

Eh? I don't know what you're saying.

Projects:
* aggregator2 (abandoned)
* Aggregation
* Leech (sort of agg2, DevSeed are doing good work but I think the
foundation has always been flawed)
* Feedparser
* SimpleFeed

That would be what I mean by non-convergence.

[Morbus Iff]

Yes, and I took a look at all of these. Only FeedParser had a strong
enough foot, unfortunately (even in its unworkable state) - SimpleFeed
had a lot of the same features, but had a fairly critical design flaw
that stopped me from doing any work with it at all: only one module can
respond to its hook API (ie., I can't have a "save in database as item"
module and a "send out through email" module both working on one feed -
it's either one or the other). I did a fairly aggressive code review of
the primary module and included upwards of 20 issues for it (a number of
which have been fixed), and ended up basically unhappy with it.

Post by Boris Mann
* aggregator2 (abandoned)

Shouldn't even be on the list.

Post by Boris Mann
* Aggregation
* Leech (sort of agg2, DevSeed are doing good work but
I think the foundation has always been flawed)

Items only as nodes, no API to stop node creation, and its own parsers.
Leech also gives me a generic feeling of "what the hell is going on?",
which is never a good thing.

[Ian Ward]

Aron is starting to pull these apart and is doing some benchmarking
and feature comparisons
http://groups.drupal.org/node/4519
http://groups.drupal.org/node/4547

I know he's putting some real time into this the next few weeks, and
one idea is a pluggable parser - so you could have one for php4, one
for php5, something using simpleXML, another using simplepie. This
is at least an idea, and I think he's going to iron them out still
further on the group wiki.

Ian

Post by Morbus Iff

Post by Boris Mann

Post by Morbus Iff
Not entirely true - the Feed Parser module does have an "items as
aggregator.module" module, but it has fallen into disrepair. I

plan to

Post by Boris Mann

Post by Morbus Iff
fix that up for my own needs.

Urgh. Double Urgh. The Feed Parser stuck in 4.7 for which there

are now

Post by Boris Mann
better options :( -- /me laments for the non-convergence of forces
around aggregation.

Eh? I don't know what you're saying.
* aggregator2 (abandoned)
* Aggregation
* Leech (sort of agg2, DevSeed are doing good work but I think the
foundation has always been flawed)
* Feedparser
* SimpleFeed
That would be what I mean by non-convergence.
--
Boris Mann
Office 604-682-2889
Skype borismann
http://www.bryght.com

Ian Ward
Development Seed Inc.
Technology Development and Discovery
http://www.developmentseed.org
http://www.developmentseed.org/blog
developmentseedperu(skype)
Tel. [phone]
Fax. [phone]

[Gabor Hojtsy]

Post by Morbus Iff

Post by Boris Mann
+1. And lo, the discussion from many years continues.

Unfortunately, yes. Can you sum up why it's never settled?

We have not seen patches (and even when we seen patches, love was
missing) to improve the in-core aggregator, the in-core profile module
and so on. It is a lot more comfortable to develop in contrib for some
cases/people so obviously out of core profile, aggregator, etc solutions
emerged.

The lone souls who do try to fix up core modules does not get the love,
reviews and tests... Eg. http://drupal.org/node/141419 where Dries noted
he is fine with that way to go...

Gabor

[Karoly Negyesi]

Aggregator is

a) a feed parser
b) a storage engine

So the first action point is to make both pluggable. Then we can have a PHP4 parser (yay we have that!) our own PHP5 parser (to be written), SimpleXML (we have contrib code for it). Then there can be the current KISS storage engine and a contrib storage engine which stores into node. This could end the multitude of parallel feed modules --hopefully.

And yes, I will work on this for Drupal 7 _if noone else does_. Comment module turned out better than I feared of, so I hope that if I can show the way someone will run with the aggregator code, too.

[Alexander Barth]

Post by Morbus Iff

Post by Dries Buytaert
I'm all for pushing PHP5, but I'm not sure I want to take such an
extreme stance. As I mentioned in my blog post, let's start by
making some features PHP5 only. Like, let's rewrite the aggregator
module with PHP5's simplified XML parser API. That's disruptive too,
but at least we'd be shooting ourselves in the foot, rather than

I'm against this, personally - I'd much rather see us standardize around
an external library like SimplePie.
RSS and Atom parsers are a very very
complicated thing, and our parsing issue queue is quite indicative of
that. I'd much rather use a library that is /dedicated/ to that task
then the parser we currently have.

Especially after Aron's measurements, that show SimplePie being
more than 20 times slower than the Drupal's core parser
(http://groups.drupal.org/node/4519) I am beginning to lose hope in a
one parser for all solution.

The said complexity and variety of feed formats and use cases is
rather asking for a pluggable architecture where you can chose what
parser suits you best.

Post by Morbus Iff
What if, instead of rewriting aggregator, we simply shove it out of core
(ala archive.module) and recommend people use something like Feed Parser
or SimpleFeed, both of which use and require SimplePie, both require a
healthy dose of loving, and both offer a much stronger API?

In core or not, we I think we need a _common_ solution. Ian just posted it
before - I repeat it nevertheless: Aron Novak is working for his SoC
project on an aggregation solution that should replace aggregator module
in core and provide a fundament for contrib modules.

See here: http://groups.drupal.org/rss-aggregation
and: http://aggregation.novaak.net

Alex

[Morbus Iff]

Post by Alexander Barth
Especially after Aron's measurements, that show SimplePie being
more than 20 times slower than the Drupal's core parser
(http://groups.drupal.org/node/4519) I am beginning to lose hope in a
one parser for all solution.

I don't want one-parser-for-all-solutions either - see my comments on
the wiki, which includes one potential workflow that I personally need:

http://groups.drupal.org/node/4624

With that said, speed is not really an issue here, IMO - just like we
can (and will) fiddle/scale with the downloading of thousands of feeds
in hook_cron, the same thing can be done with processing time. As
already mentioned in the comments of his tests, SimplePie does a lot of
filtering to ensure the data is sanitized (granted, a lot of what Drupal
could do as well, but which wouldn't have been considered part of the
original parsing times, I suspect). What SimplePie does provide,
however, is a much greater field of correct parsing - something that
Drupal's core parser fails to do (helpfully contributing to its
appearance of "speed").

Post by Alexander Barth
See here: http://groups.drupal.org/rss-aggregation
and: http://aggregation.novaak.net

Yes, I've been watching. However, this is
Drupal 7 material. I can't wait that long.

[Alexander Barth]

Post by Morbus Iff
With that said, speed is not really an issue here, IMO - just like we
can (and will) fiddle/scale with the downloading of thousands of feeds
in hook_cron, the same thing can be done with processing time. As
already mentioned in the comments of his tests, SimplePie does a lot of
filtering to ensure the data is sanitized (granted, a lot of what Drupal
could do as well, but which wouldn't have been considered part of the
original parsing times, I suspect). What SimplePie does provide,
however, is a much greater field of correct parsing - something that
Drupal's core parser fails to do (helpfully contributing to its
appearance of "speed").

Unfortunately, speed is an issue for our aggregation sites where
parsing time adds up accross all Drupal installations on the
server. I don't object to that SimplePie may well be the better
choice for a general use case - can we agree on a pluggable solution,
though?

Post by Morbus Iff

Post by Alexander Barth
See here: http://groups.drupal.org/rss-aggregation
and: http://aggregation.novaak.net

Yes, I've been watching. However, this is
Drupal 7 material. I can't wait that long.

I can understand. But let's not desmiss the new aggregator solution as
Drupal 7 only material. A backport to 6 or even 5 could be very easy
to do. Moreover, it's pretty likely that Aron actually starts
developing the new aggregator on Drupal 5.

From a more personal point of view, I really would like to get rid of
leech and build on top of common ground as soon as possible - not only
when 7's out.

Alex

[Boris Mann]

Post by Alexander Barth
Unfortunately, speed is an issue for our aggregation sites where
parsing time adds up accross all Drupal installations on the
server.

There are some good arguments to be made for decoupling parsing /
aggregating directly within Drupal and moving it outside -- just putting
nodes into Drupal.

I don't object to that SimplePie may well be the better

Post by Alexander Barth
choice for a general use case - can we agree on a pluggable solution,
though?

Yes.

I can understand. But let's not desmiss the new aggregator solution as

Post by Alexander Barth
Drupal 7 only material. A backport to 6 or even 5 could be very easy
to do. Moreover, it's pretty likely that Aron actually starts
developing the new aggregator on Drupal 5.
From a more personal point of view, I really would like to get rid of
leech and build on top of common ground as soon as possible - not only
when 7's out.

Hallelujah. So...if Morbus needs to move, can we start on the foundations of
the one, true pluggable aggregation API of doom, together, now?

[Ashraf Amayreh]

Didn't notice this thread in development. I'll read it in detail later
today.

In the meantime, I was pointed to this link by a friend of mine which shows
that simplePie is a complete no-no. This actually compares three of the
contributed aggregation modules in terms of performance, my aggregation
module is included in the comparison.

The aggregation module requires PHP 5 (since I use simpleXML), provides an
API, and if you take a look at the issue queue you'll find that you don't
need any external libraries to get very stable behavior. Aside from the
initial wave of bug I got after completing it, I haven't received a bug
issue in two months or so. SimplePie is forbiddingly performance terrible,
or so I understood from Aron Novak's article which you can view below.

http://groups.drupal.org/node/4519

Post by Boris Mann

Post by Alexander Barth
Unfortunately, speed is an issue for our aggregation sites where
parsing time adds up accross all Drupal installations on the
server.

There are some good arguments to be made for decoupling parsing /
aggregating directly within Drupal and moving it outside -- just putting
nodes into Drupal.
I don't object to that SimplePie may well be the better

Post by Alexander Barth
choice for a general use case - can we agree on a pluggable solution,
though?

Yes.
I can understand. But let's not desmiss the new aggregator solution as

Post by Alexander Barth
Drupal 7 only material. A backport to 6 or even 5 could be very easy
to do. Moreover, it's pretty likely that Aron actually starts
developing the new aggregator on Drupal 5.
From a more personal point of view, I really would like to get rid of
leech and build on top of common ground as soon as possible - not only
when 7's out.

Hallelujah. So...if Morbus needs to move, can we start on the foundations
of the one, true pluggable aggregation API of doom, together, now?
--
Boris Mann
Office 604-682-2889
Skype borismann
http://www.bryght.com

[Morbus Iff]

received a bug issue in two months or so. SimplePie is forbiddingly
performance terrible, or so I understood from Aron Novak's article which

Unfortunately, we can't take these statistics as canon:

* there's no instructions on how to duplicate.

* the SimplePie result is an estimate ("At SimplePie I have to
do an estimate, because the feed download time was accumulated
to the measure."

* it is unknown whether the other feed parsers are doing the
same sanitization that SimplePie does, again, which adds
more time to the results.

I won't be convinced that the processing time of SimplePie is an
accurate and overwhelming negative to it's more proper parsing support
until I see repeatable and clean room results (no download, no
sanitizing, etc.)

[Morbus Iff]

Post by Morbus Iff
* there's no instructions on how to duplicate.
* the SimplePie result is an estimate ("At SimplePie I have to
do an estimate, because the feed download time was accumulated
to the measure."
* it is unknown whether the other feed parsers are doing the
same sanitization that SimplePie does, again, which adds
more time to the results.

I have done some quick tests, using the same URL as Aron:

http://www.christiannewswire.com/rss/catfeed_2.xml

I downloaded this file to my desktop. I will be passing this string into
SimplePie instead of allowing SimplePie to download it. The file is 1M:

1027320 Jun 19 11:50 catfeed_2.xml

This is the script I used with SimplePie 1.0 b3.2 (20061124):

<?php
$handle = fopen('./catfeed_2.xml', "r");
$contents = fread($handle, filesize('./catfeed_2.xml'));

require './simplepie.inc';
$feed = new SimplePie();
$feed->set_raw_data($contents);
$feed->init();
$parsed = $feed->get_items();
?>

With this command line:

~/Desktop > date && php simplepie.php && date
Tue Jun 19 12:26:10 EDT 2007
Tue Jun 19 12:26:22 EDT 2007

As you can see, this does confirm the 10 or 12 second parse time -- it
is also using all the sanitation that SimplePie does by default.
However, SimpleFeed and FeedParser both ship with the latest development
version of SimplePie which includes an option to stop this sanitation:

$feed->set_stupidly_fast(TRUE);

I grabbed today's development version, added the above
line before the ->init() in the above script, and reran:

~/Desktop > date && php simplepie.php && date
Tue Jun 19 12:28:54 EDT 2007
Tue Jun 19 12:28:55 EDT 2007

You'll notice that it is only 1 second which removes all doubt in my
mind that SimplePie is a bad thing comparitively (since one would assume
you'd sanitize the data as necessary within Drupal).

[Ashraf Amayreh]

I'm not really sure about the argument to sanitize data. Can't we sanitize
it in a little less than 11 seconds? Also, isn't there a possibility the
user wants this HTML code to come in as HTML code rather than plain text?

I would guess that my module does lack many sanity checks, but at the same
time, I do assume that administrators should be responsible as to what feeds
they add to their sites.

By the way, any sanity gurus who would like to check on my module's sanity
checks and help me with additional sanity checks are very welcome and have
my full gratitude. Just drop me a line off-list.

Post by Morbus Iff

Post by Morbus Iff
* there's no instructions on how to duplicate.
* the SimplePie result is an estimate ("At SimplePie I have to
do an estimate, because the feed download time was accumulated
to the measure."
* it is unknown whether the other feed parsers are doing the
same sanitization that SimplePie does, again, which adds
more time to the results.

http://www.christiannewswire.com/rss/catfeed_2.xml
I downloaded this file to my desktop. I will be passing this string into
1027320 Jun 19 11:50 catfeed_2.xml
<?php
$handle = fopen('./catfeed_2.xml', "r");
$contents = fread($handle, filesize('./catfeed_2.xml'));
require './simplepie.inc';
$feed = new SimplePie();
$feed->set_raw_data($contents);
$feed->init();
$parsed = $feed->get_items();
?>
~/Desktop > date && php simplepie.php && date
Tue Jun 19 12:26:10 EDT 2007
Tue Jun 19 12:26:22 EDT 2007
As you can see, this does confirm the 10 or 12 second parse time -- it
is also using all the sanitation that SimplePie does by default.
However, SimpleFeed and FeedParser both ship with the latest development
$feed->set_stupidly_fast(TRUE);
I grabbed today's development version, added the above
~/Desktop > date && php simplepie.php && date
Tue Jun 19 12:28:54 EDT 2007
Tue Jun 19 12:28:55 EDT 2007
You'll notice that it is only 1 second which removes all doubt in my
mind that SimplePie is a bad thing comparitively (since one would assume
you'd sanitize the data as necessary within Drupal).
--
Morbus Iff ( and think about the bad things that I didn't do )
Technical: http://www.oreillynet.com/pub/au/779
Culture: http://www.disobey.com/ and http://www.gamegrene.com/
aim: akaMorbus / skype: morbusiff / icq: 2927491 / jabber.org: morbus

[Morbus Iff]

Post by Ashraf Amayreh
I'm not really sure about the argument to sanitize data. Can't we
sanitize it in a little less than 11 seconds? Also, isn't there a

That would be up to Drupal - regardless of what parser you're using, how
and when you sanitize it depends on the use case. My tests were merely
to remove SimpleFeed's sanitation, forcing it into the same baseline as
any other "made for Drupal" parser - use Drupal's routines as necessary.

[Larry Garfield]

Disclaimer: I am not an RSS guru, just a pedant. :-)

RSS is XML. The XML spec explicitly says that invalid files should be discarded, not guessed at the way HTML is. Trying to make sense of a broken RSS feed is explicitly contrary to the spec. So, er, why are we spending so much time trying to sanitize? If it doesn't parse correctly, report an error "this site's RSS feed is f*ed up, tell 'em to fix it". Am I missing something here?

--Larry Garfield

Post by Ashraf Amayreh
I'm not really sure about the argument to sanitize data. Can't we sanitize
it in a little less than 11 seconds? Also, isn't there a possibility the
user wants this HTML code to come in as HTML code rather than plain text?
I would guess that my module does lack many sanity checks, but at the same
time, I do assume that administrators should be responsible as to what feeds
they add to their sites.
By the way, any sanity gurus who would like to check on my module's sanity
checks and help me with additional sanity checks are very welcome and have
my full gratitude. Just drop me a line off-list.

Post by Morbus Iff

Post by Morbus Iff
* there's no instructions on how to duplicate.
* the SimplePie result is an estimate ("At SimplePie I have to
do an estimate, because the feed download time was accumulated
to the measure."
* it is unknown whether the other feed parsers are doing the
same sanitization that SimplePie does, again, which adds
more time to the results.

http://www.christiannewswire.com/rss/catfeed_2.xml
I downloaded this file to my desktop. I will be passing this string into
1027320 Jun 19 11:50 catfeed_2.xml
<?php
$handle = fopen('./catfeed_2.xml', "r");
$contents = fread($handle, filesize('./catfeed_2.xml'));
require './simplepie.inc';
$feed = new SimplePie();
$feed->set_raw_data($contents);
$feed->init();
$parsed = $feed->get_items();
?>
~/Desktop > date && php simplepie.php && date
Tue Jun 19 12:26:10 EDT 2007
Tue Jun 19 12:26:22 EDT 2007
As you can see, this does confirm the 10 or 12 second parse time -- it
is also using all the sanitation that SimplePie does by default.
However, SimpleFeed and FeedParser both ship with the latest development
$feed->set_stupidly_fast(TRUE);
I grabbed today's development version, added the above
~/Desktop > date && php simplepie.php && date
Tue Jun 19 12:28:54 EDT 2007
Tue Jun 19 12:28:55 EDT 2007
You'll notice that it is only 1 second which removes all doubt in my
mind that SimplePie is a bad thing comparitively (since one would assume
you'd sanitize the data as necessary within Drupal).
--
Morbus Iff ( and think about the bad things that I didn't do )
Technical: http://www.oreillynet.com/pub/au/779
Culture: http://www.disobey.com/ and http://www.gamegrene.com/
aim: akaMorbus / skype: morbusiff / icq: 2927491 / jabber.org: morbus

[Earl Miles]

Post by Larry Garfield
Disclaimer: I am not an RSS guru, just a pedant. :-)
RSS is XML. The XML spec explicitly says that invalid files should be discarded, not guessed at the way HTML is. Trying to make sense of a broken RSS feed is explicitly contrary to the spec. So, er, why are we spending so much time trying to sanitize? If it doesn't parse correctly, report an error "this site's RSS feed is f*ed up, tell 'em to fix it". Am I missing something here?

Because most people don't care if a site's XML is a little messed up. They care
about the data; throwing away the data for trivial errors will upset the
consumers of the data.

[Gerhard Killesreiter]

Post by Earl Miles

Post by Larry Garfield
Disclaimer: I am not an RSS guru, just a pedant. :-)
RSS is XML. The XML spec explicitly says that invalid files should be
discarded, not guessed at the way HTML is. Trying to make sense of a
broken RSS feed is explicitly contrary to the spec. So, er, why are
we spending so much time trying to sanitize? If it doesn't parse
correctly, report an error "this site's RSS feed is f*ed up, tell 'em
to fix it". Am I missing something here?

Because most people don't care if a site's XML is a little messed up.
They care about the data; throwing away the data for trivial errors will
upset the consumers of the data.

Then they should direct their complaints to the providers of said broken
data. Drupal has had a rather good record wrt not validating somebody
else's broken feeds. Did this change?

Cheers,
Gerhard

[Alexander Barth]

Post by Larry Garfield
Disclaimer: I am not an RSS guru, just a pedant. :-)
RSS is XML. The XML spec explicitly says that invalid files should
be discarded, not guessed at the way HTML is. Trying to make sense
of a broken RSS feed is explicitly contrary to the spec. So, er,
why are we spending so much time trying to sanitize? If it doesn't
parse correctly, report an error "this site's RSS feed is f*ed up,
tell 'em to fix it". Am I missing something here?

True. That's one side of the coin. The other side is a world of
non compliant feeds that all turn up on your issue queue to haunt you
if your parser complains about them :)

Alex

Post by Larry Garfield
--Larry Garfield

Post by Ashraf Amayreh
I'm not really sure about the argument to sanitize data. Can't we sanitize
it in a little less than 11 seconds? Also, isn't there a possibility the
user wants this HTML code to come in as HTML code rather than plain text?
I would guess that my module does lack many sanity checks, but at the same
time, I do assume that administrators should be responsible as to what feeds
they add to their sites.
By the way, any sanity gurus who would like to check on my module's sanity
checks and help me with additional sanity checks are very welcome and have
my full gratitude. Just drop me a line off-list.

Post by Morbus Iff

Post by Morbus Iff
* there's no instructions on how to duplicate.
* the SimplePie result is an estimate ("At SimplePie I have to
do an estimate, because the feed download time was accumulated
to the measure."
* it is unknown whether the other feed parsers are doing the
same sanitization that SimplePie does, again, which adds
more time to the results.

http://www.christiannewswire.com/rss/catfeed_2.xml
I downloaded this file to my desktop. I will be passing this string into
1027320 Jun 19 11:50 catfeed_2.xml
<?php
$handle = fopen('./catfeed_2.xml', "r");
$contents = fread($handle, filesize('./catfeed_2.xml'));
require './simplepie.inc';
$feed = new SimplePie();
$feed->set_raw_data($contents);
$feed->init();
$parsed = $feed->get_items();
?>
~/Desktop > date && php simplepie.php && date
Tue Jun 19 12:26:10 EDT 2007
Tue Jun 19 12:26:22 EDT 2007
As you can see, this does confirm the 10 or 12 second parse time -- it
is also using all the sanitation that SimplePie does by default.
However, SimpleFeed and FeedParser both ship with the latest development
$feed->set_stupidly_fast(TRUE);
I grabbed today's development version, added the above
~/Desktop > date && php simplepie.php && date
Tue Jun 19 12:28:54 EDT 2007
Tue Jun 19 12:28:55 EDT 2007
You'll notice that it is only 1 second which removes all doubt in my
mind that SimplePie is a bad thing comparitively (since one would assume
you'd sanitize the data as necessary within Drupal).
--
Morbus Iff ( and think about the bad things that I didn't do )
Technical: http://www.oreillynet.com/pub/au/779
Culture: http://www.disobey.com/ and http://www.gamegrene.com/
aim: akaMorbus / skype: morbusiff / icq: 2927491 / jabber.org: morbus

[Boris Mann]

Post by Larry Garfield
Disclaimer: I am not an RSS guru, just a pedant. :-)
RSS is XML. The XML spec explicitly says that invalid files should be
discarded, not guessed at the way HTML is. Trying to make sense of a broken
RSS feed is explicitly contrary to the spec. So, er, why are we spending so
much time trying to sanitize? If it doesn't parse correctly, report an
error "this site's RSS feed is f*ed up, tell 'em to fix it". Am I missing
something here?

And this is the point where I dive back in....

Many many many people have argued this.

Fact: many non proper XML RSS feeds exist in the wild.
Fact: if Drupal doesn't parse it, when other applications do, Drupal looks
"broken"
Fact: regular people like stuff that "just works" with any RSS feed out
there, and will pick that over XML pedantry every day.

A checkbox for "discard invalid XML" makes perfect sense....for *some feeds*
and *some use cases*.

[Ian Ward]

Post by Larry Garfield
Disclaimer: I am not an RSS guru, just a pedant. :-)
RSS is XML. The XML spec explicitly says that invalid files should
be discarded, not guessed at the way HTML is. Trying to make sense
of a broken RSS feed is explicitly contrary to the spec. So, er,
why are we spending so much time trying to sanitize? If it doesn't
parse correctly, report an error "this site's RSS feed is f*ed up,
tell 'em to fix it". Am I missing something here?

This is another +1 for a pluggable parser approach. Then you (site
admin) can decide on a case by case basis whether to use the parser
that handles it all or only valid. So maybe validation is done in
the main module and your plugin parser can flip that switch on and
off whether to proceed at own will when validation fails. Either
way, you can do it either way w/ plugability.

Post by Larry Garfield
And this is the point where I dive back in....
Many many many people have argued this.
Fact: many non proper XML RSS feeds exist in the wild.
Fact: if Drupal doesn't parse it, when other applications do,
Drupal looks "broken"
Fact: regular people like stuff that "just works" with any RSS
feed out there, and will pick that over XML pedantry every day.
A checkbox for "discard invalid XML" makes perfect sense....for
*some feeds* and *some use cases*.
--
Boris Mann
Office 604-682-2889
Skype borismann
http://www.bryght.com

Ian Ward
Development Seed Inc.
Technology Development and Discovery
http://www.developmentseed.org
http://www.developmentseed.org/blog
developmentseedperu(skype)
Tel. [phone]
Fax. [phone]

[Khalid Baheyeldin]

Post by Boris Mann

Post by Larry Garfield
Disclaimer: I am not an RSS guru, just a pedant. :-)
RSS is XML. The XML spec explicitly says that invalid files should be
discarded, not guessed at the way HTML is. Trying to make sense of a broken
RSS feed is explicitly contrary to the spec. So, er, why are we spending so
much time trying to sanitize? If it doesn't parse correctly, report an
error "this site's RSS feed is f*ed up, tell 'em to fix it". Am I missing
something here?

And this is the point where I dive back in....
Many many many people have argued this.
Fact: many non proper XML RSS feeds exist in the wild.
Fact: if Drupal doesn't parse it, when other applications do, Drupal looks
"broken"
Fact: regular people like stuff that "just works" with any RSS feed out
there, and will pick that over XML pedantry every day.
A checkbox for "discard invalid XML" makes perfect sense....for *some
feeds* and *some use cases*

I strongly agree with Boris.

Again, it goes to the point of how big a problem is it and can you afford to
ignore it?

If a web site sends bad HTML. Should browsers be so uptight as to popup
messages
for each error that is in the HTML? Or should it try to make the best of
what is passed
on silently? Guess what browsers do today?

The same goes for MS IE and how non-standards compliant it is. Do we ignore
it? No,
because of its market share, as painful as it is.

So, aggregators should do the same: try to make the best out of the data,
even if it
has some bad elements.

[Larry Garfield]

Post by Boris Mann
Fact: many non proper XML RSS feeds exist in the wild.
Fact: if Drupal doesn't parse it, when other applications do, Drupal looks
"broken"
Fact: regular people like stuff that "just works" with any RSS feed out
there, and will pick that over XML pedantry every day.

And once again, stupid people make the world a worse place with slower software.

Even with that, though, we really shouldn't agonize over being as forgiving of bad RSS as, say, a browser is of bad HTML. If it's not even well-formed XML, then we're doing everyone, including ourselves, a disservice by trying to handle it. 12 seconds worth of cleanup is a waste of 11.8 seconds.

--Larry Garfield

[Morbus Iff]

Post by Larry Garfield
Even with that, though, we really shouldn't agonize over being as forgiving of bad RSS as, say, a browser is of bad HTML. If it's not even well-formed XML, then we're doing everyone, including ourselves, a disservice by trying to handle it. 12 seconds worth of cleanup is a waste of 11.8 seconds.

Oh God, not you too.

People! Sanitation has /nothing/ to do with whether a feed is valid or
well-formed or not! SimplePie is sanitizing feeds similarly to the
Drupal equivalents of check_plain, check_markup, filter_xss, etc., etc.
It is not fixing broken XML feeds. Please stop thinking that AA knows
what he thinks I'm talking about.

[Ashraf Amayreh]

Listen, sanitizing as in protecting against XSS and SQL injection is a must.
But others seem to include parsing not fully compliant XML feeds and not
fully standards compliant feeds as part of sanitization. In my opinion the
second is not sanitization and no aggregator needs to waste the code and
time on trying to handle non-XML or non-standards compliant feeds.

I would be very surprised if I found that SimplePie is wasting 11 seconds
out of 12 in preventing XSS or SQL injection attacks alone. But hey, what do
I know about SimplePie. Does anyone know what SimplePie actually does within
these 11 seconds?

I use CURL rather than the drupal function because drupal uses fsockets,
fsockets are known to have issues when retrieving from HTTP 1.0 as opposed
to HTTP 1.1, this is solved by sending specific headers, these headers are
ignored on some Linux systems and that may increase the response time by as
much as 5x what CURL would take, normally they would be similar in speed
although CURL is slightly faster. In specific situations like firewall
presence or failure in DNS resolution fsockets may stall for a few minutes.
I read about these issues in a number of articles and I also found them
listed in the comments under fsocketopen. CURL just saved me the hassle and
probably tens of issues :-)

http://www.php.net/manual/en/function.fsockopen.php

Finally, with CURL I can support both HTTP and FTP URLs weather they are
authenticated with a username and password or not. Only con to using CURL is
that it has to be installed on the server machine and enabled in PHP
(weather as a module or compiled).

Post by Alexander Barth
I was just looking at aggregation module and I like how it keeps XML
parsing and parsing into your internal data structure apart.
How did you decide on the data structure that you use in
aggregation?

Going into details will be too long, but there's an API between my module
and the feed-specific handler (where it receives the ready to parse
SimpleXML object from my module) and another API between the feed-specific
handler and my module (where it passes the parsed data). My module chooses
which feed handler to use (RSS,ATOM,etc) based on the user's input
(specifying a term) when he creates a feed node. Thus to parse a new XML
format all you have to do is add a term to the aggregation vocabulary and
add a specific feed-handler file that conforms to the API calls. You can
check the existing feed handlers to see how easy it is to do that.

[Khalid Baheyeldin]

Post by Ashraf Amayreh
I use CURL rather than the drupal function because drupal uses fsockets,
fsockets are known to have issues when retrieving from HTTP 1.0 as opposed
to HTTP 1.1, this is solved by sending specific headers, these headers are
ignored on some Linux systems and that may increase the response time by as
much as 5x what CURL would take, normally they would be similar in speed
although CURL is slightly faster. In specific situations like firewall
presence or failure in DNS resolution fsockets may stall for a few minutes.
I read about these issues in a number of articles and I also found them
listed in the comments under fsocketopen. CURL just saved me the hassle and
probably tens of issues :-)

http://www.php.net/manual/en/function.fsockopen.php

Post by Ashraf Amayreh
Finally, with CURL I can support both HTTP and FTP URLs weather they are
authenticated with a username and password or not. Only con to using CURL is
that it has to be installed on the server machine and enabled in PHP
(weather as a module or compiled).

Ashraf,
===
EDITORS NOTE: HTTP/1.1 uses persistent connection causing this delay. Use
"Connection: close" header to disable it.
===
and
===
Just a note to everyone who is using fsockopen and fread / fgets for a HTTP
connection.

Unless you specify "Connection: Close" in your headers you will need to wait
for the socket to time out before feof($streamPointer) to return true.

This has wasted 2 days of my time, grr!
===

The first example in the page, before the comments, also does the close for
this reason.

This is detailed in the RFC too
http://www.w3.org/Protocols/rfc2616/rfc2616-sec8.html#sec8.1

Are you saying the header Connection: Close is ignored?

Any reason why Drupal should not use HTTP/1.1?

[Morbus Iff]

opinion the second is not sanitization and no aggregator needs to waste
the code and time on trying to handle non-XML or non-standards compliant

It depends entirely on your definition of "aggregator". In your module,
you have only one parser, really - PHP's SimpleXML (or whatever it's
called) that then sends the loaded data structure to the smaller "do
things with it" (ie., RSS20.inc, etc.) subparsers. However, I'd think
that it'd be far more flexible to send the raw strings around /as well/
- then one could support, for example, non-XML documents (or, in my
particular case, I could write scrapers for sites that don't support
feeds [or feeds that contain useful data]) so that I'd be able to hook
into the generic aggregating process. Aggregation != just XML, IMO.

I'd love, for example, to be able to add a "feed" that points to (pff,
making crap outta my ass here) some comic site's "latest comic" HTML,
choose a custom-made parser that expects that HTML, and return the same
data structure that the aggregation API expects as legit. This /is/
aggregation - pulling disparate sources together.

I would be very surprised if I found that SimplePie is wasting 11
seconds out of 12 in preventing XSS or SQL injection attacks alone. But
hey, what do I know about SimplePie. Does anyone know what SimplePie
actually does within these 11 seconds?

SimplePie's set_stupidly_fast is a wrapper around:

$this->enable_order_by_date(false);
$this->remove_div(false);
$this->strip_comments(false);
$this->strip_htmltags(false);
$this->strip_attributes(false);
$this->set_image_handler(false);

None of those are "fix broken XML". I reran the initial test like so:

$feed->set_stupidly_fast(TRUE);
$feed->enable_order_by_date(TRUE);

i.e. first shutting everything off, then enabling one command:

$feed->enable_order_by_date(TRUE) 2 seconds
$feed->remove_div(TRUE) 1 second
$feed->strip_comments(TRUE); 2 seconds
$feed->strip_htmltags(TRUE); 2 seconds
$feed->strip_attributes(TRUE); 2 seconds
$feed->set_image_handler(TRUE); 1 second

[Ashraf Amayreh]

Post by Khalid Baheyeldin
Are you saying the header Connection: Close is ignored?
Any reason why Drupal should not use HTTP/1.1?

Examining the drupal_http_request I found that it actually doesn't use HTTP
1.1 in the first place, so I guess there's no problem in using it at all.
But I would still like to maintain the ability to transparently accept feeds
from HTTP or FTP as well as providing the users the option to access
authenticated URLs. I don't know how widely used these features are, but I'd
hate to remove a feature that could help a user out.

Seems we were guilty in assuming what SimplePie did during these 11 seconds.
Although I still think it's going about it the wrong way. 11 seconds is
suicide. I sanitize against the extracted data, rather than the feed string
as a whole. That's what I presume SimplePie is doing. I wish I could check
it out for myself but my sleep indicators are overloaded.

Morbus' suggestion to pass along the string as whole sounds logical, I'll
see what I can do about that. Although I really had assumed that aggregation
happens from XMLs only so the module would need a considerable amount of
change to accommodate non-XML strings. I'll study the option and see what I
can do. Anyone care to give me a patch for my next birthday? :-P


AA

Post by Khalid Baheyeldin

opinion the second is not sanitization and no aggregator needs to waste
the code and time on trying to handle non-XML or non-standards compliant

It depends entirely on your definition of "aggregator". In your module,
you have only one parser, really - PHP's SimpleXML (or whatever it's
called) that then sends the loaded data structure to the smaller "do
things with it" (ie., RSS20.inc, etc.) subparsers. However, I'd think
that it'd be far more flexible to send the raw strings around /as well/
- then one could support, for example, non-XML documents (or, in my
particular case, I could write scrapers for sites that don't support
feeds [or feeds that contain useful data]) so that I'd be able to hook
into the generic aggregating process. Aggregation != just XML, IMO.
I'd love, for example, to be able to add a "feed" that points to (pff,
making crap outta my ass here) some comic site's "latest comic" HTML,
choose a custom-made parser that expects that HTML, and return the same
data structure that the aggregation API expects as legit. This /is/
aggregation - pulling disparate sources together.

I would be very surprised if I found that SimplePie is wasting 11
seconds out of 12 in preventing XSS or SQL injection attacks alone. But
hey, what do I know about SimplePie. Does anyone know what SimplePie
actually does within these 11 seconds?

$this->enable_order_by_date(false);
$this->remove_div(false);
$this->strip_comments(false);
$this->strip_htmltags(false);
$this->strip_attributes(false);
$this->set_image_handler(false);
$feed->set_stupidly_fast(TRUE);
$feed->enable_order_by_date(TRUE);
$feed->enable_order_by_date(TRUE) 2 seconds
$feed->remove_div(TRUE) 1 second
$feed->strip_comments(TRUE); 2 seconds
$feed->strip_htmltags(TRUE); 2 seconds
$feed->strip_attributes(TRUE); 2 seconds
$feed->set_image_handler(TRUE); 1 second
--
Morbus Iff ( if god is my witness, god must be blind )
Technical: http://www.oreillynet.com/pub/au/779
Culture: http://www.disobey.com/ and http://www.gamegrene.com/
aim: akaMorbus / skype: morbusiff / icq: 2927491 / jabber.org: morbus

[Morbus Iff]

Post by Larry Garfield
RSS is XML. The XML spec explicitly says that invalid files should be discarded, not guessed at the way HTML is. Trying to make sense of a broken RSS feed is explicitly contrary to the spec. So, er, why are we spending so much time trying to sanitize? If it doesn't parse correctly, report an error "this site's RSS feed is f*ed up, tell 'em to fix it". Am I missing something here?

Did you forget Postel's Law? Or the fact that for a feed to be
considered "invalid" (as opposed to "not well-formed") would mean that
Drupal would have to have a validating document type parser?

http://www.w3.org/TR/REC-xml/#dt-valid
http://www.w3.org/TR/REC-xml/#dt-wellformed

And, honestly, telling people that their RSS is malformed and "pls fix,
k thanks" is about as viable as telling someone that their HTML isn't
well formed. It just ain't going to happen.

[Ashraf Amayreh]

Ahhh... so by sanitizing you mean accepting non-fully standards compliant
feeds? If that's what you mean then definitely not. I totally agree with
Larry on this. Why waste my processing time on feeds that are not worth a
penny?

Also, in my issue queue I haven't received one complaint about a feed that
would not parse. Not because I do any sanitization, but I think the reason
for this is because I parse the feed as an XML and look for the main
components, so even if it's not fully conforming it will make do if it has
the main components. If it's beyond hope in being called XML in the first
place or just totally messed up I don't waste the time and the coding effort
that might double or triple my module's code size on it. If that's a type of
sanitization then good for me but it definitely does not affect my module's
performance :-)

Finally, I don't really care to make my module work with everyone and
everything. That's why I have clear PHP 5 and CURL requirements. In my
opinion, if a person is not prepared to get a good environment for his site
or let's something as crappy as a host dictate his platform (let's drop
drupal because our host is using php 3.0 looool) then he's definitely not
from my target audience. PHP 5 sites have become as common as PHP 4 and very
near in price.

VPS hosting is becomming cheaper by the day for anyone who's serious about a
site. Take a look for yourselves. I use this VPS provider to sharpen my LAMP
skills since this provider provides a clean slate installation. I have root
access and I can do anything I want there. The comparison between HTML and
XML feeds is simply flawed IMHO.

http://www.vpslink.com/

Post by Larry Garfield

Post by Larry Garfield
RSS is XML. The XML spec explicitly says that invalid files should be

discarded, not guessed at the way HTML is. Trying to make sense of a broken
RSS feed is explicitly contrary to the spec. So, er, why are we spending so
much time trying to sanitize? If it doesn't parse correctly, report an
error "this site's RSS feed is f*ed up, tell 'em to fix it". Am I missing
something here?
Did you forget Postel's Law? Or the fact that for a feed to be
considered "invalid" (as opposed to "not well-formed") would mean that
Drupal would have to have a validating document type parser?
http://www.w3.org/TR/REC-xml/#dt-valid
http://www.w3.org/TR/REC-xml/#dt-wellformed
And, honestly, telling people that their RSS is malformed and "pls fix,
k thanks" is about as viable as telling someone that their HTML isn't
well formed. It just ain't going to happen.
--
Morbus Iff ( be realistic. demand the impossible. )
Technical: http://www.oreillynet.com/pub/au/779
Culture: http://www.disobey.com/ and http://www.gamegrene.com/
aim: akaMorbus / skype: morbusiff / icq: 2927491 / jabber.org: morbus

[Morbus Iff]

Post by Ashraf Amayreh
Ahhh... so by sanitizing you mean accepting non-fully standards
compliant feeds? If that's what you mean then definitely not. I totally

No, I don't. I mean protecting the users from some idiot inserting XSS
or anything else in his RSS items (knowingly or not). Someone in this
thread said they "trust" (hope?) that the consumer of their module
"trusts" the RSS feeds they consume. That's uh... foolish.

The rest of your email was entirely ignorable.

[Ashraf Amayreh]

Protecting the user from an XSS or SQL injection attack is one thing,
accepting non-standard compliant feeds is another.

Did you waste the time to read a couple of threads before mine or did you
have this reply tailor made a few days ago? The discussion was on weather to
accept non-standard compliant RSS/RDF/ATOM feeds or not sweety. And a little
on weather to push for PHP 5 or not. So why don't you stick to that for a
change?

Post by Morbus Iff

Post by Ashraf Amayreh
Ahhh... so by sanitizing you mean accepting non-fully standards
compliant feeds? If that's what you mean then definitely not. I totally

No, I don't. I mean protecting the users from some idiot inserting XSS
or anything else in his RSS items (knowingly or not). Someone in this
thread said they "trust" (hope?) that the consumer of their module
"trusts" the RSS feeds they consume. That's uh... foolish.
The rest of your email was entirely ignorable.
--
Morbus Iff ( keep out of reach of children )
Technical: http://www.oreillynet.com/pub/au/779
Culture: http://www.disobey.com/ and http://www.gamegrene.com/
aim: akaMorbus / skype: morbusiff / icq: 2927491 / jabber.org: morbus

[Morbus Iff]

Post by Ashraf Amayreh
Did you waste the time to read a couple of threads before mine or did
you have this reply tailor made a few days ago? The discussion was on
weather to accept non-standard compliant RSS/RDF/ATOM feeds or not
sweety. And a little on weather to push for PHP 5 or not. So why don't
you stick to that for a change?

You're kidding, right?

[Ashraf Amayreh]

To put this distraction aside, PHP 5 is as readily available a hosting
solution as PHP 4. Developers who want to use Drupal need to upgrade to PHP
5 rather than Drupal keeping pace with PHP 4.

As to aggregation, as long as a module is not allowing an XSS or an SQL
injection attack. I see no need to try to accept feeds from sites that don't
even put the effort in authoring feeds that meet the absolute minimal feed
standards. Although some may like to aggregate the HTML content in a feed as
is, how we could set in dealing with this case is open for debate. This is
my opinion on the subject.

Post by Morbus Iff

Post by Ashraf Amayreh
Did you waste the time to read a couple of threads before mine or did
you have this reply tailor made a few days ago? The discussion was on
weather to accept non-standard compliant RSS/RDF/ATOM feeds or not
sweety. And a little on weather to push for PHP 5 or not. So why don't
you stick to that for a change?

You're kidding, right?
--
Morbus Iff ( damn you richards! i will have my revenge! )
Technical: http://www.oreillynet.com/pub/au/779
Culture: http://www.disobey.com/ and http://www.gamegrene.com/
aim: akaMorbus / skype: morbusiff / icq: 2927491 / jabber.org: morbus

[Alexander Barth]

Post by Ashraf Amayreh
To put this distraction aside, PHP 5 is as readily available a
hosting solution as PHP 4. Developers who want to use Drupal need to
upgrade to PHP 5 rather than Drupal keeping pace with PHP 4.
As to aggregation, as long as a module is not allowing an XSS or an
SQL injection attack. I see no need to try to accept feeds from
sites that don't even put the effort in authoring feeds that meet
the absolute minimal feed standards.

Somebody will see the need and will come up with a solution for it.
That's where in the past this somebody started his/her own new
aggregation module from scratch :)

Post by Ashraf Amayreh
Although some may like to
aggregate the HTML content in a feed as is, how we could set in
dealing with this case is open for debate. This is my opinion on the subject.

Post by Ashraf Amayreh
Did you waste the time to read a couple of threads before mine or did
you have this reply tailor made a few days ago? The discussion was on
weather to accept non-standard compliant RSS/RDF/ATOM feeds or not
sweety. And a little on weather to push for PHP 5 or not. So why don't
you stick to that for a change?

You're kidding, right?
--
Morbus Iff ( damn you richards! i will have my revenge! )
Technical: http://www.oreillynet.com/pub/au/779
Culture: http://www.disobey.com/ and http://www.gamegrene.com/
aim: akaMorbus / skype: morbusiff / icq: 2927491 / jabber.org: morbus

[Alexander Barth]

Post by Ashraf Amayreh
Ahhh... so by sanitizing you mean accepting non-fully standards
compliant feeds? If that's what you mean then definitely not. I
totally agree with Larry on this. Why waste my processing time on
feeds that are not worth a penny?

I think the very discussion that we are having here is proving that
there are use cases where you want to go for speed rather than
resilience, and there are equally valid use cases where resilience is
more important than speed. I guess we'll find contradicting use cases
like these all the way down Aggregation Lane. We shouldn't get caught
up in discussing which one's the one. The new aggregator should
provide a fundament that does what all aggregators have to do while
leaving all doors for improvement and adaption open.

Post by Ashraf Amayreh
Also, in my issue queue I haven't received one complaint about a
feed that would not parse. Not because I do any sanitization, but I
think the reason for this is because I parse the feed as an XML and
look for the main components, so even if it's not fully conforming
it will make do if it has the main components. If it's beyond hope
in being called XML in the first place or just totally messed up I
don't waste the time and the coding effort that might double or
triple my module's code size on it. If that's a type of sanitization
then good for me but it definitely does not affect my module's
performance :-)

Aside:
I was just looking at aggregation module and I like how it keeps XML
parsing and parsing into your internal data structure apart.
How did you decide on the data structure that you use in
aggregation?

Post by Ashraf Amayreh
Finally, I don't really care to make my module work with everyone
and everything. That's why I have clear PHP 5 and CURL requirements.

Agreed. There won't be a "one single monolithic module serves
all" solution. I do hope though that there will be a "one fundament/API
serves them all" solution.

* We are duplicating efforts between all aggregation modules that have
nothing to do with the special requirements which justify every single
module out there.
* A lot of features that are implemented on just one aggregation module
would be good to see on all aggregation modules.

Alex

[Morbus Iff]

Post by Ashraf Amayreh
Finally, I don't really care to make my module work with everyone
and everything. That's why I have clear PHP 5 and CURL requirements.

Incidentally, I'm curious: why choose curl over drupal_http_request?
What were the pros and cons? The nearest I can see (implemented) was
support for FTP feeds, which is a relatively rare occurrence.

[Morbus Iff]

Post by Boris Mann
Hallelujah. So...if Morbus needs to move, can we start on the
foundations of the one, true pluggable aggregation API of doom,
together, now?

The "problem" is the SOC project. If we help out on that, we potentially
invalidate his application and earnings. If we don't help out on that,
we do what? Work on Feed Parser? SimpleFeed? A brand new module?

[Alexander Barth]

Post by Morbus Iff

Post by Boris Mann
Hallelujah. So...if Morbus needs to move, can we start on the
foundations of the one, true pluggable aggregation API of doom,
together, now?

The "problem" is the SOC project. If we help out on that, we potentially
invalidate his application and earnings. If we don't help out on that,
we do what? Work on Feed Parser? SimpleFeed? A brand new module?

I don't see how we could invalidate Aron's application and earnings by
helping out on his SoC project. Quite the contrary, I think
everybody's input is very important to make sure that it is a success
for the Drupal community.

Alex

[Boris Mann]

Post by Alexander Barth

Post by Morbus Iff

Post by Boris Mann
Hallelujah. So...if Morbus needs to move, can we start on the
foundations of the one, true pluggable aggregation API of doom,
together, now?

The "problem" is the SOC project. If we help out on that, we potentially
invalidate his application and earnings. If we don't help out on that,
we do what? Work on Feed Parser? SimpleFeed? A brand new module?

I don't see how we could invalidate Aron's application and earnings by
helping out on his SoC project. Quite the contrary, I think
everybody's input is very important to make sure that it is a success
for the Drupal community.

Morbus meant actually working on code....which he wants/needs to do to
scratch his own itch.

So: how to work with Aaron's stuff while being able to forge ahead with
producing running code?

[Alexander Barth]

Post by Alexander Barth

Post by Morbus Iff

Post by Boris Mann
Hallelujah. So...if Morbus needs to move, can we start on the
foundations of the one, true pluggable aggregation API of doom,
together, now?

The "problem" is the SOC project. If we help out on that, we potentially
invalidate his application and earnings. If we don't help out on that,
we do what? Work on Feed Parser? SimpleFeed? A brand new module?

I don't see how we could invalidate Aron's application and earnings by
helping out on his SoC project. Quite the contrary, I think
everybody's input is very important to make sure that it is a success
for the Drupal community.
Morbus meant actually working on code....which he wants/needs to do to scratch his own itch.
So: how to work with Aaron's stuff while being able to forge ahead with producing running code?

Thanks for clarification - got that wrong.

We were discussing the same thing here at Development Seed.

I am afraid we'll have to wait until the baby takes shape and there
is actually code to build on. Obviously there are ways to make things
faster by helping Aron with vetting solutions and further on
with testing, patching and feedback. I got some time set aside for
this.

When exactly it makes sense to jump on the new platform depends a lot
on the deadlines of the project at hand. The SoC's deadlines are
published here: http://aggregation.novaak.net/

Alex

[Gerhard Killesreiter]

Post by Larry Garfield
That is precisely the intent of the GoPHP5 effort. :-) Change the
curve so that it's not 20%-30%, but 50%+. That pushes past the
tipping point, and PHP 4 can more quickly go the direction of PHP 3.
To do that, though, requires market pressure. We're trying to be that
market pressure.

Sure, I understand that much. The question is: what if we fail to bend
the curve?
My guess is that all Drupal installations take up at most 0.5% of PHP's
total install base. That doesn't make for a lot of "bending power",
does it? Add a dozen of other systems, and it might add up to 10% of
PHP's total install base?

The power lies in the fact that if the gophp5 effort succeeds, there
will be no "other place" for users to run to. This means that they will
either have to stick with the latest php compatible version or they have
to change hosters in order to find a host that supports php5.

Are we willing to take this bet and to leave most of our users behind
when we failed to significantly bend the curve 6 months down the road?

We won't leave them behind. Drupal 6 will still be available.

Clearly, being able to use PHP5 would help us help our users. Not
providing an upgrade path for 70-80% of our install base seems to be at
odd with that. It certainly has something kamikaze-like.
Will people remember gophp5.org two weeks from now after it fell of the
Digg main page? Just asking.
I'm all for pushing PHP5, but I'm not sure I want to take such an
extreme stance. As I mentioned in my blog post, let's start by making
some features PHP5 only. Like, let's rewrite the aggregator module with
PHP5's simplified XML parser API. That's disruptive too, but at least
we'd be shooting ourselves in the foot, rather than through the head.

If we feel that in Feb 08 PHP5 is still not as widely supported as we'd
like, we could extend the Drupal 6 life cycle.

Cheers,
Gerhard

[Robert Douglass]

This is not the time to be timid. We as developers know deep in our
hearts that PHP5 is the best thing for Drupal. The amazing resonance
that Larry and I have been receiving from the developers on other
projects shows that we're not alone in thinking this.

A PHP5 based Drupal 7 will not cut off the upgrade path of anybody.
Let's be honest about upgrading Drupal. If you follow the instructions
then you have to be able to export your database (and potentially import
it later if you screw things up) and be able to backup the files on your
file system, potentially replacing them if the upgrade fails. These are
coincidentally the same steps that are needed to migrate your Drupal
site to a new web host. There are already plenty of web hosts that
support PHP5. Anybody who doesn't have the skills to migrate to one is
fairly unlikely to ever upgrade their Drupal installation anyway.

We have a chance right now to act in unison with a majority of the
meaningful PHP based FOSS projects. This is an action that has no
precedent. Never before have we all spoken with one voice in a
coordinated way. I feel that failing to take a position of strong
leadership based on what we believe is best for our software is the shot
in the head, not the other way around.

-Robert

Post by Dries Buytaert
Sure, I understand that much. The question is: what if we fail to
bend the curve?

...

Post by Dries Buytaert
I'm all for pushing PHP5, but I'm not sure I want to take such an
extreme stance. As I mentioned in my blog post, let's start by making
some features PHP5 only. Like, let's rewrite the aggregator module
with PHP5's simplified XML parser API. That's disruptive too, but at
least we'd be shooting ourselves in the foot, rather than through the
head.

[Chris Johnson]

I'm in agreement with Robert Douglass. Now is the time to be bold, not timid.

Dries wrote: "My guess is that all Drupal installations take up at most 0.5% of
PHP's total install base." I don't think that measurement (PHP total
install base) is relevant. Far more important is what fraction of the
top 50 most commonly installed PHP packages Drupal is, and what
fraction of the top 50 the other PHP projects who will agree to move
to PHP5 comprise. The long tail of PHP applications is huge -- and
mostly ignorable.

..chrisxj

[Bill Fitzgerald]

As other people have stated, now is a time to lead, not to follow.

There will always be myriad reasons to make the safe choice. Others have
articulated the advantages of using PHP5, and there is widespread
support among the PHP crowd (as articulated by Rasmus at OSCMS) and
among other open source projects. Perhaps leadership in this area will
allow more people to step up and share the responsibility for
articulating why this course makes sense.

And even if nobody else stands up, it's still the right thing to do from
a strictly technical place.

Cheers,

Bill

Post by Chris Johnson
I'm in agreement with Robert Douglass. Now is the time to be bold, not timid.
Dries wrote: "My guess is that all Drupal installations take up at most 0.5% of
PHP's total install base." I don't think that measurement (PHP total
install base) is relevant. Far more important is what fraction of the
top 50 most commonly installed PHP packages Drupal is, and what
fraction of the top 50 the other PHP projects who will agree to move
to PHP5 comprise. The long tail of PHP applications is huge -- and
mostly ignorable.
..chrisxj

[Vivek Puri]

Post by Karoly Negyesi
The CentOS team is pleased to announce the
availability of CentOS 5.0. Major changes in CentOS
These updated software versions: Apache-2.2,
php-5.1.6
... RHEL 5 has moved all the way up to PHP 5.1.6
from RHEL 4's mere PHP 4.3.9
We are getting there.

Now there's another catch there. Its far easier to
upgrade the PHP/Apache than OS. Lets see:
1 ) Most people manage their servers remotely even
people who pay for dedicated servers. So upgrading PHP
means if it goes wrong I am still logged on to my
server and can revert back. If I upgrade my OS and it
goes wrong I am stuck and it goes out of my control as
most people cant install OS remotely.
2 ) Most production server will be Ok with upgrading
PHP/Apache as its still application than upgrading the
OS. You can always have php4 & php5 installed on same
machine.
3 ) RHEL5/CentOS5 have recently been launched and
there are issue with stability. So most people will
wait till its stabilized before upgrading. So as I
understand new servers may use RHEL5/CentOS5 older
servers are in no rush to upgrade.

Now even if RHEL5 becomes prominent platform we are
again stuck with PHP 5.1 , while we are planning to
target PHP 5.2 .

So Jonathan is right that we are moving at pace of
RedHat. Although I am in full favor of move to php
5.2 soon but hosting companies, which rely on "yum
update" for maintaining their servers, wont come on
board esily.





___________________________________________________________________________________
You snooze, you lose. Get messages ASAP with AutoCheck
in the all-new Yahoo! Mail Beta.
http://advision.webevents.yahoo.com/mailbeta/newmail_html.html

[Larry Garfield]

Post by Vivek Puri
Now there's another catch there. Its far easier to
1 ) Most people manage their servers remotely even
people who pay for dedicated servers. So upgrading PHP
means if it goes wrong I am still logged on to my
server and can revert back. If I upgrade my OS and it
goes wrong I am stuck and it goes out of my control as
most people cant install OS remotely.
2 ) Most production server will be Ok with upgrading
PHP/Apache as its still application than upgrading the
OS. You can always have php4 & php5 installed on same
machine.
3 ) RHEL5/CentOS5 have recently been launched and
there are issue with stability. So most people will
wait till its stabilized before upgrading. So as I
understand new servers may use RHEL5/CentOS5 older
servers are in no rush to upgrade.
Now even if RHEL5 becomes prominent platform we are
again stuck with PHP 5.1 , while we are planning to
target PHP 5.2 .
So Jonathan is right that we are moving at pace of
RedHat. Although I am in full favor of move to php
5.2 soon but hosting companies, which rely on "yum
update" for maintaining their servers, wont come on
board esily.

OS upgrades aren't really a concern for us. We're just interested in the PHP
version.

Although Red Hat's latest version ships 5.1.6 by default, that version
apparently had some nasty bugs anyway from what I have heard. I think the
most telling data, though, is this:

http://www.nexen.net/chiffres_cles/phpversion/16987-php_stats_evolution_for_april_2007.php

See the last chart especially. A month ago, 5.2 was already within spitting
distance of surpassing 5.1 installations, and 5.1 is in slight decline. By
next February, at that rate it should have a healthy chunk of the market
already, even without us pushing. Not enough that we could abandon PHP 4
without pushing, but enough that it's a safe target, I believe.
--
Larry Garfield AIM: LOLG42
***@garfieldtech.com ICQ: 6817012

"If nature has made any one thing less susceptible than all others of
exclusive property, it is the action of the thinking power called an idea,
which an individual may exclusively possess as long as he keeps it to
himself; but the moment it is divulged, it forces itself into the possession
of every one, and the receiver cannot dispossess himself of it." -- Thomas
Jefferson

[Vivek Puri]

Post by Larry Garfield
OS upgrades aren't really a concern for us. We're
just interested in the PHP
version.

true but I was responding to how PHP versions are tied
to version of OS.



____________________________________________________________________________________
Sick sense of humor? Visit Yahoo! TV's
Comedy with an Edge to see what's on, when.
http://tv.yahoo.com/collections/222

[Robert Douglass]

I suspect a lot would change very quickly if web hosts started to
perceive that they were losing customers to other webhosts who ran
modern PHP versions.

Post by Vivek Puri

Post by Larry Garfield
OS upgrades aren't really a concern for us. We're
just interested in the PHP
version.

true but I was responding to how PHP versions are tied
to version of OS.
____________________________________________________________________________________
Sick sense of humor? Visit Yahoo! TV's
Comedy with an Edge to see what's on, when.
http://tv.yahoo.com/collections/222

[Steven Peck]

Actually, it almost doesn't matter. IF a substantial portion of the larger
php based projects move next year, it doesn't mean that their products that
have been developed to work up until that point will cease working.

It just means that to do anything moving forward people will need to
upgrade.
This really isn't different then what happens now.

Besides, this strategy has worked successfully for Microsoft for years. It
will work for the Open Source community just as well.

Announce a date, stick to it, move forward. People who don't upgrade now,
are the ones who won't upgrade then.

-sp

-----Original Message-----
Sent: Thursday, June 07, 2007 12:34 AM
Subject: Re: [development] Go PHP 5, Go!
I suspect a lot would change very quickly if web hosts started to
perceive that they were losing customers to other webhosts who ran
modern PHP versions.

Post by Vivek Puri

Post by Larry Garfield
OS upgrades aren't really a concern for us. We're
just interested in the PHP
version.

true but I was responding to how PHP versions are tied
to version of OS.

_______________________________________________________________________
_____________

Post by Vivek Puri
Sick sense of humor? Visit Yahoo! TV's
Comedy with an Edge to see what's on, when.
http://tv.yahoo.com/collections/222

[Vivek Puri]

Post by Steven Peck
Announce a date, stick to it, move forward. People
who don't upgrade now,
are the ones who won't upgrade then.

I definitely agree with that , just trying to be
devil's advocate here ;)
In a scenario where Drupal moves to PHP 5.2 on Feb
2008, lets say its Drupal 7 then. Will we continue to
support Drupal 5.X & Dupal 6.X on PHP 4.X ? Or after
the cutoff date all patches and upgrade will be only
valid on PHP 5.2 ?






____________________________________________________________________________________
Got a little couch potato?
Check out fun summer activities for kids.
http://search.yahoo.com/search?fr=oni_on_mail&p=summer+activities+for+kids&cs=bz

[David Strauss]

Post by Vivek Puri
I definitely agree with that , just trying to be
devil's advocate here ;)
In a scenario where Drupal moves to PHP 5.2 on Feb
2008, lets say its Drupal 7 then. Will we continue to
support Drupal 5.X & Dupal 6.X on PHP 4.X ? Or after
the cutoff date all patches and upgrade will be only
valid on PHP 5.2 ?

I think we'll continue to maintain the PHP 4-compatible version as PHP
4-compatible until the release itself becomes obsolete.

[Larry Garfield]

Post by David Strauss

Post by Vivek Puri
I definitely agree with that , just trying to be
devil's advocate here ;)
In a scenario where Drupal moves to PHP 5.2 on Feb
2008, lets say its Drupal 7 then. Will we continue to
support Drupal 5.X & Dupal 6.X on PHP 4.X ? Or after
the cutoff date all patches and upgrade will be only
valid on PHP 5.2 ?

I think we'll continue to maintain the PHP 4-compatible version as PHP
4-compatible until the release itself becomes obsolete.

Yes, the agreement is that new feature releases post that date will assume
5.2. Projects are free to maintain support for older releases however they
like for as long as they like. Just like Drupal 5 is/will be maintained as
MySQL 3 compatible for as long as Drupal 5 is maintained even though Drupal 6
will require MySQL 4.1.
--
Larry Garfield AIM: LOLG42
***@garfieldtech.com ICQ: 6817012

"If nature has made any one thing less susceptible than all others of
exclusive property, it is the action of the thinking power called an idea,
which an individual may exclusively possess as long as he keeps it to
himself; but the moment it is divulged, it forces itself into the possession
of every one, and the receiver cannot dispossess himself of it." -- Thomas
Jefferson

[Jakub Suchy]

Post by Robert Douglass
I suspect a lot would change very quickly if web hosts started to
perceive that they were losing customers to other webhosts who ran
modern PHP versions.

This is OT, but I think that Red Hat has made a mistake by shipping
5.1.6 with RHEL5. RHEL5.1 is about to come in Q3, let's push on Red Hat
to include 5.2 in separate packages (they can't upgrade now)

And yes, i know this would be hard...

Jakub

[Larry Garfield]

Post by Jakub Suchy

Post by Robert Douglass
I suspect a lot would change very quickly if web hosts started to
perceive that they were losing customers to other webhosts who ran
modern PHP versions.

This is OT, but I think that Red Hat has made a mistake by shipping
5.1.6 with RHEL5. RHEL5.1 is about to come in Q3, let's push on Red Hat
to include 5.2 in separate packages (they can't upgrade now)
And yes, i know this would be hard...
Jakub

Hopefully seeing an effort such as this, which should go public well before
Q3, will be a notice to Red Hat as well as to web hosts. Of course, RH is
known for upping the version of software they include and patching the hell
out of it but not actually changing the version number, so who knows what it
it is they actually run. :-)

It certainly seems like everyone is in favor of this idea. We need to decide
how we're going to formally "sign on". Dries, that's your call to decide how
we go about it.
--
Larry Garfield AIM: LOLG42
***@garfieldtech.com ICQ: 6817012

"If nature has made any one thing less susceptible than all others of
exclusive property, it is the action of the thinking power called an idea,
which an individual may exclusively possess as long as he keeps it to
himself; but the moment it is divulged, it forces itself into the possession
of every one, and the receiver cannot dispossess himself of it." -- Thomas
Jefferson

[Robert Douglass]

Post by Larry Garfield
It certainly seems like everyone is in favor of this idea. We need to decide
how we're going to formally "sign on". Dries, that's your call to decide how
we go about it.

That's not so hard... we just need a page on Drupal.org that says we're
going to do it. This should go live when the website goes live. Having a
public page on the project website is the requirement that all projects
have to meet in order to be listed as having joined the effort.

[Boris Mann]

Post by Vivek Puri

Post by Larry Garfield
OS upgrades aren't really a concern for us. We're
just interested in the PHP
version.

true but I was responding to how PHP versions are tied
to version of OS.

CentOS has long had the CentOS Plus repository (see
http://www.jimohalloran.com/2005/08/30/php-5-for-centos-41/). PHP5 has
been available for quite some time.

One of the pushes of this process could be banding together and
promoting/recommending a certain minimum version. Getting the CentOS
Plus maintainers on board to support whatever version of PHP 5.2.x we
think is best would really help with adoption...

[Peter Wolanin]

Drupal has a number of custom fixes (e.g. function drupal_clone()) to
deal with differences between PHP 4 and PHP 5 - it would be nice to
not have to worry about such things.

-Peter

Post by Sean Robertson
Are there currently that many PHP4 apps that won't run on PHP5? I
thought Drupal already does. Why can't ISPs upgrade without breaking
older apps?
I am a huge fan of this idea overall, BTW. I think migrating apps to
PHP5 will start forcing ISPs to at least provide the option to upgrade
since they'll start losing customers if they don't. At my company, we
own our own servers so it'd be relatively easy for us to upgrade.

Post by Larry Garfield
This is a follow-up to the PHP 5 thread from a week or two ago. It looks like
some momentum is building. Ken Rickard, Robert Douglas, and I have been
talking with some of the Jooma folks, and have a working draft of the "core
statement and justification". That is, what the goal is and why it is.
Joomla's development team is discussing the matter and is leaning yes. Based
on the earlier thread here I am hoping that there isn't much objection to
Drupal participating in the "Go PHP5" effort. :-) So far Joomla is leaning
yes, CakePHP is interested, and I had a positive first response from Typo3.
Robert Douglas has volunteered himself to setup a web site for it.
I'm not sure how Dries wants to handle the question of Drupal's participation
(by vote, by consensus, or by fiat). Dries?
Anyway, here's the working statement. Consider this an official
recommendation that Drupal commit to participating in this effort.
------------------------------------
PHP 4 has served the web developer community for seven years now, and served
it well. However, it also shows its age. Most of PHP 4's shortcomings have
been addressed by PHP 5, released three years ago, but the transition from
PHP 4 to PHP 5 has been slow for a number of reasons.
PHP developers cannot leverage PHP 5's full potential without dropping support
for PHP 4, but PHP 4 is still installed on a majority of shared web hosts and
users would then be forced to switch to a different application. Web hosts
cannot upgrade their servers to PHP 5 without making it impossible for their
users to run PHP 4-targeted web apps, and have no incentive to go to the
effort of testing and deploying PHP 5 while most web apps are still
compatible with PHP 4 and the PHP development team still provides maintenance
support for PHP 4. The PHP development team, of course, can't drop
maintenance support for PHP 4 while most web hosts still run PHP 4.
It is a dangerous cycle, and one that needs to be broken. The open source PHP
developer community has decided that it is indeed now time to move forward,
together. Therefore, the listed open source PHP projects have all agreed
that effective 5 February 2008, any new feature release will have a minimum
required PHP version no older than PHP 5.2.0. It is our believe that this
will allow web hosts a reason to upgrade and the PHP development team the
ability to retire PHP 4 and focus efforts on PHP 5 and the forthcoming PHP 6,
all without penalizing any existing project for being "first out of the
gate".
------------------------------------
- I chose the date because I figure that will be 7-8 months after we
officially announce this thing, which I believe should be sufficient time for
web hosts. It also comes out to 5/2/2008 (using European convention), and I
just like inside references like that. :-)
- This does not preclude any project from moving before the deadline date, or
from supporting older versions for however long they wish to. That's up to
each project.
- PHP 5.2 is already the most widely installed version of PHP 5, based on the
latest published stats. I know at least two web hosts I work with that
either have jumped or are in the process of jumping from PHP 4 straight to
PHP 5.2. By the target date it will have been out for nearly a year and a
half. It also adds a number of new and useful core features (filter_input,
json, a stable PDO, etc.). It's a good version to target.
Thoughts? Comments? Support? Rotten tomatoes?

--
Sean Robertson
Web Developer
NGP Software, Inc.
[phone]
http://www.ngpsoftware.com

[Larry Garfield]

I don't know a number, but that's one of the fears that there may be. A lot of it could be custom code that someone wrote 6 years ago to PHP 4 standards that will have subtle problems in PHP 5, and they're worried about breaking that. It's the same reason so many web hosts still run register_globals On, even though it's a glaring security hole; they have thousands of clients, and if even 50 of them have code that would stop working without register_globals that's enough to make them shy away from enforcing it.

Any distributed PHP applications or frameworks that are not PHP 5 compatible at this point I would consider officially abandoned, honestly.

--Larry Garfield

Post by Sean Robertson
Are there currently that many PHP4 apps that won't run on PHP5? I
thought Drupal already does. Why can't ISPs upgrade without breaking
older apps?
I am a huge fan of this idea overall, BTW. I think migrating apps to
PHP5 will start forcing ISPs to at least provide the option to upgrade
since they'll start losing customers if they don't. At my company, we
own our own servers so it'd be relatively easy for us to upgrade.

Post by Larry Garfield
This is a follow-up to the PHP 5 thread from a week or two ago. It

looks like

Post by Larry Garfield
some momentum is building. Ken Rickard, Robert Douglas, and I have been
talking with some of the Jooma folks, and have a working draft of the

"core

Post by Larry Garfield
statement and justification". That is, what the goal is and why it is.
Joomla's development team is discussing the matter and is leaning yes.

Based

Post by Larry Garfield
on the earlier thread here I am hoping that there isn't much objection

to

Post by Larry Garfield
Drupal participating in the "Go PHP5" effort. :-) So far Joomla is

leaning

Post by Larry Garfield
yes, CakePHP is interested, and I had a positive first response from

Typo3.

Post by Larry Garfield
Robert Douglas has volunteered himself to setup a web site for it.
I'm not sure how Dries wants to handle the question of Drupal's

participation

Post by Larry Garfield
(by vote, by consensus, or by fiat). Dries?
Anyway, here's the working statement. Consider this an official
recommendation that Drupal commit to participating in this effort.
------------------------------------
PHP 4 has served the web developer community for seven years now, and

served

Post by Larry Garfield
it well. However, it also shows its age. Most of PHP 4's shortcomings

have

Post by Larry Garfield
been addressed by PHP 5, released three years ago, but the transition

from

Post by Larry Garfield
PHP 4 to PHP 5 has been slow for a number of reasons.
PHP developers cannot leverage PHP 5's full potential without dropping

support

Post by Larry Garfield
for PHP 4, but PHP 4 is still installed on a majority of shared web

hosts and

Post by Larry Garfield
users would then be forced to switch to a different application. Web

hosts

Post by Larry Garfield
cannot upgrade their servers to PHP 5 without making it impossible for

their

Post by Larry Garfield
users to run PHP 4-targeted web apps, and have no incentive to go to the
effort of testing and deploying PHP 5 while most web apps are still
compatible with PHP 4 and the PHP development team still provides

maintenance

Post by Larry Garfield
support for PHP 4. The PHP development team, of course, can't drop
maintenance support for PHP 4 while most web hosts still run PHP 4.
It is a dangerous cycle, and one that needs to be broken. The open

source PHP

Post by Larry Garfield
developer community has decided that it is indeed now time to move

forward,

Post by Larry Garfield
together. Therefore, the listed open source PHP projects have all

agreed

Post by Larry Garfield
that effective 5 February 2008, any new feature release will have a

minimum

Post by Larry Garfield
required PHP version no older than PHP 5.2.0. It is our believe that

this

Post by Larry Garfield
will allow web hosts a reason to upgrade and the PHP development team

the

Post by Larry Garfield
ability to retire PHP 4 and focus efforts on PHP 5 and the forthcoming

PHP 6,

Post by Larry Garfield
all without penalizing any existing project for being "first out of the
gate".
------------------------------------
- I chose the date because I figure that will be 7-8 months after we
officially announce this thing, which I believe should be sufficient

time for

Post by Larry Garfield
web hosts. It also comes out to 5/2/2008 (using European convention),

and I

Post by Larry Garfield
just like inside references like that. :-)
- This does not preclude any project from moving before the deadline

date, or

Post by Larry Garfield
from supporting older versions for however long they wish to. That's up

to

Post by Larry Garfield
each project.
- PHP 5.2 is already the most widely installed version of PHP 5, based

on the

Post by Larry Garfield
latest published stats. I know at least two web hosts I work with that
either have jumped or are in the process of jumping from PHP 4 straight

to

Post by Larry Garfield
PHP 5.2. By the target date it will have been out for nearly a year and

a

Post by Larry Garfield
half. It also adds a number of new and useful core features

(filter_input,

Post by Larry Garfield
json, a stable PDO, etc.). It's a good version to target.
Thoughts? Comments? Support? Rotten tomatoes?

--
Sean Robertson
Web Developer
NGP Software, Inc.
[phone]
http://www.ngpsoftware.com

[Karthik]

Post by Larry Garfield
This is a follow-up to the PHP 5 thread from a week or two ago. It looks like
some momentum is building. Ken Rickard, Robert Douglas, and I have been
talking with some of the Jooma folks, and have a working draft of the "core
statement and justification". That is, what the goal is and why it is.
Joomla's development team is discussing the matter and is leaning yes. Based
on the earlier thread here I am hoping that there isn't much objection to
Drupal participating in the "Go PHP5" effort. :-) So far Joomla is leaning
yes, CakePHP is interested, and I had a positive first response from Typo3.
Robert Douglas has volunteered himself to setup a web site for it.

Excellent work and a truly noble cause :) Are there plans to perhaps
garner more community support via /. or digg? Perhaps a website might
be in order.

If there's any way I can help out, please let me know.
-K

[Robert Douglass]

Post by Karthik
Excellent work and a truly noble cause :) Are there plans to perhaps
garner more community support via /. or digg? Perhaps a website might
be in order.
If there's any way I can help out, please let me know.
-K

A (Drupal based) website is in the works. Thanks for the offer =) There
will be plenty of /. and digg action going, when the time comes.

[Sean Robertson]

Need any help with the design? I do graphic design (web and print) for
a living. Here are some samples of my work:

http://www.murtha.org
http://www.vetsformurtha.org
http://www.johnlewisforcongress.com
http://www.ksdp.org
http://www.johnconyers.com
http://www.ngpsoftware.com/lefty08

All of those are Drupal sites, BTW.

Post by Robert Douglass

Post by Karthik
Excellent work and a truly noble cause :) Are there plans to perhaps
garner more community support via /. or digg? Perhaps a website might
be in order.
If there's any way I can help out, please let me know.
-K

A (Drupal based) website is in the works. Thanks for the offer =) There
will be plenty of /. and digg action going, when the time comes.

[Michelle Cox]

Post by Larry Garfield
Thanks for the catch. I wasn't
sure which sounded less pushy toward hosts.
At least some of them, I think/hope, will be just as in favor of breaking
the
deadlock as we are. We can tweak the exact wording as needed.

I don't think "give them a reason" is pushy... Maybe "provide them a reason"
would be better? "Allow them a reason" just doesn't make sense, IMO. But, as
you said, you can tweak it later. :)

Michelle

[Chad Kieffer]

I applaud this effort and submit the following for consideration.

Be careful in questioning the "why" question with ISPs, I would not
lump them into a single "lazy" category. There are many, including
Dreamhost, that provide customers a choice between PHP 4 or 5.
Perhaps constructive suggestions on how ISPs/Hosts can configure
their hosting environment to support the cause and why they should
(security, performance, etc.).

It's been mentioned by others in this thread already, the problem
isn't FOSS compatibility with PHP 5, it's custom app compatibility
with PHP 5. If I'm a developer and am not going to get paid to
upgrade my customer's PHP 4 site, why should I even bother? Here lies
the biggest challenge of this effort, IMHO. I think providing
information on upgrading apps from 4 to 5 would be helpful, but
probably won't yield great results.

Practical transition plans are what needed. Perhaps allowing visitors
to share their migration/upgrade experiences would be helpful.

- Chad Kieffer

Post by Larry Garfield
This is a follow-up to the PHP 5 thread from a week or two ago. It looks like
some momentum is building. Ken Rickard, Robert Douglas, and I have been
talking with some of the Jooma folks, and have a working draft of the "core
statement and justification". That is, what the goal is and why it is.
Joomla's development team is discussing the matter and is leaning yes. Based
on the earlier thread here I am hoping that there isn't much
objection to
Drupal participating in the "Go PHP5" effort. :-) So far Joomla is leaning
yes, CakePHP is interested, and I had a positive first response from Typo3.
Robert Douglas has volunteered himself to setup a web site for it.
I'm not sure how Dries wants to handle the question of Drupal's participation
(by vote, by consensus, or by fiat). Dries?
Anyway, here's the working statement. Consider this an official
recommendation that Drupal commit to participating in this effort.
------------------------------------
PHP 4 has served the web developer community for seven years now, and served
it well. However, it also shows its age. Most of PHP 4's
shortcomings have
been addressed by PHP 5, released three years ago, but the
transition from
PHP 4 to PHP 5 has been slow for a number of reasons.
PHP developers cannot leverage PHP 5's full potential without
dropping support
for PHP 4, but PHP 4 is still installed on a majority of shared web hosts and
users would then be forced to switch to a different application.
Web hosts
cannot upgrade their servers to PHP 5 without making it impossible for their
users to run PHP 4-targeted web apps, and have no incentive to go to the
effort of testing and deploying PHP 5 while most web apps are still
compatible with PHP 4 and the PHP development team still provides maintenance
support for PHP 4. The PHP development team, of course, can't drop
maintenance support for PHP 4 while most web hosts still run PHP 4.
It is a dangerous cycle, and one that needs to be broken. The open
source PHP
developer community has decided that it is indeed now time to move forward,
together. Therefore, the listed open source PHP projects have all
agreed
that effective 5 February 2008, any new feature release will have a minimum
required PHP version no older than PHP 5.2.0. It is our believe
that this
will allow web hosts a reason to upgrade and the PHP development team the
ability to retire PHP 4 and focus efforts on PHP 5 and the
forthcoming PHP 6,
all without penalizing any existing project for being "first out of the
gate".
------------------------------------
- I chose the date because I figure that will be 7-8 months after we
officially announce this thing, which I believe should be
sufficient time for
web hosts. It also comes out to 5/2/2008 (using European
convention), and I
just like inside references like that. :-)
- This does not preclude any project from moving before the
deadline date, or
from supporting older versions for however long they wish to.
That's up to
each project.
- PHP 5.2 is already the most widely installed version of PHP 5, based on the
latest published stats. I know at least two web hosts I work with that
either have jumped or are in the process of jumping from PHP 4
straight to
PHP 5.2. By the target date it will have been out for nearly a year and a
half. It also adds a number of new and useful core features
(filter_input,
json, a stable PDO, etc.). It's a good version to target.
Thoughts? Comments? Support? Rotten tomatoes?

[FGM]

If this concerted effort means anything, it is precisely that an
inflexion in the trends will be caused, causing discontinuity and a
new segment in the trend curve instead of just continuing it.
Disruptive events...

---- Original Message ----
From: ***@gmail.com
To: ***@drupal.org
Subject: Re: [development] Go PHP 5, Go!
Date: Mon, 18 Jun 2007 11:03:28 +0200

Post by Dries Buytaert

Post by Gerhard Killesreiter
If we proceed like this, there will be no way we could revert to
support
PHP 4 in February 2008.

If I can believe my own projections (http://buytaert.net/php-is-dead-
long-live-php), PHP5 will have a 30% adoption rate by February 2008,
rather than the current 20%. Maybe February 2008 is a little too
early?
--
Dries Buytaert :: http://www.buytaert.net/