Various Topics on Lisp

..r.. ..p..

Ever since Dan B. mentioned it on his diary, I've been really wondering what
some experienced Lispers thought of the PHP language. Wait. Take that back.
I know what Lispers will say about PHP. It's a joke. Not worth mentioning.
As Dan said, (paraphrasing) "it doesn't even have a good Emacs mode".
Except...

I have a special interest in PHP. It's been proved to be able to reliably
handle the traffic for very large sites. I'm programming a web site in
lisp, but I keep having this nagging worry that I'm going to regret
choosing the language if/when my site's membership grows to be 5,000-50,000
or so. Will lisp be able to handle that sort of load without me having to
know how to hack a webserver, or debug socket issues, or something esoteric
like that, or depend on project maintainers to do them? I guess it would
re***ure me to see a really big site that runs mainly on lisp.

Viaweb doesn't count. I'm probably not nearly as smart as Paul Graham.
Orbitz doesn't count. It's just a backend. It deals with entirely different
issues that something like Cliki does. And Cliki doesn't have (as far as I
know) lots and lots of traffic.

PHP does handle those kind of loads (though it does need things like
precompilers and such) and does it regularly. (According to Troutgirl at
Friendster, it does it better than java.) Lisp is sort of a question mark
here for me, though. I've heard a lot about Lisp the Language's wonders,
but not about /implementations'/ performance and reliability in these
situations.

I think this is a special case of the more general "Microsoft anxiety" that
particularly affects developers, like me, that have grown up using
Microsoft software--OS, dev. tools, etc. The perception is, with MS, /or/
any other widely used platform, that even if something doesn't work, it's
either /your/ fault, or a hundred people have already had the problem and
have posted workarounds on the web, so you're never going to be screwed (at
least by the software itself). But, in me, it's a very deep-rooted
attitude, and it manifests itself as a fear of doing anything substantial
in a language that isn't already doing something substantial is substantial
cir***stances. I'm disappointed in myself, because Lisp is the greatest,
and I'd love to have a site running in lisp, and I don't think I really
have any /rational/ fears concerning it.

Hmm.

Chris Capel
--
"Even my anxieties have anxieties." --Charlie Brown

..le.. ..rtezaog..

>>>>> "CC" == Chris Capel <ch.ris@iba.nktech.net> writes:

CC> .... I'm
CC> programming a web site in lisp, but I keep having this nagging
CC> worry that I'm going to regret choosing the language if/when
CC> my site's membership grows to be 5,000-50,000 or so. Will lisp
CC> be able to handle that sort of load without me having to know
CC> how to hack a webserver, or debug socket issues, or something
CC> esoteric like that, or depend on project maintainers to do
CC> them? I guess it would re***ure me to see a really big site
CC> that runs mainly on lisp. ...

Even if that were the case, I'd think that you'd test it under load that
you artificially created and see how it does? Or am I missing the point
here?

I understand the unease, but it seems possible -- if not easy -- to
alleviate it. I'd start with something simple like apache bench (man
ab if you have apache installed), and if your server can handle that
move on to app-specific floggers that you'd create.

cheers,

BM

.... ..ingst..

On Sun, 19 Sep 2004 12:58:32 -0500, Chris Capel <ch.ris@iba.nktech.net>
wrote:

> Ever since Dan B. mentioned it on his diary, I've been really wondering
> what
> some experienced Lispers thought of the PHP language. Wait. Take that
> back.
> I know what Lispers will say about PHP. It's a joke. Not worth
> mentioning.
> I have a special interest in PHP. It's been proved to be able to reliably
> handle the traffic for very large sites. I'm programming a web site in
> lisp, but I keep having this nagging worry that I'm going to regret
> choosing the language if/when my site's membership grows to be
> 5,000-50,000
> or so. Will lisp be able to handle that sort of load without me having to
> know how to hack a webserver, or debug socket issues, or something
> esoteric
> like that, or depend on project maintainers to do them? I guess it would
> re***ure me to see a really big site that runs mainly on lisp.

Been a while since I looked but I believe Library of Congress uses
the MIT Lisp HTTP server. They easily take loads of 5000-50000 and
can perform advanced queries on a large database. http://www.loc.gov
--
Using M2, Opera's revolutionary e-mail client: http://www.opera.com/m2/

..a.. ....

"John Thingstad" <john.thingstad@chello.no> wrote:

> Been a while since I looked but I believe Library of Congress uses
> the MIT Lisp HTTP server. They easily take loads of 5000-50000 and
> can perform advanced queries on a large database. http://www.loc.gov

the web server says "web" as server name in the HTTP header of the main
page and the search engine returns "Inktomi Search 4.3.1".

I'm interested, too, in Lisp based web servers, because it would be nice
to generate interactivly dynamic web pages with Lisp code (I've tried it
with Portable AllegroServe and clsql under CMU CL, see
http://groups.google.de/groups?selm=....netcologne.de ),
but I didn't found an example of a big traffic site and not very much
Lisp web servers at all. For example a Lisp web server page at
http://www.ai.mit.edu/projects/iiip/...home-page.html lists
many links (last updated in december 2003) which should run a Lisp
server, but I've tested the first 5 links and no link worked with
CL-HTTP. There were only 3 links which worked, but returned Apache as
the server name. I hope the reason is not that the servers crashed. With
Apache I can start the server and even if some worker threads crashs,
they are started again and it runs for months without maintanance.

Simulating traffic is a good idea, but would be nice to see a high
traffic or long running production system running a Lisp webserver.

--
Frank Buß, fb@frank-buss.de
http://www.frank-buss.de, http://www.it4-systems.de

.... ..ttya..

"Chris Capel" <ch.ris@iba.nktech.net> wrote

> I have a special interest in PHP. It's been proved to be able to reliably
> handle the traffic for very large sites. I'm programming a web site in
> lisp, but I keep having this nagging worry that I'm going to regret
> choosing the language if/when my site's membership grows to be
5,000-50,000
> or so. Will lisp be able to handle that sort of load without me having to
> know how to hack a webserver, or debug socket issues, or something
esoteric
> like that, or depend on project maintainers to do them? I guess it would
> re***ure me to see a really big site that runs mainly on lisp.

With Lispworks + mod_lisp 2.38 + my framework, I get this on my server (Dual
P4 2.8GHz):

ab -n 1000 -c 20 http://www.fractalconcept.com/asp/debug

This is ApacheBench, Version 1.3d <$Revision: 1.73 $> apache-1.3
Benchmarking www.fractalconcept.com (be patient)
[...]
Finished 1000 requests
Server Software: Apache/1.3.26
Server Hostname: www.fractalconcept.com
Server Port: 80

Do***ent Path: /asp/debug
Do***ent Length: 617 bytes

Concurrency Level: 20
Time taken for tests: 2.248 seconds
Complete requests: 1000
Failed requests: 0
Broken pipe errors: 0
Non-2xx responses: 1000
Total transferred: 817000 bytes
HTML transferred: 617000 bytes
Requests per second: 444.84 [#/sec] (mean)
Time per request: 44.96 [ms] (mean)
Time per request: 2.25 [ms] (mean, across all concurrent requests)
Transfer rate: 363.43 [Kbytes/sec] received

Connnection Times (ms)
min mean[+/-sd] median max
Connect: 0 14 6.4 14 38
Processing: 8 30 7.9 29 62
Waiting: 6 29 7.9 28 61
Total: 8 44 8.1 43 77

Percentage of the requests served within a certain time (ms)
50% 43
66% 46
75% 49
80% 50
90% 55
95% 58
98% 65
99% 69
100% 77 (last request)

This URL: http://www.fractalconcept.com/asp/debug generates a debug page
with a table that shows the different apache variable values. Here is the
lisp code:

(defun http-debug-request (request)
(log-message (format nil "http-debug-request ~%"))
(with-output-to-request (request)
(html::html-to-stream *request-stream*
"<!DOCTYPE html PUBLIC \"-//W3C//DTD HTML 4.01 Transitional//EN\">"
(:html (:head)
(:body
((:table "bgcolor" "#c0c0c0")
((:tr "bgcolor" "yellow") (:th "Key") (:th "Value"))
((:tr "bgcolor" "#F0F0c0")
(:td "apache-nb-use-socket")(:td *apache-nb-use-socket*))
(loop for (key . value) in (command request)
do (html::html ((:tr "bgcolor" "#F0F0c0")
(:td key)(:td value))))))))
(push-header "Log-Error" (format nil "request ~s not processed." (url
request)) request)))

If I remove the text generation to send a constant string as a reply, the
times are the same, so the bottleneck is probably not the html generation
part.

As I have never written any web application with anything else than Lisp
(like Java, PHP, etc.), I don't know if it is good or bad but I would be
interested to know.

Anyway I'm sure you can support a lot of simultaneous users.

Marc

.... ..meni..

Your concerns seem to center around issues dealing with heavy load.
I think one should be concerned, but concerned in a rational way.
There is more to be concerned about than Lisp when it comes to heavy
loads. There are things like:

1) The quality of your hardware. Memory, Hard Drives, Back up systems,
Power Supplies, UPS, ...
Poor hardware will fail under load, and the worst part is that you will
probably not know why.

2) The quality of your OS and its drivers. Some operating systems will
just fail under heavy utilization.

3) The interaction between your OS and hardware. (And unless you
are willing to shell out some serious dough it is likely you
will have or develop problems). For a relatively inexpensive system
my recommendation would be FreeBSD and some good PC hardware, good
motherboard (ASUS, Gigabyte), memory cooling, and other quality parts.
Then use some good Lisp (CMUCL runs on FreeBSD).

4) The design of your application. These are bugs you introduce
by making mistakes or having incomplete code.

Many of the Lisp implementations have been around for a least a
decade, used by people in heavy load situations (Viaweb and Orbitz)
for a few. It seems to take at least a decade to shake bugs out
of any significant application (this includes OSes).
Lisp meets that criteria.

One of the best features of Lisp in any application is:
HANDER-CASE/HANDLER-BIND.

When in doubt wrap everything within one of these, record any errors
that are caught and fix them.

Oh yes. Make sure you thoroughly test everything.

I really have no comments about PHP, my only experience with it
was in a project where the person delivering some functionality
was unable to deliver. Whether it was the person or PHP or
some combo of the two I am not sure.

Wade

.... ..itz..

Wade Humeniuk <whumeniu-delete-this-antispam-device@telus.net> writes:

>
> 3) The interaction between your OS and hardware. (And unless you
> are willing to shell out some serious dough it is likely you
> will have or develop problems). For a relatively inexpensive system
> my recommendation would be FreeBSD and some good PC hardware, good
> motherboard (ASUS, Gigabyte), memory cooling, and other quality parts.
> Then use some good Lisp (CMUCL runs on FreeBSD).
>

If you do go with FreeBSD, and it is a great choice, make sure to get
network cards that are able to use the polling interface, Intel cards
come to mind. The difference is instead of generating an interrupt
for each network packet as it come in every so often(configurable
interval) the kernel get any traffic outstanding and processes it.
This should make the box much happier under high load.

marc

..m ..adsh..

Chris Capel wrote:

> I think this is a special case of the more general "Microsoft
anxiety" that
> particularly affects developers, like me, that have grown up using
> Microsoft software--OS, dev. tools, etc. The perception is, with MS,
/or/
> any other widely used platform, that even if something doesn't work,
it's
> either /your/ fault, or a hundred people have already had the problem
and
> have posted workarounds on the web, so you're never going to be
screwed (at
> least by the software itself). But, in me, it's a very deep-rooted
> attitude, and it manifests itself as a fear of doing anything
substantial
> in a language that isn't already doing something substantial is
substantial
> cir***stances. I'm disappointed in myself, because Lisp is the
greatest,
> and I'd love to have a site running in lisp, and I don't think I
really
> have any /rational/ fears concerning it.

I think these worries are completely reasonable. If you use some
left-field system, then there will be much less information around on
how well it works and so on, so your risk of it not working at all, or
failing under load, is much higher. On the other hand, the `risk'
that it will help you do something radically better than the
competition is much higher too, and that's a good thing.

I think the choice depends on how risk-averse you are, how confident
you are in your ability to check that there are no show-stopper issues
without actually implementing the whole system, and how confident you
are in your ability to work-around any non show-stopper issues.

In my case I have decided not to push Lisp on at least a couple of
occasions because of what I suspected would be show-stopper issues
(once lack of an implementation that would support the large (> 32bit)
data sets we needed, and once worries about threading issues - we
needed native threads and I wasn't convinced that any of the
implementations had them in a usable form, or that we'd not run into
some horrible issue on the implementation(s) that did). I think these
were reasonable decisions, but I am pretty risk averse.

That being said, I think the whole `language supremacist' thing that
Lisp people are so keen on is really a bit of a red herring. Well, a
lot of a red herring in fact. Programming language choice makes less
difference than you might think, especially for large systems.

For instance, I'm currently working on a fairly large e-commerce
system: I think there are many tens (possibly several hundred)
machines, multiple database backends, interfaces to many suppliers and
so on. 90% of this is in Java. Arguably some benefit could be gained
by using Lisp (especially now that there are good Lisp<->Java
interfaces so it wouldn't be necessary to reimplement all the random
database & other ****: existing and reasonably good Java APIs could be
used for all the boring low-level stuff).

However changing to Lisp would only solve a tiny part of the problems
of this system. There are *much* more significant problems concerning
configuration and release management, testing, reliability, coherent
logging and auditing, debuggability, do***entation, replicability and
so on. These problems are really nothing to do with language choice,
they're just the standard `****ed up system' issues. In fact, my
guess is that Lisp would make these problems *worse*. Doing the system
in Lisp (***uming that it could be started from scratch, so there
wouldn't be some crippling reimplementation cost) would make it,
perhaps 4 times smaller (only 4 because I only partly believe the
factor-of-10-better stuff in the first place, and because even if
that's true there is still an awful lot of boring interface-to-x stuff
which would still be large). A system that was a quarter of the size
of the current one would still be large: in particular it would still
be large enough to require serious, if boring, software engineering
stuff. But, being in Lisp, it would need Lisp programmers and,
frankly, yes, they do tend to be primadonnas, and that is absolutely
the last sort of people you want in a large project like this.

To put it concisely: you can't do large systems the same way you do
small ones: boring software engineering stuff is more important than
cool programming. Lisp (or any language) can't make large systems
into small ones if they are large enough (and `large enough' isn't
very large unfortunately). In my experience, Lisp programmers are
*less* good at the boring SW engineering stuff than, say, Java
programmers.

--tim

..yss.. .... ..bbo..

Tim Bradshaw wrote:
> But, being in Lisp, it would need Lisp programmers and,
> frankly, yes, they do tend to be primadonnas, and that is absolutely
> the last sort of people you want in a large project like this.
>
> To put it concisely: you can't do large systems the same way you do
> small ones: boring software engineering stuff is more important than
> cool programming. Lisp (or any language) can't make large systems
> into small ones if they are large enough (and `large enough' isn't
> very large unfortunately). In my experience, Lisp programmers are
> *less* good at the boring SW engineering stuff than, say, Java
> programmers.

I'd be interested in knowing a little bit more about peoples'
experiences working with lisp programmers. The nice thing about
nowadays is there's less lisp fundamentalism preempting any interesting
dissections on the flaws of the lisp world. (The only way to improve is
to be honest...)


MfG,
Tayssir

..ef.. ..m

"Tim Bradshaw" <tfb+google@tfeb.org> writes:
>That being said, I think the whole `language supremacist' thing
>that Lisp people are so keen on is really a bit of a red
>herring. Well, a lot of a red herring in fact. Programming
>language choice makes less difference than you might think,
>especially for large systems.

To me, it sometimes is amazing how the "Lisp-Difference" has
changed. I mean, what makes Lisp or LISP different from the
mainstream languages of its time.

It used to be the garbage collector and the ease with which
dynamic data structures, such as trees, can be built. What was
called "AI programming" often needed such features.

Nowadays, a mainstream language, such as Java, has support for
garbage collection and several kinds of dynamic collections.

Then, LISP had support for a kind of literal of trees or
graphs, i.e., S-expressions. So the datastructures could not
only be programmed using lisp, but the constant data needed
could also be written as S-expressions. Nowadays, XML is often
used for the same purpose in Java, for example, to configure
the tool "Ant", which fulfills a purpose similar to the UNIX
"make" tool.

So, Lisp programmers nowadays have to look for other areas
that make Lisp different from other languages. Often, I hear
about macros in this regard. What is accomplished with macros
roughly correspond to building special purpose languages,
which might be implemented in other languages, too, when
required. This might still be more effort than using Lisp-
macros, but when the language is XML-based, tools to parse
it to an object are already given.

Macros, IIRC, where not present in historic variants
of LISP, so the "Lisp-Difference" of today might have not
much in common with the "LISP-Difference" of 30 years ago.