Welcome to the Second Life Forums Archive

These forums are CLOSED. Please visit the new forums HERE

Data storage via llHTTPRequest -- Beta

Zero Linden
Linden Lab Employee
Join date: 18 Oct 2005
Posts: 22
07-09-2006 22:53
I'm pleased to announce the beta release of an open source PHP script that implements data storage on your web server for LSL scripts via llHTTPRequest.

From the README:
Silo stores and retrieves data in an (almost) arbitrary tree of
URLs on a web server. It is very similar to a file system. It was
written to provide data storage for LSL scripts in Second Life.
However, it is general enough to be used from other languages and
systems, and to even store other kinds of data. (Though LSL can
only access text.)


The script implements a service similar the ones offered by other residents, but is somewhat simpler in design, includes all source, and you can (must) run it yourself. It also provides a clear example of how to interact with llHTTPRequest from PHP.

You will need your own web server on the internet, and it will need to support PHP version 4.3.0 or later.

You can download the gzip'd tar archive with the PHP source, documentation, and a python based test suite here: silo.tgz

I'm running this on my personal web server, and you are free to use it for experimenting, but I will be wiping the data store regularly, so don't use it for anything important. The base URL for that installation is http://www.notabene-sl.com/silo. See the README for how to use it.

Remember, this is BETA. It probably has flaws and bugs, and isn't feature complete. Please post comments, bugs, fixes, etc... here.
Aaron Levy
Medicated Lately?
Join date: 3 Jun 2004
Posts: 2,147
Excellent!
07-10-2006 01:08
Yay! Thanks Zero -- can't wait to try this out!
Russell Hansen
Texi pets are here!
Join date: 11 Apr 2006
Posts: 107
07-10-2006 01:09
Perfect timing, I was just thinking about doing some web based data storage for one of my objects. I'll give it a go, thanks.
Russell Hansen
Texi pets are here!
Join date: 11 Apr 2006
Posts: 107
07-10-2006 01:53
OK, had some failures running the Python test script. Apache version is 1.3.34.

I got error 500's on the wordpath, hexpath, okpath and allowedcharacters tests
Raphael Rutherford
Resident Resident
Join date: 26 Mar 2004
Posts: 236
07-10-2006 03:01
Sounds cool.
But, I gave this a shot and just got a load of errors like this :

AssertionError: expected good status, got 403, processing path /E769FCEC-3D1A-4D53-8FC7-BB3B0BBAFBEA/0123456789

Finally, it tells me:

FAILED (failures=4, errors=14)

All permissions set open, data directory set to 777. PHP 4.3.9, Apache 2.0.52

_*_*_*_*__*

UPDATE:
Added a .htaccess to the silo directory with
CODE

<Files silo.php>
Order allow,deny
Allow from all
</Files>
and all tests passed.
_____________________

Goodbye and thanks for all the prims.
Zero Linden
Linden Lab Employee
Join date: 18 Oct 2005
Posts: 22
07-10-2006 07:47
From: Russell Hansen
I got error 500's on the wordpath, hexpath, okpath and allowedcharacters tests


The only 500 status the script generates is when the PHP version is less than 4.3.0. All others are generated by Apache. So these errors are probably due to configuration issues with Apache. Things to check:

Is PHP enabled? Many installations (like Mac OS X) don't enable it by default. Look for LoadModule and AddModule directives that enable it.

Make sure that access to the silo.php script is explicitly enabled using Order and Allow directives either on the directory that contains silo.php, some parent directory of that, or on the silo.php file itself. (<Directory> or <File> sections).

Lastly, look at the error log for Apache, which should give you some indication of what went wrong.
Lex Neva
wears dorky glasses
Join date: 27 Nov 2004
Posts: 1,361
07-10-2006 12:04
Whew... this thing makes me nervous. I'm not someone who's gonna use it, since I've already homebrewed my own system, but I wanted to take a peek just to see what it was all about.

It makes me very, very nervous that the recommended setup has this script writing files into a directory that's right in the apache documentroot. The problem is that people can be writing things that will be served directly from your webserver -- not just to authenticated clients from in SL (for that matter, there's no authentication, and no file size limiting -- someone can fill up your disk for you). This means that someone can very easily use this to turn your webserver into a mirror for whatever large or high-traffic data they want you to host for them!

It gets even scarier when you consider that people could try to make the script store a CGI script for them. I tested it, and it doesn't look like the execute permissions are set for files that are created, and it doesn't look like I can force a file to get stored with a .php extension (which would make exploitation trivial). I wonder, though, what happens if someone has their umask set wrong, so files get created with execute permissions. Then it'd be trivial to make the HTTP daemon execute anything you feed it.

The other thing that makes me nervous is that the README suggests "It's not
unreasonable to simply make it writable by all." True, anyone can create anything in the data directory through the PHP script, but there are within limits. They can't create executable files, and they can't create PHP scripts. However, setting the directory world-write means that any other user on a shared hosting server could put CGI or PHP scripts into your directory and convince the webserver to run the scripts. If the server has "setuid" mode on for CGI scripts (often used in such situations), then anyone else on your shared hosting server owns you.

Two possibilities are available for mitigating the risk. One is to limit access to the data directory. No one should be able to access it directly from outside via the webserver; they should have to go through the script. Do that like this in Apache:

CODE

<Directory /path/to/silo/data>
Order deny,allow
Deny from all
</Directory>


Alternatively, you can create the data directory outside of the webserver document root and make "data" be a symlink to it. Then disable following symbolic links in the silo directory:

CODE

<Directory /path/to/silo>
Options -FollowSymlinks
</Directory>


Ideally, I think it'd be preferable to store the data in an SQL database instead, although I understand that the purpose of this script was to give anyone with limited hosting options the ability to store data.
Talarus Luan
Ancient Archaean Dragon
Join date: 18 Mar 2006
Posts: 4,831
07-10-2006 12:12
The only question I have at the moment is the utility of such things when there is such a low "fence" for the number of requests per owner per sim? I would like to do something other than the "database scripts" I have now, but any significant amount of use in something like a vendor in a mall, it will constantly be hitting the request limit, and the vendor would have to go non-functional for several minutes to clear it. That just won't do.
Zero Linden
Linden Lab Employee
Join date: 18 Oct 2005
Posts: 22
07-10-2006 12:26
Some comments -

Initially - indeed, if you are running your own web server, then you need to be aware of all the issues raised, for this or any script you install on it. And if you are using a shared web service, then you also need to be aware of how that service sets up access and how it affects things you use. I'd be happy to include good "be ware of what you do" text in the document. Suggestions?

The third sample configuration can be used with the data directory under DocumentRoot. I can change the documentation to suggest that be the recommended set up. It is somewhat easier than your symlink solution. You're suggestion of denying access to the data directory if it is under DocumentRoot is good, I'll add it to the documentation.

The script will not let you write files with .php, .cgi, .pl, or any other extension for that matter. It outlaws the period in path names, and inserts it's own extension (.data and .meta) on the files it writes. Furthermore, it never sets mod bits, so there should be no way to get the +x bit turned on.

Note that the script doesn't do any authentication that access is coming from SL. This was by design, as there are uses for getting at the info from other sources. And indeed, if someone knows your Silo base URL they can fill up your disk. Of course, using an SQL database would still have the same issue. The script is a skeleton on which to add more features. Add 'em!
Russell Hansen
Texi pets are here!
Join date: 11 Apr 2006
Posts: 107
07-10-2006 13:12
OK well I put in the .httaccess code suggested by Raphael, and now I only have 1 failure, but 25 errors.

Is there some way to stop the info scrolling off the screen? Piping to more doesn't seem to work. I'm just using an ssh console that my host provides to run the test scripts, oh and my php version is 4.4.2 and runs all my web sites fine, so it must be something local.

The errors I'm getting all seem to have the last line at line 614 in connect socket.SOCK_STREAM

gaierror: (-2, 'Name or service not known')

The failure is in Tests_A_Setup and the darn window won't let me copy and paste, but at line 108 in test.py in
test000_baseURL self.failUnless(silo.baseHost)
AssertionError
Zero Linden
Linden Lab Employee
Join date: 18 Oct 2005
Posts: 22
07-10-2006 14:06
Russell - if your personal machine has Python installed, then you can run the tests from your machine locally (giving it the URL of your web service). This should make debugging easier.

If you are running in a shell, and can't scroll back, it is probably because the output is on stderr, not stdout. You can still get this paged in less by doing this:

CODE
python test.py http://www.example.com/silo -v 2>&1 | less
Ginge Reymont
Registered User
Join date: 10 Oct 2005
Posts: 190
07-10-2006 14:10
Im getting 405 error, any ideas?
Zero Linden
Linden Lab Employee
Join date: 18 Oct 2005
Posts: 22
07-10-2006 16:51
405 is Method Not Allowed

Either you are trying to PUT to a path that ends in a slash .... or your Apache server isn't configured to actually run the PHP script ... or you are doing a POST (which isn't part of Silo's API).

Can you provide more context? Are you getting a 405 during running the python tests, or during usage? What's the path (replace your domain name with example.com).
Jesse Malthus
OMG HAX!
Join date: 21 Apr 2006
Posts: 649
07-10-2006 17:52
Wow, it's a RESTfull PHP web storage service. Nice!
I would use ParkPlace (http://code.whytheluckystiff.net/parkplace) but LSL has no Amazon S3 library (it needs SHA1 :P)
Nice Zero! I plan on installing this soon!

I happen to be having problems though.
From: Test Suite

test005_hexPath (__main__.Tests_B_PathError) ... FAIL
test006_okayPath (__main__.Tests_B_PathError) ... FAIL
test002_wordPath (__main__.Tests_B_PathError) ... FAIL
test000_clear (__main__.Tests_C_Basic) ... ERROR
test001_basic (__main__.Tests_C_Basic) ... ERROR
test002_nestedData (__main__.Tests_C_Basic) ... ERROR
test003_dirListing (__main__.Tests_C_Basic) ... ERROR
test004_caseSensitivity (__main__.Tests_C_Basic) ... ERROR
test005_putStatus (__main__.Tests_C_Basic) ... ERROR
test000_clear (__main__.Tests_D_RoundTrip) ... ERROR
test001_simple (__main__.Tests_D_RoundTrip) ... ERROR
test002_asciiPrinting (__main__.Tests_D_RoundTrip) ... ERROR
test003_encodingASCII (__main__.Tests_D_RoundTrip) ... ERROR
test004_encodingISOLatin1 (__main__.Tests_D_RoundTrip) ... ERROR
test005_encodingUTF16 (__main__.Tests_D_RoundTrip) ... ERROR
test006_encodingUTF16LE (__main__.Tests_D_RoundTrip) ... ERROR
test007_encodingUTF16BE (__main__.Tests_D_RoundTrip) ... ERROR

and 404s on subsiquent tests. It looks like you're assuming URL rewriting. Which is by far not default.
Zarf Vantongerloo
Obscure Resident
Join date: 22 Jun 2005
Posts: 110
07-10-2006 21:03
From: Jesse Malthus
Wow, it's a RESTfull PHP web storage service. Nice!
I would use ParkPlace (http://code.whytheluckystiff.net/parkplace) but LSL has no Amazon S3 library (it needs SHA1 :P)
Nice Zero! I plan on installing this soon!


Wow! You're right - it is practically all of Amazon's S3, or ParkPlace - but with a somewhat simpler API (if that were possible), no access control, and all responses in text rather than XML (since parsing XML in LSL would be quite a feat...)

From: Jesse Malthus
...404s on subsiquent tests. It looks like you're assuming URL rewriting. Which is by far not default.


Do you have a slash at the end of the base URL you are feeding the python tester? It shouldn't be there.

The script doesn't rely on rewriting at all. It can be made to work by just placing the script in the document root of apache, in which case the base URLs are like http://example.com/my-sl-stuff/silo.php/apple/fuji. Notice the .php script in the middle of the URL!

It can also be made to work with aliases or rewriting which lets you have 'nicer' URLs like http://example.com/my-sl-silo/apple/fuji. You can achieve this effect with either Apache's Alias module or the Rewrite module.
Lex Neva
wears dorky glasses
Join date: 27 Nov 2004
Posts: 1,361
07-10-2006 21:10
From: Zero Linden
Some comments -

Initially - indeed, if you are running your own web server, then you need to be aware of all the issues raised, for this or any script you install on it. And if you are using a shared web service, then you also need to be aware of how that service sets up access and how it affects things you use. I'd be happy to include good "be ware of what you do" text in the document. Suggestions?


I think my biggest suggestion is that you should remove that part about world-writable, and say instead something about "writable by your HTTP server". People may still shoot themselves in the foot and set it world-writable as the easiest solution, but what can yah do? :)

From: someone

The third sample configuration can be used with the data directory under DocumentRoot. I can change the documentation to suggest that be the recommended set up. It is somewhat easier than your symlink solution. You're suggestion of denying access to the data directory if it is under DocumentRoot is good, I'll add it to the documentation.


I think you meant "data directory not under DocumentRoot"... and yes, I agree that that'd be better. Maybe only alias the silo.php file itself. Alternatively, making the path to the data directory a global variable defined near the top would make it easy to specify a data directory, and perhaps you could suggest that they modify that variable to point to a data directory outside the documentroot in the readme. I agree, the symlink thing is a bit needlessly complicated :)

From: someone

The script will not let you write files with .php, .cgi, .pl, or any other extension for that matter. It outlaws the period in path names, and inserts it's own extension (.data and .meta) on the files it writes. Furthermore, it never sets mod bits, so there should be no way to get the +x bit turned on.


True... it's fairly clean in what it'll write, I saw. I just didn't really know what the PHP copy function's default permissions are when the source is a fopen-wrapper. Either way, the fact that others on the system can write, say, a shellscript to that directory, perhaps even through a remote exploit in ANOTHER piece of software... made me nervous.

From: someone

Note that the script doesn't do any authentication that access is coming from SL. This was by design, as there are uses for getting at the info from other sources. And indeed, if someone knows your Silo base URL they can fill up your disk. Of course, using an SQL database would still have the same issue. The script is a skeleton on which to add more features. Add 'em!


Righto :)
Lex Neva
wears dorky glasses
Join date: 27 Nov 2004
Posts: 1,361
07-10-2006 21:13
From: Ginge Reymont
Im getting 405 error, any ideas?


I got 405s too. The problem is that Apache, due to the way I'd set it up (and probably by default), only allows GET and POST requests, and silo.php uses PUT and DELETE too. Do this:

CODE

<Directory /path/to/silo/>
<Limit GET PUT POST DELETE>
Order allow,deny
Allow from all
</Limit>
</Directory>


If your server admin has configured Apache to allow it, you can put just the <Limit>...</Limit> part into a .htaccess file in the directory.

Generally I don't think it'd be a good idea to allow PUT and DELETE in all directories in the server.
Ginge Reymont
Registered User
Join date: 10 Oct 2005
Posts: 190
07-11-2006 01:06
From: Zero Linden
405 is Method Not Allowed

Either you are trying to PUT to a path that ends in a slash .... or your Apache server isn't configured to actually run the PHP script ... or you are doing a POST (which isn't part of Silo's API).

Can you provide more context? Are you getting a 405 during running the python tests, or during usage? What's the path (replace your domain name with example.com).


I cant provide more sorry, its the LSL script that is returning the 405 error.
Bobbyb30 Zohari
SL Mentor Coach
Join date: 11 Nov 2006
Posts: 466
02-25-2008 16:34
This still up?
_____________________
Darien Caldwell
Registered User
Join date: 12 Oct 2006
Posts: 3,127
02-26-2008 12:21
well, the website still seems to be operating. So maybe.
_____________________
Cloudseer Writer
Registered User
Join date: 21 Apr 2007
Posts: 3
re 405 errors
03-01-2008 07:55
the python script wouldn't pass the test on the base url, but when I added /silo.php to the end of the base url (http://www.example.com/silo/silo.php) all tests were passed and any scripts worked fine.
Hewee Zetkin
Registered User
Join date: 20 Jul 2006
Posts: 2,702
03-01-2008 10:38
From: Cloudseer Writer
the python script wouldn't pass the test on the base url, but when I added /silo.php to the end of the base url (http://www.example.com/silo/silo.php) all tests were passed and any scripts worked fine.

That's more than likely a webserver configuration issue.
Arda Xi
Registered User
Join date: 23 Feb 2008
Posts: 2
03-09-2008 04:30
From: Hewee Zetkin
That's more than likely a webserver configuration issue.

I would really like to know, what configuration issue? Because I have the same problem. Or can I then just use www.example.com/silo/silo.php/stored/text?
Hewee Zetkin
Registered User
Join date: 20 Jul 2006
Posts: 2,702
03-09-2008 11:27
From: Arda Xi
I would really like to know, what configuration issue? Because I have the same problem. Or can I then just use www.example.com/silo/silo.php/stored/text?

Hmm. One related to the mapping of URL paths, I'd imagine. Consult your web server's documentation.
Willard Writer
Registered User
Join date: 12 Feb 2007
Posts: 5
anything better than this out there?
06-11-2009 09:55
This was posted quite a while ago.. I was just wondering if there is something out there better now days.
1 2