MantisBT

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0004875AContentImport/Exportpublic2011-10-05 10:162011-10-08 20:28
Reportergreg 
Assigned Toharris 
PrioritynormalSeverityminorReproducibilityalways
StatusassignedResolutionopen 
PlatformOSOS Version
Product Version 
Target Version1.2Fixed in Version 
Summary0004875: Import from URL not working
DescriptionImporting a CC or CP from Openlearn fails when importing via http, either from OL, or from a file located on atutor.ca. Same file imports okay when uploaded.
TagsNo tags attached.
Attached Filespatch file icon 0001-HTTPRequest-class-that-supports-cookies-and-redirect.patch [^] (4,574 bytes) 2011-10-05 19:11 [Show Content]
patch file icon 0002-Use-HTTPRequest-class-for-importing-IMS-Common-Cartr.patch [^] (1,609 bytes) 2011-10-05 19:12 [Show Content]
patch file icon 0001-Remove-HTTPRequest-debug-logging.patch [^] (836 bytes) 2011-10-05 19:42 [Show Content]

- Relationships
duplicate of 0004424closedharris ATutor Import Direct from Openlearn 

-  Notes
(0005443)
greg (administrator)
2011-10-05 10:17
edited on: 2011-10-05 10:24

http://atutor.ca/atutor/files/OER_1_imscc.zip [^]
this one fails

(0005444)
greg (administrator)
2011-10-05 10:24

http://atutor.ca/atutor/files/LIB_1_imscc.zip [^]
this one works
(0005445)
IntuitiveNipple (developer)
2011-10-05 10:54
edited on: 2011-10-05 11:07

I originally reported this issue via IRC after experiencing it with the UK Open University's OpenLearn Common Cartridge downloads.

http://openlearn.open.ac.uk/blocks/formats/logrequest.php?url=/file.php/2370/exports/LIB_5_imscc.zip [^]

The OU server performs several HTTP 303 'See Other' Location redirects which appears to cause PHP's fopen() to fail. The error was TR_ERROR_IMPORT_FAILED.

Note: If the ZIP is provided directly, the package will fail validation due to a separate bug in namespace usage for the forum DTD.

(0005446)
IntuitiveNipple (developer)
2011-10-05 11:31

I tested both the files pointed to by greg by importing into my AContent instance and both are fetched correctly, but fail validation of the XML. I've opened issue 0004876 to cover that problem so they don't get conflated here.

http://atutor.ca/atutor/mantis/view.php?id=4876 [^]
(0005447)
IntuitiveNipple (developer)
2011-10-05 14:02
edited on: 2011-10-05 14:08

Updates from further tests and exploration of the PHP source code.

The server I'm using is Ubuntu Lucid 10.04 LTS with:

apt-cache policy php5-common libcurl3 | grep -B 1 ' Installed'
php5-common:
  Installed: 5.3.2-1ubuntu4.9
--
libcurl3:
  Installed: 7.19.7-1ubuntu1.1

Looking at the source-code for php5 I investigated the HTTP stream wrappers for fopen in ext/standard/basic_functions.c:

#ifndef PHP_CURL_URL_WRAPPERS
    php_register_url_stream_wrapper("http", &php_stream_http_wrapper TSRMLS_CC);
    php_register_url_stream_wrapper("ftp", &php_stream_ftp_wrapper TSRMLS_CC);
#endif

I looked at the internal HTTP protocol wrapper in ext/standard/http_fopen_wrapper.c. This code handles up to 20 levels of 'Location' redirect *if* the server is NOT configured with safe_mode or open_basedir.

However, this code is NOT used since the php5 ./configure script has

#define PHP_CURL_URL_WRAPPERS 1

This means the libcurl wrapper in ext/curl/ is used.

The code appears to enable curl's CURLOPT_FOLLOWLOCATION in ext/curl/streams.c using:

curl_easy_setopt(curlstream->curl, CURLOPT_FOLLOWLOCATION, 1);

However, testing curl manually in a terminal I found that by default the option is disabled and in such case only the first document is fetched (I used the "-I" option to only fetch headers).

With follow-location enabled ("-L") the redirects are followed but the final document fetched is the Open University home-page - the same result as experienced in the AContent import code:

curl -LI http://openlearn.open.ac.uk/blocks/formats/logrequest.php?url=/file.php/2370/exports/LIB_5_imscc.zipHTTP/1.1 [^] 303 See Other
Date: Wed, 05 Oct 2011 17:00:42 GMT
Server: Apache/2.2.3 (Red Hat)
X-Powered-By: PHP/5.3.2
Set-Cookie: MoodleSession=pfq8emugg0une90f30pui0no15; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Set-Cookie: MoodleSessionTest=mclsQ0zo4S; path=/
Location: http://openlearn.open.ac.uk/file.php/2370/exports/LIB_5_imscc.zip [^]
Content-Type: text/html; charset=UTF-8
Set-Cookie: NSC_ith-pqfomfbso-80=ffffffff8a64916745525d5f4f58455e445a4a423660;expires=Wed, 05-Oct-2011 17:28:59 GMT;path=/

HTTP/1.1 303 See Other
Date: Wed, 05 Oct 2011 17:00:42 GMT
Server: Apache/2.2.3 (Red Hat)
X-Powered-By: PHP/5.3.2
Set-Cookie: MoodleSession=ii732o4g9kilmkvlsb0lmb8m73; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Set-Cookie: MoodleSessionTest=6cRseDH6qT; path=/
Location: http://openlearn.open.ac.uk/login/index.php?loginguest=true [^]
Content-Type: text/html; charset=UTF-8
Set-Cookie: NSC_ith-pqfomfbso-80=ffffffff8a64916745525d5f4f58455e445a4a423660;expires=Wed, 05-Oct-2011 17:28:59 GMT;path=/

HTTP/1.1 303 See Other
Date: Wed, 05 Oct 2011 17:00:42 GMT
Server: Apache/2.2.3 (Red Hat)
X-Powered-By: PHP/5.3.2
Set-Cookie: MoodleSession=tkibev52fe0kvsa7v68k12fqc3; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Set-Cookie: MoodleSessionTest=9n4KShIVxs; path=/
Set-Cookie: MoodleSession=25eo2vccru5u7fb3qhd2551bs2; path=/
Location: http://openlearn.open.ac.uk/ [^]
Content-Type: text/html; charset=UTF-8
Set-Cookie: NSC_ith-pqfomfbso-80=ffffffff8a64916745525d5f4f58455e445a4a423660;expires=Wed, 05-Oct-2011 17:28:59 GMT;path=/

HTTP/1.1 200 OK
Date: Wed, 05 Oct 2011 17:00:42 GMT
Server: Apache/2.2.3 (Red Hat)
X-Powered-By: PHP/5.3.2
Set-Cookie: MoodleSession=f2i9rctgnphcq2kptft3uj55u3; path=/
Expires:
Cache-Control: private, pre-check=0, post-check=0, max-age=0
Pragma: no-cache
Set-Cookie: MoodleSessionTest=xin4ANYY7p; path=/
Set-Cookie: MOODLEID_=deleted; expires=Tue, 05-Oct-2010 17:00:41 GMT; path=/
Set-Cookie: MOODLEID_=%25ED%25C3%251CC%25B7d; expires=Sun, 04-Dec-2011 17:00:42 GMT; path=/
Content-Script-Type: text/javascript
Content-Style-Type: text/css
Content-Language: en
Accept-Ranges: none
Content-Type: text/html; charset=utf-8
Set-Cookie: NSC_ith-pqfomfbso-80=ffffffff8a64916745525d5f4f58455e445a4a423660;expires=Wed, 05-Oct-2011 17:28:59 GMT;path=/

I then added the Cookie-jar option "-c cookies.txt" and found the correct resource was fetched:

curl -c cookies.txt -LI http://openlearn.open.ac.uk/blocks/formats/logrequest.php?url=/file.php/2370/exports/LIB_5_imscc.zip [^]
HTTP/1.1 303 See Other
Date: Wed, 05 Oct 2011 17:22:39 GMT
Server: Apache/2.2.3 (Red Hat)
X-Powered-By: PHP/5.3.2
Set-Cookie: MoodleSession=j6gth3aitv1eodm3rdubickh22; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Set-Cookie: MoodleSessionTest=gUzLnd9tw8; path=/
Location: http://openlearn.open.ac.uk/file.php/2370/exports/LIB_5_imscc.zip [^]
Content-Type: text/html; charset=UTF-8
Set-Cookie: NSC_ith-pqfomfbso-80=ffffffff8a64916745525d5f4f58455e445a4a423660;expires=Wed, 05-Oct-2011 17:50:55 GMT;path=/

HTTP/1.1 303 See Other
Date: Wed, 05 Oct 2011 17:22:39 GMT
Server: Apache/2.2.3 (Red Hat)
X-Powered-By: PHP/5.3.2
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Location: http://openlearn.open.ac.uk/login/index.php?loginguest=true [^]
Content-Type: text/html; charset=UTF-8
Set-Cookie: NSC_ith-pqfomfbso-80=ffffffff8a64916745525d5f4f58455e445a4a423660;expires=Wed, 05-Oct-2011 17:50:56 GMT;path=/

HTTP/1.1 303 See Other
Date: Wed, 05 Oct 2011 17:22:39 GMT
Server: Apache/2.2.3 (Red Hat)
X-Powered-By: PHP/5.3.2
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Set-Cookie: MoodleSession=o69hkicu2ev0no6b86vm5c0sb3; path=/
Location: http://openlearn.open.ac.uk/file.php/2370/exports/LIB_5_imscc.zip [^]
Content-Type: text/html; charset=UTF-8
Set-Cookie: NSC_ith-pqfomfbso-80=ffffffff8a64916745525d5f4f58455e445a4a423660;expires=Wed, 05-Oct-2011 17:50:56 GMT;path=/

HTTP/1.1 200 OK
Date: Wed, 05 Oct 2011 17:22:39 GMT
Server: Apache/2.2.3 (Red Hat)
X-Powered-By: PHP/5.3.2
Expires: Thu, 06 Oct 2011 17:22:39 GMT
Cache-Control: max-age=86400
Pragma:
Last-Modified: Wed, 27 Apr 2011 15:43:46 GMT
Content-Disposition: inline; filename="LIB_5_imscc.zip"
Set-Cookie: OUFULLSIZE=F; path=/; domain=.open.ac.uk
Accept-Ranges: bytes
Content-Length: 3079307
Content-Type: application/zip
Set-Cookie: NSC_ith-pqfomfbso-80=ffffffff8a64916745525d5f4f58455e445a4a423660;expires=Wed, 05-Oct-2011 17:50:56 GMT;path=/

So, it appears as if setting the PHP option CURLOPT_COOKIEJAR using curl_setopt () would help here. Unfortunately, it doesn't appear possible from within the AContent code since the PHP curl resource handle is not available.

(0005451)
harris (developer)
2011-10-05 16:03

More, I tried using lwp and here is a list of where they redirected us to, like you said, seems to be a session/cookie problem.

User-Agent: lwp-request/5.834 libwww-perl/5.837

GET http://openlearn.open.ac.uk/blocks/formats/logrequest.php?url=/file.php/3524/exports/TA212_2_imscc.zip [^] --> 303 See Other
GET http://openlearn.open.ac.uk/file.php/3524/exports/TA212_2_imscc.zip [^] --> 303 See Other
GET http://openlearn.open.ac.uk/login/index.php?loginguest=true [^] --> 303 See Other
GET http://openlearn.open.ac.uk/ [^] --> 200 OK
Cache-Control: private, pre-check=0, post-check=0, max-age=0
Connection: Keep-Alive
Date: Wed, 05 Oct 2011 20:02:12 GMT
Pragma: no-cache
Accept-Ranges: none
Server: Apache/2.2.3 (Red Hat)
Content-Language: en
Content-Type: text/html; charset=utf-8
Expires:
Client-Date: Wed, 05 Oct 2011 20:02:12 GMT
Client-Peer: 137.108.198.56:80
Client-Response-Num: 1
Client-Transfer-Encoding: chunked
Content-Script-Type: text/javascript
Content-Style-Type: text/css
Keep-Alive: timeout=15, max=68
Link: <http://openlearn.open.ac.uk/theme/standard/styles.php>; [^] rel="stylesheet"; type="text/css"
Link: <http://openlearn.open.ac.uk/theme/ou/styles.php>; [^] rel="stylesheet"; type="text/css"
Link: <http://openlearn.open.ac.uk/theme/oci/styles.php>; [^] rel="stylesheet"; type="text/css"
Link: <http://openlearn.open.ac.uk/filter/embed/style.css>; [^] rel="stylesheet"; type="text/css"
Link: <http://www.open.ac.uk/includes/header.css>; [^] rel="stylesheet"; type="text/css"
Link: <http://www.open.ac.uk/includes/headers-footers/header-public-wide-gradient.css>; [^] rel="stylesheet"; type="text/css"
Link: <http://openlearn.open.ac.uk/theme/oci/ipad.css>; [^] media="only screen and (max-device-width: 1024px)"; rel="stylesheet"; type="text/css"
Link: <http://openlearn.open.ac.uk/theme/oci/favicon.ico>; [^] rel="shortcut icon"
Set-Cookie: MoodleSession=dgk1eja11cskb5ddn6ivuc30l5; path=/
Set-Cookie: MoodleSessionTest=w89tb1ASgl; path=/
Set-Cookie: MOODLEID_=deleted; expires=Tue, 05-Oct-2010 20:02:11 GMT; path=/
Set-Cookie: MOODLEID_=%25ED%25C3%251CC%25B7d; expires=Sun, 04-Dec-2011 20:02:12 GMT; path=/
Set-Cookie: NSC_ith-pqfomfbso-80=ffffffff8a64919945525d5f4f58455e445a4a423660;expires=Wed, 05-Oct-2011 20:30:29 GMT;path=/
Title: OpenLearn - The Open University
X-Meta-Description: Visit OpenLearn - open educational resources from The Open University. Free online learning materials for all to use.
X-Meta-Keywords: OpenLearn, Download, Remix, Open, Educational Resources, OER, Experimental, Upload, Free, Lab, XML, Knowledge maps, Flashmeeting, Compendium, Learning journal, Tools, Research, Improve, Evolve, Learning, Units, OCW, courseware, opencourseware
X-Meta-Robots: index,follow
X-Meta-Verify-V1: fuC6cBOC/+APyX0DmxsoW+w97mVW72ZeicHyFnq9Q08=
X-Powered-By: PHP/5.3.2
(0005452)
IntuitiveNipple (developer)
2011-10-05 16:20

I'm just finalising adding cookie support to the HTTPRequest class I referred to in my IRC session with Gregg. Once I've fully tested it I'll add a patch here.

As I said earlier, we can't rely on using the current include/classes/XML/XML_HTMLSax/PEAR/Common.php::file_get_contents($filename) since it uses fopen() internally which will use either curl or the internal HTTP stream wrapper (depending on the build configuration of PHP on the end-user's server), and we do not have access to control the cookie.
(0005453)
IntuitiveNipple (developer)
2011-10-05 19:12
edited on: 2011-10-06 08:18

My fix is now available. If you use git my local repository is based on SVN r333. Please pull from:

git pull git://iam.tj/acontent.git [^] issue_4875

Alternatively, I've attached the two patches here.

(0005455)
IntuitiveNipple (developer)
2011-10-05 19:42

From the apache2 error.log (for the virtual domain):

PHP Notice: HTTPRequest("http://openlearn.open.ac.uk/blocks/formats/logrequest.php?url=/file.php/2370/exports/LIB_5_imscc.zip" [^]
PHP Notice: HTTPRequest("http://openlearn.open.ac.uk/file.php/2370/exports/LIB_5_imscc.zip" [^]
PHP Notice: Cookie: MoodleSession=ekbsj115o42qd4e1v19ji7d6e2; path=/
PHP Notice: Cookie: MoodleSessionTest=JEJAaxSkmd; path=/
PHP Notice: Cookie: NSC_ith-pqfomfbso-80=ffffffff8a64919945525d5f4f58455e445a4a423660;expires=Wed, 05-Oct-2011 22:02:25 GMT;path=/
PHP Notice: HTTPRequest("http://openlearn.open.ac.uk/login/index.php?loginguest=true" [^]
PHP Notice: Cookie: MoodleSession=ekbsj115o42qd4e1v19ji7d6e2; path=/
PHP Notice: Cookie: MoodleSessionTest=JEJAaxSkmd; path=/
PHP Notice: Cookie: NSC_ith-pqfomfbso-80=ffffffff8a64919945525d5f4f58455e445a4a423660;expires=Wed, 05-Oct-2011 22:02:25 GMT;path=/
PHP Notice: HTTPRequest("http://openlearn.open.ac.uk/file.php/2370/exports/LIB_5_imscc.zip" [^]
PHP Notice: Cookie: MoodleSession=083guh6qdbo5n2meu2f2n3g7s2; path=/
PHP Notice: Cookie: MoodleSessionTest=JEJAaxSkmd; path=/
PHP Notice: Cookie: NSC_ith-pqfomfbso-80=ffffffff8a64919945525d5f4f58455e445a4a423660;expires=Wed, 05-Oct-2011 22:02:25 GMT;path=/

This shows HTTPRequest recursively calling itself with the redirects and for each redirect, the cookies it is sending in the HTTP request.

This debugging output is removed by the third patch in my issue_4875 series.
(0005458)
harris (developer)
2011-10-06 11:53

A note for applying the attached patch "0001-HTTPRequest-class-that-supports-cookies-and-redirect.patch", there is an empty line after the file, causing a header already sent error. Remove that empty line after applying the patch and it should work without a giving you a blank page.
(0005464)
IntuitiveNipple (developer)
2011-10-08 16:55

I've enhanced the patches to provide Exception and error handling and reporting. They are available via:

git pull git://iam.tj/acontent.git [^] issue_4875

Or see http://iam.tj/gitweb/?p=acontent.git;a=shortlog;h=refs/heads/issue_4875 [^]
(0005467)
IntuitiveNipple (developer)
2011-10-08 20:28

The error report will not display unless the fix for issue 0004883 "Unreachable error-handling code in ims_import.php" is also applied.

- Issue History
Date Modified Username Field Change
2011-10-05 10:16 greg New Issue
2011-10-05 10:16 greg Status new => assigned
2011-10-05 10:16 greg Assigned To => harris
2011-10-05 10:17 greg Note Added: 0005443
2011-10-05 10:24 greg Note Edited: 0005443 View Revisions
2011-10-05 10:24 greg Note Added: 0005444
2011-10-05 10:54 IntuitiveNipple Note Added: 0005445
2011-10-05 11:07 IntuitiveNipple Note Edited: 0005445 View Revisions
2011-10-05 11:31 IntuitiveNipple Note Added: 0005446
2011-10-05 14:02 IntuitiveNipple Note Added: 0005447
2011-10-05 14:08 IntuitiveNipple Note Edited: 0005447 View Revisions
2011-10-05 16:03 harris Note Added: 0005451
2011-10-05 16:20 IntuitiveNipple Note Added: 0005452
2011-10-05 19:11 IntuitiveNipple File Added: 0001-HTTPRequest-class-that-supports-cookies-and-redirect.patch
2011-10-05 19:12 IntuitiveNipple File Added: 0002-Use-HTTPRequest-class-for-importing-IMS-Common-Cartr.patch
2011-10-05 19:12 IntuitiveNipple Note Added: 0005453
2011-10-05 19:42 IntuitiveNipple Note Added: 0005455
2011-10-05 19:42 IntuitiveNipple File Added: 0001-Remove-HTTPRequest-debug-logging.patch
2011-10-06 08:18 IntuitiveNipple Note Edited: 0005453 View Revisions
2011-10-06 10:08 harris Relationship added duplicate of 0004424
2011-10-06 11:53 harris Note Added: 0005458
2011-10-08 16:55 IntuitiveNipple Note Added: 0005464
2011-10-08 20:28 IntuitiveNipple Note Added: 0005467


Copyright © 2000 - 2017 MantisBT Team
Powered by Mantis Bugtracker