github/overleaf
1
0
Fork 0
mirror of https://github.com/overleaf/overleaf.git synced 2024-11-21 20:47:08 -05:00
Code Issues Projects Releases Packages Wiki Activity
19824 commits 3 branches 7 tags 206 MiB
Commit graph

639 commits

Author SHA1 Message Date
Simon Detheridge
32b7f25404 Merge pull request #49 from das7pad/misc/node10plus-support 
[misc] add support for node 10+
 2019-12-16 09:39:00 +00:00
Simon Detheridge
56b38af678 Explicitly wait for S3 in acceptance tests 2019-12-10 17:43:34 +00:00
Simon Detheridge
237c4113cd Ensure fakes3 is healthy before running tests 2019-12-10 16:36:33 +00:00
Simon Detheridge
cf684dcd98 Fix fakes3 configuration in ci yml 2019-12-10 11:38:19 +00:00
Simon Detheridge
6f326d5650 Use SSL setting based on url protocol 2019-12-09 17:41:20 +00:00
Simon Detheridge
48aa141591 Add metric for s3 ingress 2019-12-09 17:06:00 +00:00
Simon Detheridge
1d1106bc67 Add metric for s3 egress 2019-12-09 17:06:00 +00:00
Simon Detheridge
c01603b1e7 Support custom S3 endpoints 2019-12-09 17:06:00 +00:00
Simon Detheridge
96457597ac Add fake s3 server and initial config 2019-12-09 17:06:00 +00:00
Simon Detheridge
86b9e4b53a Rename request -> s3Request to prevent overwriting main import 2019-12-09 17:05:58 +00:00
Brian Gough
2ec38068aa add comments about aws-sdk and s3 backends 2019-12-06 14:35:13 +00:00
Nate Stemen
aba0d14edd bump build script to 1.1.24 2019-10-25 11:52:54 -04:00
Nate Stemen
4798589213 replace private link with public one 2019-10-25 11:51:58 -04:00
Shane Kilkelly
dbede8509c Merge pull request #54 from overleaf/sk-dep-upgrades-2019-06 
update logger and metrics
 2019-07-09 10:18:16 +01:00
Eric Mc Sween
7d900b57bf Fix createUnbufferedStream() function call 
In 49a21155f642670dfea264ac73fb60241f37cb87, I managed to incorrectly
write the `createUnbufferedStream()` function from the AWS SDK as
`getUnbufferedStream()` and to consistently use that naming in the unit
tests.

This commit fixes that. I have tested again on S3.
 2019-06-19 12:58:17 -04:00
Shane Kilkelly
013400d7a4 Re-add environment variable for conversions 2019-06-19 14:04:57 +01:00
Shane Kilkelly
f865762c29 update logger and metrics 2019-06-19 13:50:58 +01:00
Eric Mc Sween
f2521a29b9 Use AWS SDK for getFileStream() 
The AWS SDK has a retry strategy to deal with rate limiting or transient
unavailability of S3. We hope it will reduce our error rates.
 2019-06-19 08:18:11 -04:00
Eric Mc Sween
3575c89d03 Return file size on HEAD request 
This will be used by the file preview feature when it gets partial
content.
 2019-06-17 08:23:59 -04:00
Timothée Alby
f081546ec0 update Git URL in Jenkinsfile 2019-05-07 17:37:07 +01:00
Timothée Alby
32a54a7e37 Update README.md 2019-05-07 17:22:35 +02:00
Jakob Ackermann
a79adcd325 [logging] do not overwrite the logger name 
Signed-off-by: Jakob Ackermann <das7pad@outlook.com>
 2019-05-02 12:48:43 +02:00
Jakob Ackermann
c59a3db4e8 [FSPersistorManager] fix the stream opening for node10+ 
Attaching a `readable` listener causes the stream to hang otherwise.

Signed-off-by: Jakob Ackermann <das7pad@outlook.com>
 2019-05-02 02:47:53 +02:00
Brian Gough
b9d3b1a842 fix beginShutdown to use server.close() 2019-03-15 15:35:28 +00:00
Henry Oswald
37b039abf0 add sentry to settings.defaults.coffee 2019-03-05 17:20:16 +00:00
Henry Oswald
2adae20368 remove a console.log 2019-02-06 11:43:11 +00:00
Henry Oswald
0f48c77e7e enable conversions for local docker compose 2019-02-05 17:19:33 +00:00
Henry Oswald
5eec00db51 don't run conversions test when disabled 2019-02-05 14:06:51 +00:00
Henry Oswald
3fccf79ca8 cleanup 2019-02-05 12:50:08 +00:00
Henry Oswald
ca9231b04e set enable conversions to false by default but enable in compse 2019-02-05 11:19:18 +00:00
Henry Oswald
19007befb2 Merge branch 'master' into ho-docker 2019-02-01 16:26:45 +00:00
Henry Oswald
ea845f4935 bulk upgrade to 1.1.12, moved to npm rather than git 2019-01-31 16:01:20 +00:00
Brian Gough
9d93eee3e8 return a 404 error (instead of a 500) when copying a missing file 2019-01-09 10:31:59 +00:00
Henry Oswald
3614f217e6 add enableConversions flag to disable conversions which we can't do on 
k8
 2019-01-07 15:54:24 +00:00
Christopher Hoskin
0421192449 Ignore any file ending in .map, wherever it is 2019-01-04 16:22:13 +00:00
Christopher Hoskin
c8d146b571 Bump settings to v1.1.0 2019-01-04 16:03:45 +00:00
Christopher Hoskin
b522fe1531 Remove grunt 2019-01-04 16:03:22 +00:00
Christopher Hoskin
6763809748 Bump logger to v1.5.9 2019-01-03 16:27:45 +00:00
Christopher Hoskin
594b403bd3 Move metrics init to start of app.coffee 2019-01-02 09:55:31 +00:00
Christopher Hoskin
3b6eba33ba Remove Metrics.inc "startup" from app.coffee 2019-01-02 09:53:53 +00:00
Christopher Hoskin
aab5a1af6a Merge branch 'master' into csh-ho-docker 2019-01-02 09:51:35 +00:00
Christopher Hoskin
6214256568 Bump logger to v1.5.8 2019-01-02 09:47:38 +00:00
Brian Gough
cf12ec1154 use the aws sdk to copy files in S3PersistorManager 
to work around problems with knox
 2018-12-20 16:11:18 +00:00
Brian Gough
28271655e1 avoid problems with caching of apt-get operations 2018-12-06 08:19:39 +00:00
Henry Oswald
25275212af bump metrics to 2.0.7 2018-12-05 12:36:32 +00:00
Henry Oswald
5d927d30b6 update build scripts to 1.1.10 2018-12-05 10:53:11 +00:00
Henry Oswald
8743527452 bump metrics to 2.0.4 2018-11-29 16:42:26 +00:00
Henry Oswald
e64156cd27 clean npm-shrinkwrap and package.json 2018-11-29 16:41:30 +00:00
Henry Oswald
95f0a02dee stub out metrics in tests & inject metrics endpoint 2018-11-29 13:53:43 +00:00
Henry Oswald
ee1c14a410 bump metrics 2018-11-29 12:46:14 +00:00
Brian Gough
fdc5370f94 Merge pull request #36 from sharelatex/bg-upgrade-logger 
upgrade logger-sharelatex to v1.5.8
 2018-11-13 14:06:44 +00:00
Brian Gough
8951f297aa Merge pull request #37 from sharelatex/bg-reduce-logging-of-not-found-errors 
reduce logging of not found errors
 2018-11-13 14:06:19 +00:00
Brian Gough
0d6de10b5c Merge pull request #34 from sharelatex/bg-make-backends-consistent 
make filestore backends consistent
 2018-11-13 10:12:18 +00:00
Brian Gough
e930c7106b Merge pull request #35 from sharelatex/bg-serve-converted-file-from-disk 
serve file from disk to avoid read-after-write inconsistency
 2018-11-13 10:12:00 +00:00
Brian Gough
23a6d6e81d consider 403 and 404 as NotFound errors 2018-11-09 14:05:38 +00:00
Brian Gough
69b164092c suppress unnecessary error logging for NotFound 2018-11-09 14:05:38 +00:00
Brian Gough
c2bfdd095c upgrade logger-sharelatex to v1.5.8 2018-11-08 16:29:23 +00:00
Brian Gough
8744b3aa4e serve file from disk to avoid read-after-write inconsistency 2018-11-08 15:09:34 +00:00
Brian Gough
78ccf453db update unit tests 2018-11-07 16:51:06 +00:00
Brian Gough
40ac59090b make backends consistent by deleting temporary files 2018-11-07 16:22:09 +00:00
Brian Gough
06bfd4c270 add dockerfile to install packages needed by filestore 2018-11-07 16:12:49 +00:00
Christopher Hoskin
038b3476fe Add some details to the README 2018-10-22 14:09:56 +01:00
Henry Oswald
38f565d35f logout settings 2018-09-28 15:53:57 +01:00
Henry Oswald
aacb54d54c Merge branch 'master' into ho-docker 2018-09-14 17:16:18 +01:00
Brian Gough
96e9d8b0ec fix typo 2018-09-11 10:51:10 +01:00
Brian Gough
f3adf5a653 fix shrinkwrap for upgraded logger-sharelatex module 2018-09-06 17:01:40 +01:00
Brian Gough
f4ec677296 add sentry to logging 2018-09-06 17:01:18 +01:00
Christopher Hoskin
53dacc04ef Bump metrics from 1.3.0 to 1.8.1 2018-07-26 16:06:30 +01:00
Christopher Hoskin
0e9a535e1c Bump build script to 1.1.9 2018-07-26 16:05:11 +01:00
Henry Oswald
cba013cd71 change ordering of docker rmi 2018-07-23 17:06:31 +01:00
Christopher Hoskin
f9f599e3e7 Push container images to overleaf-ops project 2018-07-18 11:30:01 +01:00
Christopher Hoskin
1044069535 Remove csh-gcdm-test and csh-staging repos 2018-07-17 12:03:14 +01:00
Christopher Hoskin
c7733d05cc Remove trailing / from project name 2018-07-17 11:06:33 +01:00
Christopher Hoskin
d182e5329c Add cr-test2 GCR project 2018-07-17 10:52:20 +01:00
Christopher Hoskin
4784ae33a2 Bump build scripts to 1.1.8 2018-07-17 10:51:15 +01:00
Henry Oswald
1e4e68c64a fix settings test 2018-07-10 14:55:40 +01:00
Henry Oswald
089bf3e084 seperate the standard s3 creds from the bucket s3 creds 2018-07-10 14:17:37 +01:00
Michael Mazour
c4e3f9eb02 Amend: tests for populating S3 settings from environment variable 2018-07-06 11:05:40 +01:00
Michael Mazour
3e1ef3af63 Populate S3 settings from environment variable 2018-07-06 11:05:19 +01:00
Michael Mazour
2da15f2eb3 Amend - improve documentation of settings 2018-07-06 10:28:02 +01:00
Michael Mazour
7feafccf31 Amend: safely navigate to bucket credentials 2018-07-06 09:28:09 +01:00
Michael Mazour
03033409c6 Amend: remove unused params 2018-07-06 09:12:59 +01:00
Michael Mazour
336a38ec1e Amend: scrub secrets from logs 
Calls to `getFile` can now include S3 credentials in `opts`, so sanitize before writing to opts to log.
 2018-07-04 16:41:31 +01:00
Michael Mazour
600ab3ce67 Amend: remove problematic ampersand 2018-07-04 16:39:41 +01:00
Michael Mazour
cfbf0d81ba Amend: fix params retrieval 2018-07-04 12:11:09 +01:00
Michael Mazour
ece650741a Amend per several review comments 
- Removed unused vars
- Label the metric with the bucket name
 2018-07-04 12:02:09 +01:00
Michael Mazour
feca8933f1 Add endpoint for arbitrary bucket fetch 
Add `/bucket/:bucket/key/*`, which fetches the file from the given bucket at the given path. Uses auth stored at `settings.filestore.s3.{{bucketName}}` if present, and otherwise default auth.
 2018-07-04 11:22:36 +01:00
Christopher Hoskin
f0cf8e6622 Fix typo in buildscript 2018-06-14 11:37:18 +01:00
Christopher Hoskin
46c61aa78c Update build scripts to 1.1.7, add csh-staging GCR 2018-06-13 16:21:32 +01:00
Henry Oswald
1763715004 remove black package-lock.json 2018-05-22 13:45:32 +01:00
Henry Oswald
ba8a5cf3e1 bump to 1.1.3 scripts 2018-05-22 13:45:01 +01:00
Henry Oswald
58ab4c72dc update build scripts to 1.1.1 2018-05-21 15:14:24 +01:00
Henry Oswald
ee81a6af00 updated mocha and added --exit back in 2018-05-21 15:00:32 +01:00
Henry Oswald
8efd562eb3 fix broken acceptence tests, bad quotes in file path 2018-05-21 13:40:03 +01:00
Henry Oswald
6657b6c1dc make all needed directories in install deps 2018-05-21 11:31:14 +01:00
Henry Oswald
e3514e84cd fix up mocha migration tests 2018-05-21 11:02:53 +01:00
Henry Oswald
768a4cc1ad remove console.log from settings 2018-05-21 10:34:16 +01:00
Henry Oswald
09527d804e ignore js acceptence tests 2018-05-21 10:21:13 +01:00
Henry Oswald
3cf2e1d7c3 add null checks and new body parser 2018-05-21 10:11:47 +01:00
Henry Oswald
ac990f2aba remove app.configure 2018-05-21 09:49:32 +01:00
Henry Oswald
d9eac8dd8b upgrade mocha 2018-05-21 09:41:14 +01:00
Henry Oswald
fda6cb0084 fix acceptence tests, add them back into jenkins, upgrade express 2018-05-21 09:37:54 +01:00
Henry Oswald
df6b5203a1 point ci to gcr.io/csh-gcdm-test/ 2018-05-18 17:18:39 +01:00
Henry Oswald
28fc39d6c9 don't use quay.io in image name 2018-05-18 17:08:18 +01:00
Henry Oswald
bfb98050e3 don't push to quay.io 2018-05-18 16:57:53 +01:00
Henry Oswald
088d8850ba don't run acceptence tests yet 2018-05-18 16:48:59 +01:00
Henry Oswald
fcb2d18f8b don't set command in make test acceptence 2018-05-18 16:15:44 +01:00
Christopher Hoskin
b32c63e16c Remove --exit option to support older mocha 2018-05-18 14:59:34 +01:00
Christopher Hoskin
7fc7252c35 Remove touch package-lock.json 2018-05-18 14:29:20 +01:00
Christopher Hoskin
8db861258a Add files generated by bin/update_build_scripts 2018-05-18 14:27:46 +01:00
Christopher Hoskin
5d5cbbef7e Create fake package-lock.json during build tagret 2018-05-18 14:15:02 +01:00
Christopher Hoskin
a2165d85a0 Update build script to use build-target docker 2018-05-18 13:55:33 +01:00
Christopher Hoskin
de74ec5094 Change build and publish to use the GCR of one of my projects (hard code for now) 2018-05-18 13:47:06 +01:00
Henry Oswald
46f2ff051c cd into tmp 2018-03-23 17:29:21 +00:00
Henry Oswald
b45e2a922c install gs correctly 2018-03-23 17:21:15 +00:00
Henry Oswald
8c782ba6b2 debuging 2018-03-23 17:12:37 +00:00
Henry Oswald
801adf8e9e install gs manually 2018-03-23 17:06:31 +00:00
Henry Oswald
092a151d51 Use new build system 2018-03-23 16:36:41 +00:00
Henry Oswald
837a8ff35f change uploads permission after creation of user 2018-03-23 16:26:42 +00:00
Henry Oswald
1af209cd24 chown uploads 2018-03-23 16:23:04 +00:00
Henry Oswald
2f0cc4e64a chown uploads 2018-03-23 16:19:52 +00:00
Henry Oswald
73f77b5667 copy everything into docker container 2018-03-23 16:14:30 +00:00
Henry Oswald
9e321658ed make s3 configurable via env vars 2018-03-23 15:29:24 +00:00
Henry Oswald
77e34d33df hello james 2018-01-16 17:00:27 +00:00
Henry Oswald
08c734e850 another bump 
git log
 2018-01-16 14:48:21 +00:00
Henry Oswald
dbfebc0d1c bump 2018-01-16 14:36:09 +00:00
Henry Oswald
e235651e9c bump version 2018-01-16 12:44:26 +00:00
Henry Oswald
c23b700927 wip 2018-01-16 12:27:36 +00:00
James Allen
b834e53e13 Provide hosts as environment settings and add npm run start script 2017-12-29 08:14:23 +00:00
Joe Green
a70a216c52 Update Jenkinsfile 2017-10-16 14:10:06 +01:00
Joe Green
6dcef9fdea alert only on master 2017-10-12 17:07:06 +01:00
Henry Oswald
0e2de1e5ad added shrinkwrap file 2017-09-15 14:57:01 +01:00
Joe Green
bfef9a9d67 added build.txt 2017-09-04 14:49:14 +01:00
Joe Green
706dc917dd Create Jenkinsfile 2017-08-11 11:12:32 +01:00
Shane Kilkelly
e162c779d3 Merge pull request #24 from sharelatex/sk-ignore-png-optimiser-timeout 
If optipng is sigkilled by timeout, then ignore that error
 2017-08-03 09:22:43 +01:00
Shane Kilkelly
eaa72d0747 upgrade to node 6 2017-06-27 14:53:20 +01:00
Shane Kilkelly
23fe93ccfd Prevent double-logging 2017-06-22 15:28:08 +01:00
Shane Kilkelly
ba9477cdb3 Re-work error checking 2017-06-22 09:41:36 +01:00
Shane Kilkelly
d0bbe8beae If optipng is sigkilled by timeout, then ignore that error 2017-06-21 15:18:36 +01:00
Shane Kilkelly
2f8c693be3 Add a .nvmrc file 2017-03-27 14:49:25 +01:00
Shane Kilkelly
b12f69722f Begin fixing health-check 2017-03-08 14:59:34 +00:00
Henry Oswald
e0f39da616 null check fsPath on unlink file 2016-12-19 14:48:09 +00:00
Henry Oswald
9e8c27f46b don’t shutdown filestore on uncaught exception 2016-12-19 14:47:54 +00:00
Henry Oswald
bf93fc3f50 run health check max 1 a at a time 2016-12-19 13:26:21 +00:00
Henry Oswald
fd99887753 running check in background 2016-12-19 12:29:08 +00:00
Henry Oswald
6c81d18627 bumped up rlimit-nproc to 600 2016-12-14 18:15:09 +00:00
Henry Oswald
d41bf2e974 result path not in final cb 2016-12-14 18:11:28 +00:00
Henry Oswald
4bba56ed28 added missing files 2016-12-14 17:14:15 +00:00
Henry Oswald
27139bc433 increased rlimit-nproc and added health check for converting image 2016-12-14 16:47:53 +00:00
Henry Oswald
b166c86197 added missing originalFsPath 2016-12-13 14:29:04 +00:00
Henry Oswald
b38e7d3078 * delete both file and converted file if there is an error 
* increase timeout of optpng
 2016-12-13 12:43:57 +00:00
Shane Kilkelly
4064956cea working minimal profile 2016-09-28 10:05:27 +01:00
Shane Kilkelly
edb5987c3d A working profile, tested. 2016-09-27 15:54:41 +01:00
Shane Kilkelly
ff39a9da22 sort the list of syscalls 2016-09-26 15:35:53 +01:00
Shane Kilkelly
21be88482b Change the seccomp whitelist to match that of docker default list 2016-09-26 15:33:22 +01:00
Brian Gough
fd95599609 remove old firejail includes 2016-08-11 11:52:58 +01:00
James Allen
a95aae16e6 Tweak convert profile to work on api-2 2016-05-10 17:50:46 +01:00
James Allen
a41a307b58 Update firejail profile to allow pdf previews for templates to work 2016-05-10 14:34:48 +01:00
James Allen
92f5234580 Add convert firejail profile 2016-05-09 14:44:49 +01:00
James Allen
be43330208 Allow convert command to be prefixed by security commands 2016-05-09 11:37:35 +01:00
Henrique Santos
ffaac4c81b add unit tests 2016-03-13 20:45:48 -03:00
Henrique Santos
0bb12f32fa remove project controller / handler 
add unit test scaffolding
 2016-03-13 16:22:14 -03:00
Henrique Santos
1f20744a68 add size function for each persitor 2016-03-12 11:08:07 -03:00
Henrique Santos
5d2fc89137 scaffolding project methods 2016-03-12 04:35:49 -03:00
Henrique Santos
099d0fef2f scaffolding project size 2016-03-11 22:31:46 -03:00
James Allen
8ddfbb99de Only parse body when we need to for copyFile request 2015-11-25 17:01:19 +00:00
James Allen
bc6c8d6afc Don't send multiple status codes on insertFile error 2015-11-25 16:07:51 +00:00
Xavier Trochu
81a7203315 Fix the checkIfFileExists API. 2015-11-25 15:40:35 +01:00
Xavier Trochu
3b6270236c Replace indentation from 2 space to tabs. 2015-11-20 14:38:23 +01:00
Xavier Trochu
b6486cb825 Cleanup the dependencies 2015-11-20 14:05:54 +01:00
Xavier Trochu
0219900933 Forward the options argument to getFileStream 2015-11-20 12:07:04 +01:00
Xavier Trochu
bfd41fdaf9 Add aws-sdk unit test. 
Fix Aws-Sdk persistor to return a correct error on file not found.
Fix FileHandler after some change were lost on a previous merge
 2015-11-20 12:02:22 +01:00
Xavier Trochu
476db58c3f Add the opts parameter to getFileStream and implement Range requests 2015-11-17 12:52:25 +01:00
Xavier Trochu
26523b011a small fixes 2015-11-16 09:59:27 +01:00
Xavier Trochu
63aef6b832 Fix postmerge issue 2015-11-05 12:11:02 +01:00
Xavier Trochu
a367190a21 Merge remote-tracking branch 'sharelatex/master' 2015-11-04 15:20:15 +01:00
James Allen
f723b1699b Mongo/redis no longer needed without acceptance tests 2015-10-07 18:39:47 +01:00
James Allen
9b40d34b2a Acceptance tests no longer run on Travis 2015-10-07 18:39:24 +01:00
Henry Oswald
cf28ee4f9b locked down packages 2015-09-17 10:34:27 +01:00
Shane Kilkelly
0d450b4e33 Tighten up error handling. 2015-09-10 11:10:30 +01:00
Shane Kilkelly
fcfc595c35 Handle potential error before calling LocalFileWriter.writeStream. 2015-09-10 09:12:49 +01:00
Shane Kilkelly
a6fff2d4e4 Check for a large response body, rather than exact match on the first few bytes. 2015-09-08 11:50:43 +01:00
Shane Kilkelly
c6a81c94eb Test the cacheWarm option for preview. 2015-09-08 11:47:43 +01:00
Shane Kilkelly
b8042ae771 Add an acceptance test that uses a pdf file. 2015-09-08 11:43:19 +01:00
Shane Kilkelly
297ad78b1e Add an acceptance test to check the preview endpoint doesn't time out. 
This endpoint needs a lot more testing.
 2015-09-08 10:32:41 +01:00
Shane Kilkelly
da5a538095 Fix calls to PersistorManager.getFileStream which were missed in initial refactor. 
This should fix a breakage in the template publishing workflow.
Ref: b70b837b7ba29b0631f3105ec1dd5f6cdebbd195
 2015-09-08 09:23:08 +01:00
Shane Kilkelly
9b8f11f517 Fix the check for a success response from S3. 
We need to handle both a 200 and 206 (partial content) response.
 2015-09-01 12:25:10 +01:00
Shane Kilkelly
82af1be756 Improve testing of error conditions in the FSPersistorManager. 2015-09-01 12:10:30 +01:00
Shane Kilkelly
edc06c82e5 Correct usage of '==' rather than '=' when checking err.code. 2015-09-01 12:10:05 +01:00
Shane Kilkelly
8bbf81f4bc If the S3 service produces an error, handle it. 2015-09-01 11:36:28 +01:00
Shane Kilkelly
47fdf3c5c9 Ensure that Filestore responds with 404 when a file does not exist. 2015-08-31 16:47:16 +01:00
Shane Kilkelly
4d60872782 Merge branch 'master' of github.com:sharelatex/filestore-sharelatex 2015-08-28 14:48:25 +01:00
Shane Kilkelly
959191ae50 Add an acceptance test for using range header when getting a file. 2015-08-28 14:20:14 +01:00
Shane Kilkelly
ed980a21e1 Test the FileHandler layer. 2015-08-28 13:50:40 +01:00
Shane Kilkelly
64f70eddbd Add cluster.js to .gitignore. 2015-08-28 13:38:54 +01:00
Shane Kilkelly
8a631e70f7 Refactor, and log when we have a 'range' header. 2015-08-28 12:16:41 +01:00
Shane Kilkelly
387e96ab67 Refactor to make _get_range testable. 2015-08-28 12:02:50 +01:00
Shane Kilkelly
97cdc243d3 Parse out the range header and set start and end properties on options. 2015-08-28 11:45:16 +01:00
Shane Kilkelly
b905d9fba1 Add range-parser dependency. 2015-08-28 11:44:47 +01:00
Shane Kilkelly
76506ed91e Pass start/end options down to the S3PersistorManager. 
Test that those options cause a 'Range' header to be attached to the
request sent to S3.
 2015-08-28 10:09:41 +01:00
Shane Kilkelly
d88736e3b5 Pass the start and end options down to fs.createReadStream, and test for same. 2015-08-27 16:12:11 +01:00
Shane Kilkelly
57aedefdd3 Begin work towards obeying 'Range' headers, thread the opts down to persistence layer. 2015-08-27 15:21:33 +01:00
Brian Gough
bea9ee6b19 add gc to filestore 2015-08-18 15:36:07 +01:00
Brian Gough
2e08281b88 replace the /gc endpoint with a automatic gc every 3 minutes 2015-08-05 16:52:01 +01:00
Brian Gough
e1683b8e1c added a /gc endpoint to debug filestore memory usage 2015-08-04 11:26:51 +01:00
Xavier Trochu
f417b9b171 Fix the AWS-SDK Persistor 2015-07-24 15:48:46 +02:00
Xavier Trochu
7ef46a79a0 Fix aws-sdk persistor. 
Also fix some typos.
 2015-07-24 09:06:52 +02:00
Xavier Trochu
41397821b1 Add a manager using the aws-sdk library 
The knox library does not support the AWS Instance Role.
So use the official AWS SDK to connect to S3
 2015-07-22 16:42:45 +02:00
Xavier Trochu
3289451ece Revert "Make S3 Key/Secret pair optional." 
This reverts commit 65122e603099a4f9ef5c5186b7eb8832340a993f.

knox requires the key and secret arguments.
 2015-07-22 16:41:45 +02:00
Xavier Trochu
566e69c6cc Make S3 Key/Secret pair optional. 
When using Instance Role in AWS, the credentials are unneeded. So
make them optional to use.
 2015-07-21 10:00:59 +02:00
Xavier Trochu
c9b5bb028d Merge branch 'master' of https://github.com/sharelatex/filestore-sharelatex 2015-07-21 09:02:16 +02:00
Brian Gough
9b84422092 added event loop monitor 2015-06-23 13:41:14 +01:00
Brian Gough
8ebbacee5a move pipe after error handler setup 2015-05-08 15:19:45 +01:00
Brian Gough
7e9f8b885b add support for public files 2015-05-08 15:15:48 +01:00
Henry Oswald
d5f98c9800 make startup message consistent 2015-04-30 15:04:55 +01:00
Henry Oswald
33e654accf get the logging locally to work with bunyan 2015-04-22 18:27:34 +01:00
Henry Oswald
7c2e4b00ed no file subtype can be an issue, added a check on it in the keybuilder 2015-04-13 12:01:49 +01:00
Henry Oswald
158b640b59 Merge branch 'master' of https://github.com/sharelatex/filestore-sharelatex 2015-04-09 17:16:23 +01:00
Henry Oswald
b3432f016c allow proxying of template images though filestore 2015-04-09 17:14:47 +01:00
Brian Gough
07e53bbb2f handle error when listing bucket 2015-03-31 10:35:33 +01:00
Xavier Trochu
da1b7b9e82 Merge remote-tracking branch 'origin/master' 2015-03-23 16:25:18 +01:00
James Allen
801b5653e4 Fix unit test when false is not at /bin/false 2015-03-20 14:27:25 +00:00
James Allen
a370739722 Release version 0.1.4 2015-03-20 14:20:13 +00:00
Brian Gough
a5e63f7261 Merge pull request #18 from sharelatex/terminate-processes 
Use process group to terminate processes
 2015-03-13 11:32:57 +00:00
Brian Gough
143d44e54b add tests for SafeExec module 2015-03-13 10:15:53 +00:00
Brian Gough
a7b9376919 use SIGTERM instead of SIGKILL to allow process to shut down cleanly 2015-03-13 10:15:37 +00:00
Brian Gough
9aaef729ad cleanup and comments 2015-03-13 10:10:51 +00:00
Brian Gough
63ee4d1e7d use close event instead of exit to capture stdout/stderr correctly 2015-03-13 10:10:21 +00:00
Brian Gough
0e5abe1ff8 update tests to use safe_exec 2015-03-13 09:31:43 +00:00
Brian Gough
33d8974d02 kill process group on timeout of convert commands 2015-03-12 17:09:27 +00:00
James Allen
eb89337ada Release version 0.1.3 2015-02-26 11:32:21 +00:00
James Allen
45689fd2b8 Only call getFileStream callback once 2015-02-26 11:32:05 +00:00
James Allen
eb40dbbe96 Release version 0.1.2 2015-02-10 13:19:04 +00:00
Xavier Trochu
4b54372466 miscellaneous cleanup 
_ Remove some references to scribtex
_ Fix some issue when templates are unavailable
 2015-01-16 11:04:15 +01:00
Brian Gough
2c731586a8 Merge pull request #16 from sharelatex/cluster-restart 
Graceful cluster restart on domain exception
 2015-01-07 11:50:24 +00:00
Brian Gough
e03ee96216 added simple clustering master 2015-01-07 11:33:42 +00:00
Brian Gough
79292b4602 fix log message 2015-01-07 10:35:17 +00:00
Brian Gough
191d0a4986 handle SIGTERM in filestore 2015-01-07 10:35:17 +00:00
Brian Gough
02101868e1 put all shutdown logic in beginShutdown, force program to exit after 2 mins 2015-01-07 10:35:17 +00:00
Brian Gough
e8e2338c0c move connection:close middleware ahead of other routes 2015-01-05 16:56:19 +00:00
Brian Gough
d5478a8ebb make sure shutdown only happens once 2014-12-23 16:55:34 +00:00
Brian Gough
d3fa6b4a01 catch exceptions within the body of the domain error handler 2014-12-23 16:49:34 +00:00
Brian Gough
48f62990b4 server can exit immediately, does not need a domain 
exit with an exception as no requests can be served
 2014-12-23 16:48:11 +00:00
Brian Gough
74fc095913 close HTTP keep-alive connections when shutting down 2014-12-23 16:46:31 +00:00
Brian Gough
f4ac7d422b run acceptance tests on travis under forever 2014-12-22 13:41:07 +00:00
Brian Gough
ad379ec103 add grunt target for test:acceptance 2014-12-22 13:13:36 +00:00
Brian Gough
d4c281e22c add acceptance tests to travis.yml 2014-12-22 13:10:40 +00:00
Brian Gough
8b9213d09b fix getFileStream unit test as callback no longer fires immediately 
success callback is now called only when file is readable
 2014-12-22 13:07:30 +00:00
Brian Gough
efd6035849 test against rimraf module when deleting directory recursively 2014-12-22 13:01:39 +00:00
Brian Gough
3f18f6a714 update unit tests - FileHandler now calls deleteDirectory on cache directory 2014-12-22 12:58:41 +00:00
Brian Gough
64c6341844 check delete status code in copy test 2014-12-22 11:34:13 +00:00
Brian Gough
de465aee94 change copy test to match usage in web, check http status 2014-12-22 11:27:48 +00:00
Brian Gough
dad209262f execute success callback when filesystem copyfile completed 2014-12-22 11:27:00 +00:00
Brian Gough
72ebaea5e2 log FS delete errors only on failure 2014-12-22 11:11:31 +00:00
Brian Gough
7d96f3aadd use 'response' to send 'NoSuchKey' when local file is not found 2014-12-22 11:11:03 +00:00
Brian Gough
8744cf94a5 delete directory with rimraf 2014-12-19 16:58:31 +00:00
Brian Gough
af4c72150f fix path in directory deletion 2014-12-19 16:57:44 +00:00
Brian Gough
55c4e220fa prevent double callback in getFileStream 2014-12-19 16:28:42 +00:00
Brian Gough
c74356daeb Merge branch 'master' of github.com:sharelatex/filestore-sharelatex 
# Please enter a commit message to explain why this merge is necessary,
# especially if it merges an updated upstream into a topic branch.
#
# Lines starting with '#' will be ignored, and an empty message aborts
# the commit.
 2014-12-18 16:57:27 +00:00
Brian Gough
c2e50c619d request restart on unhandled exceptions to avoid memory leak 2014-12-18 16:55:11 +00:00
Henry Oswald
e90eb4b3fb bump knox 2014-11-29 13:02:06 +00:00
Henry Oswald
ca851589c9 Merge branch 'master' of https://github.com/sharelatex/filestore-sharelatex 
Conflicts:
	package.json
 2014-11-27 13:49:16 +00:00
Henry Oswald
dee7799fd4 added heapdump and endpoint to make dump 2014-11-27 13:48:39 +00:00
James Allen
9d97d1dc53 Handle templates locally ok 2014-11-27 11:08:29 +00:00
James Allen
72ad81887a Release version 0.1.0 2014-08-19 14:10:48 +01:00
James Allen
b79dfcd477 Lock down module versions 2014-08-19 14:10:37 +01:00
Henry Oswald
72df59e81e added copy file acceptence test 2014-06-19 17:32:04 +01:00
Henry Oswald
5d00f70a31 added delete accp test 2014-06-19 17:16:45 +01:00
Henry Oswald
f83ab25b35 basic acceptence test of sending and getting file back added 2014-06-19 17:07:50 +01:00
Henry Oswald
04bffaeceb get local file writer tests using fake settings 2014-06-17 13:33:36 +01:00
Henry Oswald
c8c758642a removed reference to easy image 2014-06-17 12:47:53 +01:00
Henry Oswald
088e494c30 Merge branch 'master' of github.com:sharelatex/filestore-sharelatex 2014-06-17 12:44:41 +01:00
Henry Oswald
8a052d4c71 added coffeescript to package.json 2014-06-17 12:44:10 +01:00
c4live
408bf43b98 Update FSPersistorManager.coffee 
Using code 'ENOENT' for error checking instead of errno 34.
 2014-06-05 16:01:38 +02:00
c4live
5bc89c3cbf Update FileHandler.coffee 
Restoring error check.
 2014-06-05 14:10:49 +02:00
c4live
1e42221954 Update FSPersistorManager.coffee 
Do not propagate the error if trying to delete a directory that does not exist.
 2014-06-05 14:10:05 +02:00
c4live
a9be97622b Update FileHandler.coffee: ignore any error that happens while deleting directory during insertFile. 2014-06-05 10:31:05 +02:00
James Allen
df6b5132c2 Tidy up testing/logging 2014-06-02 17:25:40 +01:00
James Allen
e17aaba3ba Catch errors correctly 2014-06-02 17:11:25 +01:00
James Allen
b3f796a093 More logging 2014-06-02 17:02:45 +01:00
James Allen
415d937f84 Add more logging and don't delete temp files 2014-06-02 16:49:49 +01:00
James Allen
e529875245 Revert "Send content-length header when getting file" 
This reverts commit 6f5f7a98fd3ce53091aacc744f43c7704ba73461.
 2014-05-23 13:57:18 +01:00
James Allen
daf4f1ffd4 Send content-length header when getting file 2014-05-23 13:54:20 +01:00
James Allen
de0d79e1bf Use configurable upload directory 2014-05-17 21:01:48 +01:00
Henry Oswald
1f1f535c8b removed lines which may be causing issue with domain 2014-05-12 18:33:09 +01:00
Henry Oswald
ce67c8c655 added logging to err domain 2014-05-12 15:57:08 +01:00
Henry Oswald
3bbeff5aff moved all metrics to use the module, deleting old metrics file and added different check in err domain 2014-05-12 15:45:15 +01:00
James Allen
76f929f415 Add in new http monitoring 2014-05-09 13:30:35 +01:00
Henry Oswald
09224007a7 on domain error wait 3 seconds to send 500 to allow for other cleanup. 
Also check inside function if the res has finished. #confused
 2014-05-07 09:06:05 +01:00
Henry Oswald
be8fdcfd6a added null check on send property of res 2014-05-07 08:23:44 +01:00
James Allen
d0816096b6 Use default settings file 2014-04-02 17:45:41 +01:00
James Allen
5e5f9cf4e5 Replace all instances of / in file paths in FSPersistorManager 2014-03-31 15:17:17 +01:00
Henry Oswald
191170509f bumped convert timeout to 40 seconds 2014-03-14 08:55:28 +00:00
Henry Oswald
41b32deac3 changed conversions to work with new image magick feature pdf:fit-page= 2014-03-13 14:04:46 +00:00
Henry Oswald
e8b245e857 check that res has not already been sent before sending catch 500 2014-03-05 18:13:40 +00:00
Henry Oswald
2f22563d59 changed key for templates to put /v/ in for version 2014-03-05 17:39:27 +00:00
Henry Oswald
1e1c14e5fe Merge pull request #11 from cwoac/FSPersistorManager 
Fs persistor manager - merge from cwoac
 2014-03-05 10:39:26 +00:00
Oliver Matthews
f920fd0b16 match refactor_config on cwoac/sharelatex 2014-03-04 15:24:50 +00:00
Oliver Matthews
957df0eb04 Don't dump streams to log files. 2014-03-04 14:45:32 +00:00
Henry Oswald
711e95a82a delete converted file when finished to clean up 2014-03-04 14:30:52 +00:00
Henry Oswald
94a6504635 cranked up kill to child processes to killSignal: "SIGKILL" 2014-03-04 12:44:16 +00:00
Oliver Matthews
40e2cb1c6d fix tabbing 2014-03-01 15:23:11 +00:00
Oliver Matthews
88cc89a0d1 filter /s from key ids 2014-03-01 15:10:47 +00:00
Oliver Matthews
7c5634044f Allow selection of FS persistor manager. Only require the chosen persistor manager. 2014-02-26 15:15:03 +00:00
Oliver Matthews
ff96f37b2d unit test passing version of FSPersistorManager 2014-02-26 15:10:55 +00:00
Oliver Matthews
53bcdccc04 change the method for assuming s3 2014-02-26 11:58:48 +00:00
Oliver Matthews
e0f0bfb0ea revert settings.persistorManager 2014-02-26 11:49:15 +00:00
Oliver Matthews
4b5f9ad4fb make s3 chosen if no config option specified 
rename *wrapper  to FilestoreManager
remove ToS3 from filestore commands
rename configuration option for filestore.
add details of interface to FilestoreWrapper.coffee for reference.
remove api comments
rename settings.filestoreManager to settings.persistorManager
 2014-02-26 10:24:51 +00:00
Henry Oswald
c9af5e5420 Renamed fsWrapper to PersistorManager 
not 100% happy with that still but its the best I can see so far
 2014-02-26 09:15:37 +00:00
Henry Oswald
49ffeb633d fix where cacheWarm param is read from and improved some logging 2014-02-26 09:06:47 +00:00
Henry Oswald
d41eca68a2 Merge pull request #8 from cwoac/localfs 
Allow multiple backends
 2014-02-25 16:37:16 +00:00
Oliver Matthews
5f87dfb3bf add unit tests for fsWrapper 
make other unit tests no longer depend on fsWrapper
 2014-02-25 15:52:30 +00:00
Oliver Matthews
c43533b5cb move test wrapper into fsWrapper where appropriate 2014-02-25 10:54:51 +00:00
Oliver Matthews
f68080d56c create mixin to allow switching of backend 2014-02-25 09:10:22 +00:00
James Allen
64fec8aed7 Update README.md 2014-02-24 14:44:03 +00:00
James Allen
49feb6b839 Update .travis.yml 2014-02-24 14:40:42 +00:00
James Allen
4756b488ee Create .travis.yml 2014-02-24 14:26:38 +00:00
goodbest
aa9f8882ee change repo URL from git+ssh:// to git+https:// 2014-02-21 23:09:38 +08:00
Henry Oswald
159670a3d4 reduced timeout to 20 seconds 2014-02-20 11:45:51 +00:00
Henry Oswald
99e15dc12f upped https connections to 300 and added debug logging 2014-02-19 21:03:03 +00:00
Henry Oswald
fada5ecaf4 converted get stream to use knox, test to see if it is more robust 2014-02-19 20:56:45 +00:00
Henry Oswald
3f54688ec8 added max sockets to s3 wrapper, getting some ETIMEDOUT under laod, may be cause 2014-02-19 18:39:25 +00:00
Henry Oswald
b24ae77f4c call callback when error returned from getign status of file 2014-02-19 15:26:00 +00:00
Henry Oswald
0a2ce3b869 fixed missleading log file 2014-02-19 15:24:29 +00:00
Henry Oswald
90ad2c24c6 call the callback when no response is returned from s3 when checking if file exists 2014-02-19 14:58:36 +00:00
Henry Oswald
960b46dda0 increased timeout to 4 mins.... 2014-02-19 14:48:16 +00:00
Henry Oswald
b85017759e added more logging 2014-02-19 14:48:16 +00:00
Henry Oswald
531b83a032 added more logging 2014-02-19 14:48:15 +00:00
Henry Oswald
d0600a4b64 changed optimisation to optipng and added timeouts to conversions 2014-02-19 14:48:15 +00:00
Henry Oswald
f63cdb3515 changed thumbnail and preview to be smaller 2014-02-19 14:48:15 +00:00
James Allen
f77906e01e Create README.md 2014-02-19 14:48:15 +00:00
James Allen
5d86dfacd7 Create LICENSE 2014-02-19 14:48:15 +00:00
Henry Oswald
f99a6dc39b changed file converted to use child process with nice 2014-02-19 14:48:12 +00:00
Henry Oswald
1ff5a5adb8 changed error log message 2014-02-17 18:50:28 +00:00
Henry Oswald
5b49ec1144 don't exit on error 2014-02-17 16:32:28 +00:00
Henry Oswald
7501046ebc improved logging 2014-02-17 16:31:33 +00:00
James Allen
110e623a0d Point npm modules at github 2014-02-17 15:09:53 +00:00
James Allen
fb02e40423 Standardize grunt file 2014-02-14 16:46:08 +00:00
James Allen
3199aad601 Initial open source commit 2014-02-14 16:39:05 +00:00