github/overleaf
1
0
Fork 0
mirror of https://github.com/overleaf/overleaf.git synced 2024-11-21 20:47:08 -05:00
Code Issues Projects Releases Packages Wiki Activity
23886 commits 3 branches 7 tags 206 MiB
Commit graph

717 commits

Author SHA1 Message Date
Jakob Ackermann
6f27f7a1bf [misc] drop the /heapdump route and related heapdump package 2020-03-02 17:09:35 +01:00
Jakob Ackermann
32557ab1d7 [SafeExec] replace _.once with lodash.once 2020-03-02 17:08:42 +01:00
Jakob Ackermann
0db4a17a14 [HealthCheckController] use fs.copyFile instead of fs-extra.copy 
the fs-extra method has a HUGE overhead of JS code and also syscalls for
 no particular benefit in this case: just copy the tiny.pdf file.

Here is an overview of the major operations: paths are relative to
  https://github.com/jprichardson/node-fs-extra/blob/1.0.0

We start in /lib/copy/copy.js
- sys: check that the source file exists
- sys: check that the source file has an existing parent directory?!
Continue in /lib/copy/ncp.js
- sys: more stat calls on both source and dest to determine permissions
- read/write streams to pipe the file content through the process
- sys: chmod on the destination to match the source permissions

What we actually need is a call to the binding and let node/the os
 figure out the best way to copy the contents.

Signed-off-by: Jakob Ackermann <das7pad@outlook.com>
 2020-03-02 17:02:35 +01:00
Simon Detheridge
c13d7f4197 Bump aws-sdk version in package.json 2020-03-02 11:18:06 +00:00
Brian Gough
ebabe52632 fix settings unit test 2020-02-28 15:27:29 +00:00
Brian Gough
e3011b2521 run npm update 2020-02-28 14:51:57 +00:00
Jakob Ackermann
b4e3d7de62 [config] add support for third party s3 which uses path-style buckets 
Signed-off-by: Jakob Ackermann <das7pad@outlook.com>
 2020-02-28 12:48:52 +01:00
Jakob Ackermann
2b9d0868c5 [misc] test/acceptance: retrieve ingress metrics just before using it 
The upload request can bump the ingress metric.
The content hash validation might require a full download
in case the ETag field of the upload response is not a md5 sum.
 2020-02-28 12:27:06 +01:00
Jakob Ackermann
516102e6fb [misc] test/acceptance: do not hard code fake credentials 2020-02-28 12:26:46 +01:00
Jakob Ackermann
847f124d7b [misc] test/acceptance: skip the shutdown in case we did not start yet 
Signed-off-by: Jakob Ackermann <das7pad@outlook.com>
 2020-02-28 12:26:24 +01:00
Jakob Ackermann
6589aa6ae0 [misc] test/acceptance: harden the startup check for s3 
Signed-off-by: Jakob Ackermann <das7pad@outlook.com>
 2020-02-28 12:26:24 +01:00
Jakob Ackermann
94b8a7f891 [misc] test/unit: KeybuilderTests: use a unique suite label 2020-02-28 12:26:23 +01:00
Jakob Ackermann
54e3b57772 [misc] test/unit: add missing globals that are lazy loaded 2020-02-28 12:26:23 +01:00
Jakob Ackermann
7920570dd8 [misc] test/unit: add missing require stubs for metrics and settings 2020-02-28 12:26:22 +01:00
Simon Detheridge
6e19a650d2 Write to error log on unhandled rejection/exception 2020-02-27 10:33:48 +00:00
Simon Detheridge
492f8abc9f Update config/settings.defaults.coffee 
Co-Authored-By: Jakob Ackermann <das7pad@outlook.com>
 2020-02-18 10:24:29 +00:00
Simon Detheridge
8acc9ef0ae Make S3 partSize a setting 2020-02-17 14:06:15 +00:00
Simon Detheridge
7c4bf97a7f Avoid fetching file a second time when source md5 can't be fetched 2020-02-17 14:06:05 +00:00
Simon Detheridge
9e7eec7a19 Add acceptance test for large files 2020-02-17 14:05:56 +00:00
Simon Detheridge
2f2a819b74 Re-fetch file to calculate md5 if etag is not in correct format 2020-02-17 14:05:43 +00:00
Simon Detheridge
5adfb3e2c0 Use large upload chunks 2020-02-17 14:05:31 +00:00
Simon Detheridge
4e507197fc Revert "Merge pull request #91 from overleaf/revert-78-spd-migration-persistor" 
This reverts commit 0cce792604c258538cf012486e7329fa1be7da0f, reversing
changes made to 08b606e22f855121b2d8a2832a5d7e2f45836c3f.
 2020-02-17 14:04:42 +00:00
Simon Detheridge
df2ddbe0e1 Revert "Add Migration Persistor, to send missing file requests to a fallback persistor" 2020-02-14 14:26:33 +00:00
Simon Detheridge
e4926e09a7 Move error handler to be the last middleware 2020-02-14 10:58:46 +00:00
dependabot[bot]
31f2572e57 Bump lodash from 4.17.11 to 4.17.15 
Bumps [lodash](https://github.com/lodash/lodash) from 4.17.11 to 4.17.15.
- [Release notes](https://github.com/lodash/lodash/releases)
- [Commits](https://github.com/lodash/lodash/compare/4.17.11...4.17.15)

Signed-off-by: dependabot[bot] <support@github.com>
 2020-02-14 09:39:06 +00:00
Simon Detheridge
485cfce6d3 Merge pull request #85 from overleaf/jpa-dependencies-cleanup 
[misc] rename npm-shrinkwrap.json to package-lock.json and run npm i
 2020-02-14 09:24:43 +00:00
Simon Detheridge
16d1674eab Merge pull request #84 from overleaf/bg-remove-travis-yml 
Remove unused .travis.yml file
 2020-02-14 09:21:03 +00:00
Jakob Ackermann
84a80930eb [misc] rename npm-shrinkwrap.json to package-lock.json and run npm i 2020-02-14 09:19:29 +00:00
Simon Detheridge
44fbc6a733 Merge pull request #82 from overleaf/spd-error-handler-error 
Fix log info assignment in error handler
 2020-02-14 09:18:16 +00:00
Simon Detheridge
3b011258d2 Tidy up md5 hash generation 2020-02-12 13:27:12 +00:00
Simon Detheridge
49ad408b30 Remove unused imports and format correctly 2020-02-12 13:27:12 +00:00
Simon Detheridge
93cd55fb79 Refactor persistors to use a helper for common things 2020-02-12 13:27:12 +00:00
Simon Detheridge
304fdfd35c Explicitly resume stream after adding listener 2020-02-12 13:21:53 +00:00
Simon Detheridge
6dcf351377 Remove unnecessary 'async' 2020-02-12 13:21:53 +00:00
Simon Detheridge
f4a16cd972 Update tests to properly check for copied files 2020-02-12 13:21:53 +00:00
Simon Detheridge
42a5d168df Remove unused packages 2020-02-12 13:21:52 +00:00
Simon Detheridge
9e0b378948 Remove minipass as dependency and refactor to make things clearer 2020-02-12 13:21:52 +00:00
Simon Detheridge
5d5d325691 Preserve all error information when cleanup of copied file fails 2020-02-12 13:21:52 +00:00
Simon Detheridge
b4b7fd226e Add mechanisms to transfer files with md5-based integrity checks 
Fix error in settings and tidy up tests

Remove unused variable declaration

Remove .only from tests and update eslint rules to catch it in future

Use  to catch errors more safely getting md5 hash

Avoid unnecessary call to S3 to get md5 response
 2020-02-12 13:21:52 +00:00
Simon Detheridge
2625e03a31 Add MigrationPersistor for sending 404 requests to a fallback persistor 2020-02-12 13:19:48 +00:00
Simon Detheridge
f877f51775 Rename *PersistorManager to *Persistor 2020-02-12 13:19:48 +00:00
Brian Gough
a2e1591e93 remove unused .travis.yml file 2020-02-12 12:37:00 +00:00
Simon Detheridge
e3613e52db Fix log info assignment in error handler 2020-02-12 10:34:56 +00:00
Jakob Ackermann
1341af04f5 [misc] update the build scripts to 1.3.5 2020-02-11 18:58:55 +01:00
Jakob Ackermann
ca94225c19 Merge pull request #80 from overleaf/jpa-cleanup-install-deps 
[misc] cleanup install deps
 2020-02-10 17:16:44 +01:00
Brian Gough
14ba1ea1c1 update to node 10.19.0 2020-02-07 14:16:31 +00:00
Jakob Ackermann
ac4361d560 [misc] dockerignore: hide the local data dirs 2020-01-31 15:50:53 +01:00
Jakob Ackermann
c1d81dc1ea [misc] install a maintained ghostscript version and delete package lists 
also do not install vim and bail out in case any command exited with a
 non zero exit code.
 2020-01-31 15:50:53 +01:00
Simon Detheridge
997434c930 Remove redundant 'res.logInfo' 2020-01-14 20:55:33 +00:00
Simon Detheridge
035364b62e Fix RequestLogger object lifecycle 2020-01-14 17:15:01 +00:00
Simon Detheridge
28fb998719 Refactor request logger into a class 2020-01-14 12:02:39 +00:00
Simon Detheridge
9615a06e0f Improve logging middleware to add info methods to request object 2020-01-10 17:22:06 +00:00
Simon Detheridge
f40fbe77aa Simplify logging to log only once per http request 2020-01-10 15:41:36 +00:00
Simon Detheridge
e331a3130c Handle AccessDenied and stream-premature-close errors 
These errors tend to occur as part of normal operation and should not generate `logger.err` messages
 2020-01-09 14:15:01 +00:00
Simon Detheridge
d8470d4a59 Use correct object path for s3BucketCreds 2020-01-08 11:32:05 +00:00
Simon Detheridge
3bf51cac67 Improve naming on internal '_client' method and use Map over object 2020-01-08 09:17:30 +00:00
Simon Detheridge
80d41cf51b Move bucket-specific file endpoint into FileController 2020-01-07 15:05:51 +00:00
Simon Detheridge
6cc5d94f13 Re-add bucket-specific credentials support for S3 2020-01-07 10:55:00 +00:00
Simon Detheridge
2bb2caf7b3 Clean up settings tests 2020-01-07 09:46:53 +00:00
Simon Detheridge
ffd002dc76 Add explanatory comment 2020-01-07 09:30:30 +00:00
Simon Detheridge
b642b3cf3a Rename incorrectly-named 'getDirectorySize' method 2020-01-07 09:30:30 +00:00
Simon Detheridge
a6d9d34fe3 Configure PersistorManager to use the new S3 manager for 'aws-sdk' 2020-01-07 09:30:30 +00:00
Simon Detheridge
2ca74fdf15 Cleanup and refactor S3PersistorManager to use aws-sdk only 2020-01-07 09:30:30 +00:00
Simon Detheridge
473aea4e60 Remove AWSSDKPersistorManager 2020-01-07 09:30:30 +00:00
Simon Detheridge
e27cf4db7b Pass pipeline errors onto 'next' error handler 2020-01-07 09:30:30 +00:00
Simon Detheridge
87b51d0c35 Remove old exception-handling and shutdown-related mechanisms 2020-01-07 09:30:28 +00:00
Simon Detheridge
74b480fc55 Post-decaf cleanup of app.js 2020-01-07 09:29:33 +00:00
Simon Detheridge
85d3c0a852 Remove old exception-handling and shutdown-related mechanisms 2020-01-07 09:27:02 +00:00
Simon Detheridge
6a679023d3 Fix order of .status().send() 2020-01-07 09:27:02 +00:00
Simon Detheridge
fce275e1d4 Post-decaf cleanup of app.js 2020-01-07 09:27:02 +00:00
Simon Detheridge
0329c759dc Remove wrapper function in PersistorManager 2020-01-07 09:26:43 +00:00
Simon Detheridge
dbfacce988 Post-decaf cleanup of PersistorManager 2020-01-07 09:26:43 +00:00
Simon Detheridge
10f7087912 Post-decaf cleanup of KeyBuilderTests 2020-01-07 09:26:43 +00:00
Simon Detheridge
ec80052d9a Decaf cleanup of FileController 
Remove deprecated res.send
 2020-01-07 09:26:23 +00:00
Simon Detheridge
37d44d5d86 Return full path in stubbed 'glob' method 2020-01-07 09:26:01 +00:00
Simon Detheridge
fccb288985 Add comment regarding deleteDirectory being internal-only 2020-01-07 09:26:01 +00:00
Simon Detheridge
f1b6b35c69 Throw errors more consistently in FSPersistorManager 2020-01-07 09:26:00 +00:00
Simon Detheridge
25f1c2bfc4 Delete temporary file when error in writing to stream 2020-01-07 09:26:00 +00:00
Simon Detheridge
039bec02f7 Use directory names in acceptance tests, and add test for getting size 2020-01-07 09:26:00 +00:00
Simon Detheridge
21ba083877 Use 'glob' to determine 'directory' size. 
Check for filtered directory paths in all tests.
 2020-01-07 09:26:00 +00:00
Simon Detheridge
ce90292394 Decaf cleanup for FSPersistorManager 2020-01-07 09:26:00 +00:00
Simon Detheridge
4315824d3c Minor cleanup of FilestoreApp acceptance-test helper 2020-01-07 09:25:04 +00:00
Simon Detheridge
a8158d6c8c Cleanup and promisify acceptance tests. Run tests for all backends. 2020-01-07 09:25:04 +00:00
Simon Detheridge
006f84abeb Decaf and promisify ImageOptimiser 2020-01-07 09:24:29 +00:00
Simon Detheridge
42adc59d01 Clean up and promisify health-check controller and KeyBuilder 2020-01-07 09:23:41 +00:00
Simon Detheridge
ac2d05ecb3 Remove unnecessary logging from FileConverter 2020-01-07 09:22:53 +00:00
Simon Detheridge
b6a1ea6a30 Cleanup and promisify FileConverter 2020-01-07 09:22:53 +00:00
Simon Detheridge
a216be5fd3 Improve error handling in FileHandler 2020-01-07 09:18:59 +00:00
Simon Detheridge
27aaff7843 Decaf cleanup for FileHandler and LocalFileWriter 
Simplified code and tests where possible
 2020-01-07 09:18:56 +00:00
Simon Detheridge
eacad77112 Cleanup SafeExec 2019-12-19 15:42:46 +00:00
Simon Detheridge
f19d82369d Bump eslintrc up again to last-known-compatible version with eslit-prettier 2019-12-18 09:52:18 +00:00
Simon Detheridge
b94df9952b Add OError 2019-12-18 09:34:25 +00:00
Simon Detheridge
42f6d10031 Remove mongo and redis from dependencies 2019-12-17 15:19:19 +00:00
Simon Detheridge
0b3b71b857 Fall back to old AWS env vars if present 2019-12-17 09:57:51 +00:00
Simon Detheridge
b7e098208f Update build files from build_scripts 2019-12-16 17:17:25 +00:00
Simon Detheridge
29f333d60c Placate eslint 2019-12-16 15:38:00 +00:00
Simon Detheridge
7870c8185e Add linting to makefile 2019-12-16 11:58:22 +00:00
Simon Detheridge
230baef8fa Add linting tasks to package.json 2019-12-16 11:32:58 +00:00
Simon Detheridge
35d050a49c Prettier fixes 2019-12-16 11:32:46 +00:00
Simon Detheridge
6974fd640d Remove unusued cluster.coffee 2019-12-16 11:29:25 +00:00
Simon Detheridge
495424a94b Fix metrics includes in SandboxedModule 2019-12-16 11:28:24 +00:00
Simon Detheridge
eae7d28495 Prettier: convert test/unit decaffeinated files to Prettier format 2019-12-16 11:20:29 +00:00
Simon Detheridge
ec60f778e6 Rename test/unit/coffee to test/unit/js 2019-12-16 11:20:27 +00:00
decaffeinate
ff04085bdd decaffeinate: Run post-processing cleanups on AWSSDKPersistorManagerTests.coffee and 12 other files 2019-12-16 11:20:25 +00:00
decaffeinate
5cf7138ef1 decaffeinate: Convert AWSSDKPersistorManagerTests.coffee and 12 other files to JS 2019-12-16 11:20:22 +00:00
decaffeinate
72c83bdaff decaffeinate: Rename AWSSDKPersistorManagerTests.coffee and 12 other files from .coffee to .js 2019-12-16 11:20:20 +00:00
Simon Detheridge
e06c03e536 Fix decaf error in FSPersistorManager 2019-12-16 11:19:56 +00:00
Simon Detheridge
0b6e725137 Remove coffeescript compiles from package.json 2019-12-16 11:19:56 +00:00
Simon Detheridge
9943e2cc8f Prettier: convert individual decaffeinated files to Prettier format 2019-12-16 11:19:56 +00:00
Simon Detheridge
1d295ef7e4 Decaffeinate: convert individual files to js 2019-12-16 11:19:55 +00:00
Simon Detheridge
d70a33c4f5 Rename individual coffee files to js files 2019-12-16 11:19:43 +00:00
Simon Detheridge
e4b1106761 Prettier: convert test/acceptance decaffeinated files to Prettier format 2019-12-16 11:19:43 +00:00
Simon Detheridge
b8e7abd25e Rename test/acceptance/coffee to test/acceptance/js 2019-12-16 11:19:43 +00:00
decaffeinate
738e1d821e decaffeinate: Run post-processing cleanups on FilestoreApp.coffee and 1 other file 2019-12-16 11:19:43 +00:00
decaffeinate
e6e75d4e19 decaffeinate: Convert FilestoreApp.coffee and 1 other file to JS 2019-12-16 11:19:43 +00:00
decaffeinate
1c7d1af4ca decaffeinate: Rename FilestoreApp.coffee and 1 other file from .coffee to .js 2019-12-16 11:19:43 +00:00
Simon Detheridge
6225f2f236 Prettier: convert app/js decaffeinated files to Prettier format 2019-12-16 11:19:36 +00:00
Simon Detheridge
601861ee58 Remove js from dockerignore 2019-12-16 10:54:12 +00:00
Simon Detheridge
3718c90c7a Rename app/coffee dir to app/js 2019-12-16 10:54:08 +00:00
decaffeinate
fcbb13b5bf decaffeinate: Run post-processing cleanups on AWSSDKPersistorManager.coffee and 13 other files 2019-12-16 10:54:06 +00:00
decaffeinate
6bd8452f19 decaffeinate: Convert AWSSDKPersistorManager.coffee and 13 other files to JS 2019-12-16 10:54:03 +00:00
decaffeinate
8d2c87420e decaffeinate: Rename AWSSDKPersistorManager.coffee and 13 other files from .coffee to .js 2019-12-16 10:53:59 +00:00
Simon Detheridge
ae3d25c969 Decaffeinate: add eslint and prettier rc files 2019-12-16 10:53:56 +00:00
Simon Detheridge
1798efd4be Install eslint and prettier 2019-12-16 10:53:52 +00:00
Simon Detheridge
c9106eff95 Upgrade Docker image to node 10 2019-12-16 10:53:42 +00:00
Jakob Ackermann
ed97fcfcd0 [logging] do not overwrite the logger name 
Signed-off-by: Jakob Ackermann <das7pad@outlook.com>
 2019-12-16 10:53:34 +00:00
Jakob Ackermann
c5e1584fcc [FSPersistorManager] fix the stream opening for node10+ 
Attaching a `readable` listener causes the stream to hang otherwise.

Signed-off-by: Jakob Ackermann <das7pad@outlook.com>
 2019-12-16 10:53:23 +00:00
Simon Detheridge
dab44f6fec Merge branch 'master' into logging/keep-logger-name 2019-12-16 09:42:18 +00:00
Simon Detheridge
32b7f25404 Merge pull request #49 from das7pad/misc/node10plus-support 
[misc] add support for node 10+
 2019-12-16 09:39:00 +00:00
Simon Detheridge
56b38af678 Explicitly wait for S3 in acceptance tests 2019-12-10 17:43:34 +00:00
Simon Detheridge
237c4113cd Ensure fakes3 is healthy before running tests 2019-12-10 16:36:33 +00:00
Simon Detheridge
cf684dcd98 Fix fakes3 configuration in ci yml 2019-12-10 11:38:19 +00:00
Simon Detheridge
6f326d5650 Use SSL setting based on url protocol 2019-12-09 17:41:20 +00:00
Simon Detheridge
48aa141591 Add metric for s3 ingress 2019-12-09 17:06:00 +00:00
Simon Detheridge
1d1106bc67 Add metric for s3 egress 2019-12-09 17:06:00 +00:00
Simon Detheridge
c01603b1e7 Support custom S3 endpoints 2019-12-09 17:06:00 +00:00
Simon Detheridge
96457597ac Add fake s3 server and initial config 2019-12-09 17:06:00 +00:00
Simon Detheridge
86b9e4b53a Rename request -> s3Request to prevent overwriting main import 2019-12-09 17:05:58 +00:00
Brian Gough
2ec38068aa add comments about aws-sdk and s3 backends 2019-12-06 14:35:13 +00:00
Nate Stemen
aba0d14edd bump build script to 1.1.24 2019-10-25 11:52:54 -04:00
Nate Stemen
4798589213 replace private link with public one 2019-10-25 11:51:58 -04:00
Shane Kilkelly
dbede8509c Merge pull request #54 from overleaf/sk-dep-upgrades-2019-06 
update logger and metrics
 2019-07-09 10:18:16 +01:00
Eric Mc Sween
7d900b57bf Fix createUnbufferedStream() function call 
In 49a21155f642670dfea264ac73fb60241f37cb87, I managed to incorrectly
write the `createUnbufferedStream()` function from the AWS SDK as
`getUnbufferedStream()` and to consistently use that naming in the unit
tests.

This commit fixes that. I have tested again on S3.
 2019-06-19 12:58:17 -04:00
Shane Kilkelly
013400d7a4 Re-add environment variable for conversions 2019-06-19 14:04:57 +01:00
Shane Kilkelly
f865762c29 update logger and metrics 2019-06-19 13:50:58 +01:00
Eric Mc Sween
f2521a29b9 Use AWS SDK for getFileStream() 
The AWS SDK has a retry strategy to deal with rate limiting or transient
unavailability of S3. We hope it will reduce our error rates.
 2019-06-19 08:18:11 -04:00
Eric Mc Sween
3575c89d03 Return file size on HEAD request 
This will be used by the file preview feature when it gets partial
content.
 2019-06-17 08:23:59 -04:00
Timothée Alby
f081546ec0 update Git URL in Jenkinsfile 2019-05-07 17:37:07 +01:00
Timothée Alby
32a54a7e37 Update README.md 2019-05-07 17:22:35 +02:00
Jakob Ackermann
a79adcd325 [logging] do not overwrite the logger name 
Signed-off-by: Jakob Ackermann <das7pad@outlook.com>
 2019-05-02 12:48:43 +02:00
Jakob Ackermann
c59a3db4e8 [FSPersistorManager] fix the stream opening for node10+ 
Attaching a `readable` listener causes the stream to hang otherwise.

Signed-off-by: Jakob Ackermann <das7pad@outlook.com>
 2019-05-02 02:47:53 +02:00
Brian Gough
b9d3b1a842 fix beginShutdown to use server.close() 2019-03-15 15:35:28 +00:00
Henry Oswald
37b039abf0 add sentry to settings.defaults.coffee 2019-03-05 17:20:16 +00:00
Henry Oswald
2adae20368 remove a console.log 2019-02-06 11:43:11 +00:00
Henry Oswald
0f48c77e7e enable conversions for local docker compose 2019-02-05 17:19:33 +00:00
Henry Oswald
5eec00db51 don't run conversions test when disabled 2019-02-05 14:06:51 +00:00
Henry Oswald
3fccf79ca8 cleanup 2019-02-05 12:50:08 +00:00
Henry Oswald
ca9231b04e set enable conversions to false by default but enable in compse 2019-02-05 11:19:18 +00:00
Henry Oswald
19007befb2 Merge branch 'master' into ho-docker 2019-02-01 16:26:45 +00:00
Henry Oswald
ea845f4935 bulk upgrade to 1.1.12, moved to npm rather than git 2019-01-31 16:01:20 +00:00
Brian Gough
9d93eee3e8 return a 404 error (instead of a 500) when copying a missing file 2019-01-09 10:31:59 +00:00
Henry Oswald
3614f217e6 add enableConversions flag to disable conversions which we can't do on 
k8
 2019-01-07 15:54:24 +00:00
Christopher Hoskin
0421192449 Ignore any file ending in .map, wherever it is 2019-01-04 16:22:13 +00:00
Christopher Hoskin
c8d146b571 Bump settings to v1.1.0 2019-01-04 16:03:45 +00:00
Christopher Hoskin
b522fe1531 Remove grunt 2019-01-04 16:03:22 +00:00
Christopher Hoskin
6763809748 Bump logger to v1.5.9 2019-01-03 16:27:45 +00:00
Christopher Hoskin
594b403bd3 Move metrics init to start of app.coffee 2019-01-02 09:55:31 +00:00
Christopher Hoskin
3b6eba33ba Remove Metrics.inc "startup" from app.coffee 2019-01-02 09:53:53 +00:00
Christopher Hoskin
aab5a1af6a Merge branch 'master' into csh-ho-docker 2019-01-02 09:51:35 +00:00
Christopher Hoskin
6214256568 Bump logger to v1.5.8 2019-01-02 09:47:38 +00:00
Brian Gough
cf12ec1154 use the aws sdk to copy files in S3PersistorManager 
to work around problems with knox
 2018-12-20 16:11:18 +00:00
Brian Gough
28271655e1 avoid problems with caching of apt-get operations 2018-12-06 08:19:39 +00:00
Henry Oswald
25275212af bump metrics to 2.0.7 2018-12-05 12:36:32 +00:00
Henry Oswald
5d927d30b6 update build scripts to 1.1.10 2018-12-05 10:53:11 +00:00
Henry Oswald
8743527452 bump metrics to 2.0.4 2018-11-29 16:42:26 +00:00
Henry Oswald
e64156cd27 clean npm-shrinkwrap and package.json 2018-11-29 16:41:30 +00:00
Henry Oswald
95f0a02dee stub out metrics in tests & inject metrics endpoint 2018-11-29 13:53:43 +00:00
Henry Oswald
ee1c14a410 bump metrics 2018-11-29 12:46:14 +00:00
Brian Gough
fdc5370f94 Merge pull request #36 from sharelatex/bg-upgrade-logger 
upgrade logger-sharelatex to v1.5.8
 2018-11-13 14:06:44 +00:00
Brian Gough
8951f297aa Merge pull request #37 from sharelatex/bg-reduce-logging-of-not-found-errors 
reduce logging of not found errors
 2018-11-13 14:06:19 +00:00
Brian Gough
0d6de10b5c Merge pull request #34 from sharelatex/bg-make-backends-consistent 
make filestore backends consistent
 2018-11-13 10:12:18 +00:00
Brian Gough
e930c7106b Merge pull request #35 from sharelatex/bg-serve-converted-file-from-disk 
serve file from disk to avoid read-after-write inconsistency
 2018-11-13 10:12:00 +00:00
Brian Gough
23a6d6e81d consider 403 and 404 as NotFound errors 2018-11-09 14:05:38 +00:00
Brian Gough
69b164092c suppress unnecessary error logging for NotFound 2018-11-09 14:05:38 +00:00
Brian Gough
c2bfdd095c upgrade logger-sharelatex to v1.5.8 2018-11-08 16:29:23 +00:00
Brian Gough
8744b3aa4e serve file from disk to avoid read-after-write inconsistency 2018-11-08 15:09:34 +00:00
Brian Gough
78ccf453db update unit tests 2018-11-07 16:51:06 +00:00
Brian Gough
40ac59090b make backends consistent by deleting temporary files 2018-11-07 16:22:09 +00:00
Brian Gough
06bfd4c270 add dockerfile to install packages needed by filestore 2018-11-07 16:12:49 +00:00
Christopher Hoskin
038b3476fe Add some details to the README 2018-10-22 14:09:56 +01:00
Henry Oswald
38f565d35f logout settings 2018-09-28 15:53:57 +01:00
Henry Oswald
aacb54d54c Merge branch 'master' into ho-docker 2018-09-14 17:16:18 +01:00
Brian Gough
96e9d8b0ec fix typo 2018-09-11 10:51:10 +01:00
Brian Gough
f3adf5a653 fix shrinkwrap for upgraded logger-sharelatex module 2018-09-06 17:01:40 +01:00
Brian Gough
f4ec677296 add sentry to logging 2018-09-06 17:01:18 +01:00
Christopher Hoskin
53dacc04ef Bump metrics from 1.3.0 to 1.8.1 2018-07-26 16:06:30 +01:00
Christopher Hoskin
0e9a535e1c Bump build script to 1.1.9 2018-07-26 16:05:11 +01:00
Henry Oswald
cba013cd71 change ordering of docker rmi 2018-07-23 17:06:31 +01:00
Christopher Hoskin
f9f599e3e7 Push container images to overleaf-ops project 2018-07-18 11:30:01 +01:00
Christopher Hoskin
1044069535 Remove csh-gcdm-test and csh-staging repos 2018-07-17 12:03:14 +01:00
Christopher Hoskin
c7733d05cc Remove trailing / from project name 2018-07-17 11:06:33 +01:00
Christopher Hoskin
d182e5329c Add cr-test2 GCR project 2018-07-17 10:52:20 +01:00
Christopher Hoskin
4784ae33a2 Bump build scripts to 1.1.8 2018-07-17 10:51:15 +01:00
Henry Oswald
1e4e68c64a fix settings test 2018-07-10 14:55:40 +01:00
Henry Oswald
089bf3e084 seperate the standard s3 creds from the bucket s3 creds 2018-07-10 14:17:37 +01:00
Michael Mazour
c4e3f9eb02 Amend: tests for populating S3 settings from environment variable 2018-07-06 11:05:40 +01:00
Michael Mazour
3e1ef3af63 Populate S3 settings from environment variable 2018-07-06 11:05:19 +01:00
Michael Mazour
2da15f2eb3 Amend - improve documentation of settings 2018-07-06 10:28:02 +01:00
Michael Mazour
7feafccf31 Amend: safely navigate to bucket credentials 2018-07-06 09:28:09 +01:00
Michael Mazour
03033409c6 Amend: remove unused params 2018-07-06 09:12:59 +01:00
Michael Mazour
336a38ec1e Amend: scrub secrets from logs 
Calls to `getFile` can now include S3 credentials in `opts`, so sanitize before writing to opts to log.
 2018-07-04 16:41:31 +01:00
Michael Mazour
600ab3ce67 Amend: remove problematic ampersand 2018-07-04 16:39:41 +01:00
Michael Mazour
cfbf0d81ba Amend: fix params retrieval 2018-07-04 12:11:09 +01:00
Michael Mazour
ece650741a Amend per several review comments 
- Removed unused vars
- Label the metric with the bucket name
 2018-07-04 12:02:09 +01:00
Michael Mazour
feca8933f1 Add endpoint for arbitrary bucket fetch 
Add `/bucket/:bucket/key/*`, which fetches the file from the given bucket at the given path. Uses auth stored at `settings.filestore.s3.{{bucketName}}` if present, and otherwise default auth.
 2018-07-04 11:22:36 +01:00
Christopher Hoskin
f0cf8e6622 Fix typo in buildscript 2018-06-14 11:37:18 +01:00
Christopher Hoskin
46c61aa78c Update build scripts to 1.1.7, add csh-staging GCR 2018-06-13 16:21:32 +01:00
Henry Oswald
1763715004 remove black package-lock.json 2018-05-22 13:45:32 +01:00
Henry Oswald
ba8a5cf3e1 bump to 1.1.3 scripts 2018-05-22 13:45:01 +01:00
Henry Oswald
58ab4c72dc update build scripts to 1.1.1 2018-05-21 15:14:24 +01:00
Henry Oswald
ee81a6af00 updated mocha and added --exit back in 2018-05-21 15:00:32 +01:00
Henry Oswald
8efd562eb3 fix broken acceptence tests, bad quotes in file path 2018-05-21 13:40:03 +01:00
Henry Oswald
6657b6c1dc make all needed directories in install deps 2018-05-21 11:31:14 +01:00
Henry Oswald
e3514e84cd fix up mocha migration tests 2018-05-21 11:02:53 +01:00
Henry Oswald
768a4cc1ad remove console.log from settings 2018-05-21 10:34:16 +01:00
Henry Oswald
09527d804e ignore js acceptence tests 2018-05-21 10:21:13 +01:00
Henry Oswald
3cf2e1d7c3 add null checks and new body parser 2018-05-21 10:11:47 +01:00
Henry Oswald
ac990f2aba remove app.configure 2018-05-21 09:49:32 +01:00
Henry Oswald
d9eac8dd8b upgrade mocha 2018-05-21 09:41:14 +01:00
Henry Oswald
fda6cb0084 fix acceptence tests, add them back into jenkins, upgrade express 2018-05-21 09:37:54 +01:00
Henry Oswald
df6b5203a1 point ci to gcr.io/csh-gcdm-test/ 2018-05-18 17:18:39 +01:00
Henry Oswald
28fc39d6c9 don't use quay.io in image name 2018-05-18 17:08:18 +01:00
Henry Oswald
bfb98050e3 don't push to quay.io 2018-05-18 16:57:53 +01:00
Henry Oswald
088d8850ba don't run acceptence tests yet 2018-05-18 16:48:59 +01:00
Henry Oswald
fcb2d18f8b don't set command in make test acceptence 2018-05-18 16:15:44 +01:00
Christopher Hoskin
b32c63e16c Remove --exit option to support older mocha 2018-05-18 14:59:34 +01:00
Christopher Hoskin
7fc7252c35 Remove touch package-lock.json 2018-05-18 14:29:20 +01:00
Christopher Hoskin
8db861258a Add files generated by bin/update_build_scripts 2018-05-18 14:27:46 +01:00
Christopher Hoskin
5d5cbbef7e Create fake package-lock.json during build tagret 2018-05-18 14:15:02 +01:00
Christopher Hoskin
a2165d85a0 Update build script to use build-target docker 2018-05-18 13:55:33 +01:00
Christopher Hoskin
de74ec5094 Change build and publish to use the GCR of one of my projects (hard code for now) 2018-05-18 13:47:06 +01:00
Henry Oswald
46f2ff051c cd into tmp 2018-03-23 17:29:21 +00:00
Henry Oswald
b45e2a922c install gs correctly 2018-03-23 17:21:15 +00:00
Henry Oswald
8c782ba6b2 debuging 2018-03-23 17:12:37 +00:00
Henry Oswald
801adf8e9e install gs manually 2018-03-23 17:06:31 +00:00
Henry Oswald
092a151d51 Use new build system 2018-03-23 16:36:41 +00:00
Henry Oswald
837a8ff35f change uploads permission after creation of user 2018-03-23 16:26:42 +00:00
Henry Oswald
1af209cd24 chown uploads 2018-03-23 16:23:04 +00:00
Henry Oswald
2f0cc4e64a chown uploads 2018-03-23 16:19:52 +00:00
Henry Oswald
73f77b5667 copy everything into docker container 2018-03-23 16:14:30 +00:00
Henry Oswald
9e321658ed make s3 configurable via env vars 2018-03-23 15:29:24 +00:00
Henry Oswald
77e34d33df hello james 2018-01-16 17:00:27 +00:00
Henry Oswald
08c734e850 another bump 
git log
 2018-01-16 14:48:21 +00:00
Henry Oswald
dbfebc0d1c bump 2018-01-16 14:36:09 +00:00
Henry Oswald
e235651e9c bump version 2018-01-16 12:44:26 +00:00
Henry Oswald
c23b700927 wip 2018-01-16 12:27:36 +00:00
James Allen
b834e53e13 Provide hosts as environment settings and add npm run start script 2017-12-29 08:14:23 +00:00
Joe Green
a70a216c52 Update Jenkinsfile 2017-10-16 14:10:06 +01:00
Joe Green
6dcef9fdea alert only on master 2017-10-12 17:07:06 +01:00
Henry Oswald
0e2de1e5ad added shrinkwrap file 2017-09-15 14:57:01 +01:00
Joe Green
bfef9a9d67 added build.txt 2017-09-04 14:49:14 +01:00
Joe Green
706dc917dd Create Jenkinsfile 2017-08-11 11:12:32 +01:00
Shane Kilkelly
e162c779d3 Merge pull request #24 from sharelatex/sk-ignore-png-optimiser-timeout 
If optipng is sigkilled by timeout, then ignore that error
 2017-08-03 09:22:43 +01:00
Shane Kilkelly
eaa72d0747 upgrade to node 6 2017-06-27 14:53:20 +01:00
Shane Kilkelly
23fe93ccfd Prevent double-logging 2017-06-22 15:28:08 +01:00
Shane Kilkelly
ba9477cdb3 Re-work error checking 2017-06-22 09:41:36 +01:00
Shane Kilkelly
d0bbe8beae If optipng is sigkilled by timeout, then ignore that error 2017-06-21 15:18:36 +01:00
Shane Kilkelly
2f8c693be3 Add a .nvmrc file 2017-03-27 14:49:25 +01:00
Shane Kilkelly
b12f69722f Begin fixing health-check 2017-03-08 14:59:34 +00:00
Henry Oswald
e0f39da616 null check fsPath on unlink file 2016-12-19 14:48:09 +00:00
Henry Oswald
9e8c27f46b don’t shutdown filestore on uncaught exception 2016-12-19 14:47:54 +00:00
Henry Oswald
bf93fc3f50 run health check max 1 a at a time 2016-12-19 13:26:21 +00:00
Henry Oswald
fd99887753 running check in background 2016-12-19 12:29:08 +00:00
Henry Oswald
6c81d18627 bumped up rlimit-nproc to 600 2016-12-14 18:15:09 +00:00
Henry Oswald
d41bf2e974 result path not in final cb 2016-12-14 18:11:28 +00:00
Henry Oswald
4bba56ed28 added missing files 2016-12-14 17:14:15 +00:00
Henry Oswald
27139bc433 increased rlimit-nproc and added health check for converting image 2016-12-14 16:47:53 +00:00
Henry Oswald
b166c86197 added missing originalFsPath 2016-12-13 14:29:04 +00:00
Henry Oswald
b38e7d3078 * delete both file and converted file if there is an error 
* increase timeout of optpng
 2016-12-13 12:43:57 +00:00
Shane Kilkelly
4064956cea working minimal profile 2016-09-28 10:05:27 +01:00
Shane Kilkelly
edb5987c3d A working profile, tested. 2016-09-27 15:54:41 +01:00
Shane Kilkelly
ff39a9da22 sort the list of syscalls 2016-09-26 15:35:53 +01:00
Shane Kilkelly
21be88482b Change the seccomp whitelist to match that of docker default list 2016-09-26 15:33:22 +01:00
Brian Gough
fd95599609 remove old firejail includes 2016-08-11 11:52:58 +01:00
James Allen
a95aae16e6 Tweak convert profile to work on api-2 2016-05-10 17:50:46 +01:00
James Allen
a41a307b58 Update firejail profile to allow pdf previews for templates to work 2016-05-10 14:34:48 +01:00
James Allen
92f5234580 Add convert firejail profile 2016-05-09 14:44:49 +01:00
James Allen
be43330208 Allow convert command to be prefixed by security commands 2016-05-09 11:37:35 +01:00
Henrique Santos
ffaac4c81b add unit tests 2016-03-13 20:45:48 -03:00
Henrique Santos
0bb12f32fa remove project controller / handler 
add unit test scaffolding
 2016-03-13 16:22:14 -03:00
Henrique Santos
1f20744a68 add size function for each persitor 2016-03-12 11:08:07 -03:00
Henrique Santos
5d2fc89137 scaffolding project methods 2016-03-12 04:35:49 -03:00
Henrique Santos
099d0fef2f scaffolding project size 2016-03-11 22:31:46 -03:00
James Allen
8ddfbb99de Only parse body when we need to for copyFile request 2015-11-25 17:01:19 +00:00
James Allen
bc6c8d6afc Don't send multiple status codes on insertFile error 2015-11-25 16:07:51 +00:00
Xavier Trochu
81a7203315 Fix the checkIfFileExists API. 2015-11-25 15:40:35 +01:00
Xavier Trochu
3b6270236c Replace indentation from 2 space to tabs. 2015-11-20 14:38:23 +01:00
Xavier Trochu
b6486cb825 Cleanup the dependencies 2015-11-20 14:05:54 +01:00
Xavier Trochu
0219900933 Forward the options argument to getFileStream 2015-11-20 12:07:04 +01:00
Xavier Trochu
bfd41fdaf9 Add aws-sdk unit test. 
Fix Aws-Sdk persistor to return a correct error on file not found.
Fix FileHandler after some change were lost on a previous merge
 2015-11-20 12:02:22 +01:00