github/overleaf
1
0
Fork 0
mirror of https://github.com/overleaf/overleaf.git synced 2024-09-16 02:52:31 -04:00
Code Issues Projects Releases Packages Wiki Activity
18326 commits 3 branches 7 tags 221 MiB
Commit graph

619 commits

Author SHA1 Message Date
Simon Detheridge
a1ae68f6b5 Tidy up 'getReadyPipeline' 2020-03-26 16:57:55 +00:00
Simon Detheridge
1f037ef653 Catch errors from pipeline and stream when waiting for streams to be readable 2020-03-26 16:57:55 +00:00
Simon Detheridge
238d96ed44 Fail acceptance tests on unhandled promise rejection 2020-03-26 16:57:55 +00:00
Simon Detheridge
7e45a82c35 Use autodestroy on Transform stream 2020-03-26 15:09:56 +00:00
Simon Detheridge
82599bce67 Merge pull request #105 from overleaf/jpa-logger-1-9-1 
[misc] bump logger-sharelatex to 1.9.1
 2020-03-26 10:31:33 +00:00
Simon Detheridge
d073fe75ca Use single pipeline when calculating md5 2020-03-25 16:59:51 +00:00
Jakob Ackermann
2ce3c8f97e [misc] bump logger-sharelatex to 1.9.1 2020-03-23 16:18:07 +01:00
Simon Detheridge
65f19346f0 Fix fallback to custom s3 buckets 2020-03-19 11:10:31 +00:00
Simon Detheridge
cb4bdd99f4 Use an ISODate for deleted file names 2020-03-18 20:19:12 +00:00
Simon Detheridge
9d32d4ec16 Don't modify 'opts' parameter 2020-03-16 15:57:37 +00:00
Simon Detheridge
b37c52fc3a Make GCS delete concurrency configurable 2020-03-16 15:54:05 +00:00
Simon Detheridge
9b658dda18 Copy-on-delete before unlocking 2020-03-16 15:53:45 +00:00
Simon Detheridge
06c4c0f74f Fix incorrect key when deleting projects 2020-03-16 11:35:01 +00:00
Simon Detheridge
9f74aac1a0 Add acceptance tests for directory deletion 2020-03-16 11:34:45 +00:00
Simon Detheridge
ce52f8aa60 Update FSPersistor deleteDirectory behaviour to match S3 and GCS 2020-03-16 11:33:51 +00:00
Simon Detheridge
47e96a4d94 Add endpoint to delete a project 
Needs acceptance tests
 2020-03-14 14:56:29 +00:00
Simon Detheridge
fc80aa3954 Move directory key validation into FileHandler 2020-03-14 14:31:30 +00:00
Simon Detheridge
58db14456a Add timestamp to files in deleted bucket 2020-03-14 14:11:17 +00:00
Simon Detheridge
edf1ce1f7e Delete files from a directory in parallel 2020-03-14 14:02:58 +00:00
Simon Detheridge
183cb0179a Add support to GCS persistor for unlocking files and copying on delete 2020-03-13 16:18:45 +00:00
Simon Detheridge
28c3fe4a56 Validate key names when deleting directory with a configurable regex 2020-03-10 17:54:09 +00:00
Simon Detheridge
2509b51883 Add optional gcs config to override gcs settings (for fake gcs server) 2020-03-05 17:23:47 +00:00
Simon Detheridge
eb93ae4b10 Use Bucket.deleteFiles to delete directory contents, instead of iterating 2020-03-05 14:12:15 +00:00
Simon Detheridge
460dd96b17 Cosmetic clean-up of GCS Persistor & tests 2020-03-05 13:45:46 +00:00
Simon Detheridge
3bb956b38e Use http for the fake GCS server 2020-03-04 17:01:20 +00:00
Simon Detheridge
30114cd79b Remove unnecessary try/catch around 'createReadStream' 2020-03-04 16:38:05 +00:00
Simon Detheridge
a7198764cb Improve/add some comments for clarity 2020-03-04 16:25:11 +00:00
Simon Detheridge
def383574e Simplify getMeteredStream to record metric directly 2020-03-04 16:17:36 +00:00
Simon Detheridge
76243fd75a 'bucket' -> 'bucketName' in GCS Persistor 2020-03-04 16:04:26 +00:00
Simon Detheridge
6979b8638a Add 'Buffer' global for GCS unit tests 2020-03-04 15:42:16 +00:00
Simon Detheridge
12274e1427 Add unit tests for GCS Persistor 2020-03-04 15:38:18 +00:00
Simon Detheridge
e58284aefe Move base64/hex methods to PersistorHelper 
Also add some null-safety checks
 2020-03-04 15:38:18 +00:00
Simon Detheridge
d9c9d74994 Remove unnecessary test for S3 file deletion 
S3 does not throw a not-found error when deleting a file that does not exist
 2020-03-04 15:38:18 +00:00
Simon Detheridge
e2f3dd23c9 Switch back to official fake-gcs-server image 2020-03-04 15:38:18 +00:00
Simon Detheridge
9dddf25209 Add note on gcs config to config file 2020-03-04 15:38:18 +00:00
Simon Detheridge
2cfab8d313 Add GCS-specific acceptance tests 2020-03-04 15:38:16 +00:00
Simon Detheridge
366ce97169 Add GCS Persistor 2020-03-04 15:09:14 +00:00
Simon Detheridge
e6cf0687a9 Add gcs client library 2020-03-04 15:09:14 +00:00
Simon Detheridge
7292d93dab Add fake GCS server, and fix health checks 2020-03-04 15:08:19 +00:00
Simon Detheridge
d06b74b0f8 Bump metrics to 2.5.0 2020-03-04 10:55:01 +00:00
Simon Detheridge
9807568de8 Add missing 'process' global for settings tests 2020-03-04 10:08:31 +00:00
Simon Detheridge
a3782422b7 Upgrade to node 12 2020-03-04 10:06:18 +00:00
Simon Detheridge
8905d393f7 Merge pull request #95 from overleaf/jpa-test-cleanup 
[misc] test cleanup
 2020-03-04 10:01:39 +00:00
Simon Detheridge
b5e4345855 Merge pull request #96 from overleaf/jpa-dependencies-cleanup 
[misc] dependencies cleanup
 2020-03-04 10:00:56 +00:00
Simon Detheridge
467af9885a Merge pull request #97 from overleaf/jpa-3rd-party-s3-via-env 
[config] add support for third party s3 which uses path-style buckets
 2020-03-04 09:59:48 +00:00
Simon Detheridge
62b2aa8f02 Merge pull request #98 from overleaf/jpa-promisify-filehandler 
[misc] promisify FileHandler and remove dependency on async
 2020-03-04 09:59:15 +00:00
Jakob Ackermann
2b15729658 [misc] promisify FileHandler and remove dependency on async 
Signed-off-by: Jakob Ackermann <das7pad@outlook.com>
 2020-03-02 18:23:17 +01:00
Jakob Ackermann
ada6ac0764 [misc] revert eslint rules 2020-03-02 18:12:55 +01:00
Jakob Ackermann
1276fb426a [misc] bump buildscript version to 1.3.6 2020-03-02 17:50:45 +01:00
Simon Detheridge
0312b00150 Fix prettier task 2020-03-02 16:31:35 +00:00
Jakob Ackermann
4b7c6fafd2 [misc] drop unused dependency pngcrush 2020-03-02 17:11:56 +01:00
Jakob Ackermann
3c61e53918 [misc] move mocha to the devDependencies 2020-03-02 17:11:39 +01:00
Jakob Ackermann
6f27f7a1bf [misc] drop the /heapdump route and related heapdump package 2020-03-02 17:09:35 +01:00
Jakob Ackermann
32557ab1d7 [SafeExec] replace _.once with lodash.once 2020-03-02 17:08:42 +01:00
Jakob Ackermann
0db4a17a14 [HealthCheckController] use fs.copyFile instead of fs-extra.copy 
the fs-extra method has a HUGE overhead of JS code and also syscalls for
 no particular benefit in this case: just copy the tiny.pdf file.

Here is an overview of the major operations: paths are relative to
  https://github.com/jprichardson/node-fs-extra/blob/1.0.0

We start in /lib/copy/copy.js
- sys: check that the source file exists
- sys: check that the source file has an existing parent directory?!
Continue in /lib/copy/ncp.js
- sys: more stat calls on both source and dest to determine permissions
- read/write streams to pipe the file content through the process
- sys: chmod on the destination to match the source permissions

What we actually need is a call to the binding and let node/the os
 figure out the best way to copy the contents.

Signed-off-by: Jakob Ackermann <das7pad@outlook.com>
 2020-03-02 17:02:35 +01:00
Simon Detheridge
c13d7f4197 Bump aws-sdk version in package.json 2020-03-02 11:18:06 +00:00
Brian Gough
ebabe52632 fix settings unit test 2020-02-28 15:27:29 +00:00
Brian Gough
e3011b2521 run npm update 2020-02-28 14:51:57 +00:00
Jakob Ackermann
b4e3d7de62 [config] add support for third party s3 which uses path-style buckets 
Signed-off-by: Jakob Ackermann <das7pad@outlook.com>
 2020-02-28 12:48:52 +01:00
Jakob Ackermann
2b9d0868c5 [misc] test/acceptance: retrieve ingress metrics just before using it 
The upload request can bump the ingress metric.
The content hash validation might require a full download
in case the ETag field of the upload response is not a md5 sum.
 2020-02-28 12:27:06 +01:00
Jakob Ackermann
516102e6fb [misc] test/acceptance: do not hard code fake credentials 2020-02-28 12:26:46 +01:00
Jakob Ackermann
847f124d7b [misc] test/acceptance: skip the shutdown in case we did not start yet 
Signed-off-by: Jakob Ackermann <das7pad@outlook.com>
 2020-02-28 12:26:24 +01:00
Jakob Ackermann
6589aa6ae0 [misc] test/acceptance: harden the startup check for s3 
Signed-off-by: Jakob Ackermann <das7pad@outlook.com>
 2020-02-28 12:26:24 +01:00
Jakob Ackermann
94b8a7f891 [misc] test/unit: KeybuilderTests: use a unique suite label 2020-02-28 12:26:23 +01:00
Jakob Ackermann
54e3b57772 [misc] test/unit: add missing globals that are lazy loaded 2020-02-28 12:26:23 +01:00
Jakob Ackermann
7920570dd8 [misc] test/unit: add missing require stubs for metrics and settings 2020-02-28 12:26:22 +01:00
Simon Detheridge
6e19a650d2 Write to error log on unhandled rejection/exception 2020-02-27 10:33:48 +00:00
Simon Detheridge
492f8abc9f Update config/settings.defaults.coffee 
Co-Authored-By: Jakob Ackermann <das7pad@outlook.com>
 2020-02-18 10:24:29 +00:00
Simon Detheridge
8acc9ef0ae Make S3 partSize a setting 2020-02-17 14:06:15 +00:00
Simon Detheridge
7c4bf97a7f Avoid fetching file a second time when source md5 can't be fetched 2020-02-17 14:06:05 +00:00
Simon Detheridge
9e7eec7a19 Add acceptance test for large files 2020-02-17 14:05:56 +00:00
Simon Detheridge
2f2a819b74 Re-fetch file to calculate md5 if etag is not in correct format 2020-02-17 14:05:43 +00:00
Simon Detheridge
5adfb3e2c0 Use large upload chunks 2020-02-17 14:05:31 +00:00
Simon Detheridge
4e507197fc Revert "Merge pull request #91 from overleaf/revert-78-spd-migration-persistor" 
This reverts commit 0cce792604c258538cf012486e7329fa1be7da0f, reversing
changes made to 08b606e22f855121b2d8a2832a5d7e2f45836c3f.
 2020-02-17 14:04:42 +00:00
Simon Detheridge
df2ddbe0e1 Revert "Add Migration Persistor, to send missing file requests to a fallback persistor" 2020-02-14 14:26:33 +00:00
Simon Detheridge
e4926e09a7 Move error handler to be the last middleware 2020-02-14 10:58:46 +00:00
dependabot[bot]
31f2572e57 Bump lodash from 4.17.11 to 4.17.15 
Bumps [lodash](https://github.com/lodash/lodash) from 4.17.11 to 4.17.15.
- [Release notes](https://github.com/lodash/lodash/releases)
- [Commits](https://github.com/lodash/lodash/compare/4.17.11...4.17.15)

Signed-off-by: dependabot[bot] <support@github.com>
 2020-02-14 09:39:06 +00:00
Simon Detheridge
485cfce6d3 Merge pull request #85 from overleaf/jpa-dependencies-cleanup 
[misc] rename npm-shrinkwrap.json to package-lock.json and run npm i
 2020-02-14 09:24:43 +00:00
Simon Detheridge
16d1674eab Merge pull request #84 from overleaf/bg-remove-travis-yml 
Remove unused .travis.yml file
 2020-02-14 09:21:03 +00:00
Jakob Ackermann
84a80930eb [misc] rename npm-shrinkwrap.json to package-lock.json and run npm i 2020-02-14 09:19:29 +00:00
Simon Detheridge
44fbc6a733 Merge pull request #82 from overleaf/spd-error-handler-error 
Fix log info assignment in error handler
 2020-02-14 09:18:16 +00:00
Simon Detheridge
3b011258d2 Tidy up md5 hash generation 2020-02-12 13:27:12 +00:00
Simon Detheridge
49ad408b30 Remove unused imports and format correctly 2020-02-12 13:27:12 +00:00
Simon Detheridge
93cd55fb79 Refactor persistors to use a helper for common things 2020-02-12 13:27:12 +00:00
Simon Detheridge
304fdfd35c Explicitly resume stream after adding listener 2020-02-12 13:21:53 +00:00
Simon Detheridge
6dcf351377 Remove unnecessary 'async' 2020-02-12 13:21:53 +00:00
Simon Detheridge
f4a16cd972 Update tests to properly check for copied files 2020-02-12 13:21:53 +00:00
Simon Detheridge
42a5d168df Remove unused packages 2020-02-12 13:21:52 +00:00
Simon Detheridge
9e0b378948 Remove minipass as dependency and refactor to make things clearer 2020-02-12 13:21:52 +00:00
Simon Detheridge
5d5d325691 Preserve all error information when cleanup of copied file fails 2020-02-12 13:21:52 +00:00
Simon Detheridge
b4b7fd226e Add mechanisms to transfer files with md5-based integrity checks 
Fix error in settings and tidy up tests

Remove unused variable declaration

Remove .only from tests and update eslint rules to catch it in future

Use  to catch errors more safely getting md5 hash

Avoid unnecessary call to S3 to get md5 response
 2020-02-12 13:21:52 +00:00
Simon Detheridge
2625e03a31 Add MigrationPersistor for sending 404 requests to a fallback persistor 2020-02-12 13:19:48 +00:00
Simon Detheridge
f877f51775 Rename *PersistorManager to *Persistor 2020-02-12 13:19:48 +00:00
Brian Gough
a2e1591e93 remove unused .travis.yml file 2020-02-12 12:37:00 +00:00
Simon Detheridge
e3613e52db Fix log info assignment in error handler 2020-02-12 10:34:56 +00:00
Jakob Ackermann
1341af04f5 [misc] update the build scripts to 1.3.5 2020-02-11 18:58:55 +01:00
Jakob Ackermann
ca94225c19 Merge pull request #80 from overleaf/jpa-cleanup-install-deps 
[misc] cleanup install deps
 2020-02-10 17:16:44 +01:00
Brian Gough
14ba1ea1c1 update to node 10.19.0 2020-02-07 14:16:31 +00:00
Jakob Ackermann
ac4361d560 [misc] dockerignore: hide the local data dirs 2020-01-31 15:50:53 +01:00
Jakob Ackermann
c1d81dc1ea [misc] install a maintained ghostscript version and delete package lists 
also do not install vim and bail out in case any command exited with a
 non zero exit code.
 2020-01-31 15:50:53 +01:00
Simon Detheridge
997434c930 Remove redundant 'res.logInfo' 2020-01-14 20:55:33 +00:00
Simon Detheridge
035364b62e Fix RequestLogger object lifecycle 2020-01-14 17:15:01 +00:00
Simon Detheridge
28fb998719 Refactor request logger into a class 2020-01-14 12:02:39 +00:00
Simon Detheridge
9615a06e0f Improve logging middleware to add info methods to request object 2020-01-10 17:22:06 +00:00
Simon Detheridge
f40fbe77aa Simplify logging to log only once per http request 2020-01-10 15:41:36 +00:00
Simon Detheridge
e331a3130c Handle AccessDenied and stream-premature-close errors 
These errors tend to occur as part of normal operation and should not generate `logger.err` messages
 2020-01-09 14:15:01 +00:00
Simon Detheridge
d8470d4a59 Use correct object path for s3BucketCreds 2020-01-08 11:32:05 +00:00
Simon Detheridge
3bf51cac67 Improve naming on internal '_client' method and use Map over object 2020-01-08 09:17:30 +00:00
Simon Detheridge
80d41cf51b Move bucket-specific file endpoint into FileController 2020-01-07 15:05:51 +00:00
Simon Detheridge
6cc5d94f13 Re-add bucket-specific credentials support for S3 2020-01-07 10:55:00 +00:00
Simon Detheridge
2bb2caf7b3 Clean up settings tests 2020-01-07 09:46:53 +00:00
Simon Detheridge
ffd002dc76 Add explanatory comment 2020-01-07 09:30:30 +00:00
Simon Detheridge
b642b3cf3a Rename incorrectly-named 'getDirectorySize' method 2020-01-07 09:30:30 +00:00
Simon Detheridge
a6d9d34fe3 Configure PersistorManager to use the new S3 manager for 'aws-sdk' 2020-01-07 09:30:30 +00:00
Simon Detheridge
2ca74fdf15 Cleanup and refactor S3PersistorManager to use aws-sdk only 2020-01-07 09:30:30 +00:00
Simon Detheridge
473aea4e60 Remove AWSSDKPersistorManager 2020-01-07 09:30:30 +00:00
Simon Detheridge
e27cf4db7b Pass pipeline errors onto 'next' error handler 2020-01-07 09:30:30 +00:00
Simon Detheridge
87b51d0c35 Remove old exception-handling and shutdown-related mechanisms 2020-01-07 09:30:28 +00:00
Simon Detheridge
74b480fc55 Post-decaf cleanup of app.js 2020-01-07 09:29:33 +00:00
Simon Detheridge
85d3c0a852 Remove old exception-handling and shutdown-related mechanisms 2020-01-07 09:27:02 +00:00
Simon Detheridge
6a679023d3 Fix order of .status().send() 2020-01-07 09:27:02 +00:00
Simon Detheridge
fce275e1d4 Post-decaf cleanup of app.js 2020-01-07 09:27:02 +00:00
Simon Detheridge
0329c759dc Remove wrapper function in PersistorManager 2020-01-07 09:26:43 +00:00
Simon Detheridge
dbfacce988 Post-decaf cleanup of PersistorManager 2020-01-07 09:26:43 +00:00
Simon Detheridge
10f7087912 Post-decaf cleanup of KeyBuilderTests 2020-01-07 09:26:43 +00:00
Simon Detheridge
ec80052d9a Decaf cleanup of FileController 
Remove deprecated res.send
 2020-01-07 09:26:23 +00:00
Simon Detheridge
37d44d5d86 Return full path in stubbed 'glob' method 2020-01-07 09:26:01 +00:00
Simon Detheridge
fccb288985 Add comment regarding deleteDirectory being internal-only 2020-01-07 09:26:01 +00:00
Simon Detheridge
f1b6b35c69 Throw errors more consistently in FSPersistorManager 2020-01-07 09:26:00 +00:00
Simon Detheridge
25f1c2bfc4 Delete temporary file when error in writing to stream 2020-01-07 09:26:00 +00:00
Simon Detheridge
039bec02f7 Use directory names in acceptance tests, and add test for getting size 2020-01-07 09:26:00 +00:00
Simon Detheridge
21ba083877 Use 'glob' to determine 'directory' size. 
Check for filtered directory paths in all tests.
 2020-01-07 09:26:00 +00:00
Simon Detheridge
ce90292394 Decaf cleanup for FSPersistorManager 2020-01-07 09:26:00 +00:00
Simon Detheridge
4315824d3c Minor cleanup of FilestoreApp acceptance-test helper 2020-01-07 09:25:04 +00:00
Simon Detheridge
a8158d6c8c Cleanup and promisify acceptance tests. Run tests for all backends. 2020-01-07 09:25:04 +00:00
Simon Detheridge
006f84abeb Decaf and promisify ImageOptimiser 2020-01-07 09:24:29 +00:00
Simon Detheridge
42adc59d01 Clean up and promisify health-check controller and KeyBuilder 2020-01-07 09:23:41 +00:00
Simon Detheridge
ac2d05ecb3 Remove unnecessary logging from FileConverter 2020-01-07 09:22:53 +00:00
Simon Detheridge
b6a1ea6a30 Cleanup and promisify FileConverter 2020-01-07 09:22:53 +00:00
Simon Detheridge
a216be5fd3 Improve error handling in FileHandler 2020-01-07 09:18:59 +00:00
Simon Detheridge
27aaff7843 Decaf cleanup for FileHandler and LocalFileWriter 
Simplified code and tests where possible
 2020-01-07 09:18:56 +00:00
Simon Detheridge
eacad77112 Cleanup SafeExec 2019-12-19 15:42:46 +00:00
Simon Detheridge
f19d82369d Bump eslintrc up again to last-known-compatible version with eslit-prettier 2019-12-18 09:52:18 +00:00
Simon Detheridge
b94df9952b Add OError 2019-12-18 09:34:25 +00:00
Simon Detheridge
42f6d10031 Remove mongo and redis from dependencies 2019-12-17 15:19:19 +00:00
Simon Detheridge
0b3b71b857 Fall back to old AWS env vars if present 2019-12-17 09:57:51 +00:00
Simon Detheridge
b7e098208f Update build files from build_scripts 2019-12-16 17:17:25 +00:00
Simon Detheridge
29f333d60c Placate eslint 2019-12-16 15:38:00 +00:00
Simon Detheridge
7870c8185e Add linting to makefile 2019-12-16 11:58:22 +00:00
Simon Detheridge
230baef8fa Add linting tasks to package.json 2019-12-16 11:32:58 +00:00