Simon Detheridge
28fb998719
Refactor request logger into a class
2020-01-14 12:02:39 +00:00
Simon Detheridge
9615a06e0f
Improve logging middleware to add info methods to request object
2020-01-10 17:22:06 +00:00
Simon Detheridge
f40fbe77aa
Simplify logging to log only once per http request
2020-01-10 15:41:36 +00:00
Simon Detheridge
e331a3130c
Handle AccessDenied and stream-premature-close errors
...
These errors tend to occur as part of normal operation and should not generate `logger.err` messages
2020-01-09 14:15:01 +00:00
Simon Detheridge
d8470d4a59
Use correct object path for s3BucketCreds
2020-01-08 11:32:05 +00:00
Simon Detheridge
3bf51cac67
Improve naming on internal '_client' method and use Map over object
2020-01-08 09:17:30 +00:00
Simon Detheridge
80d41cf51b
Move bucket-specific file endpoint into FileController
2020-01-07 15:05:51 +00:00
Simon Detheridge
6cc5d94f13
Re-add bucket-specific credentials support for S3
2020-01-07 10:55:00 +00:00
Simon Detheridge
2bb2caf7b3
Clean up settings tests
2020-01-07 09:46:53 +00:00
Simon Detheridge
ffd002dc76
Add explanatory comment
2020-01-07 09:30:30 +00:00
Simon Detheridge
b642b3cf3a
Rename incorrectly-named 'getDirectorySize' method
2020-01-07 09:30:30 +00:00
Simon Detheridge
a6d9d34fe3
Configure PersistorManager to use the new S3 manager for 'aws-sdk'
2020-01-07 09:30:30 +00:00
Simon Detheridge
2ca74fdf15
Cleanup and refactor S3PersistorManager to use aws-sdk only
2020-01-07 09:30:30 +00:00
Simon Detheridge
473aea4e60
Remove AWSSDKPersistorManager
2020-01-07 09:30:30 +00:00
Simon Detheridge
e27cf4db7b
Pass pipeline errors onto 'next' error handler
2020-01-07 09:30:30 +00:00
Simon Detheridge
87b51d0c35
Remove old exception-handling and shutdown-related mechanisms
2020-01-07 09:30:28 +00:00
Simon Detheridge
74b480fc55
Post-decaf cleanup of app.js
2020-01-07 09:29:33 +00:00
Simon Detheridge
85d3c0a852
Remove old exception-handling and shutdown-related mechanisms
2020-01-07 09:27:02 +00:00
Simon Detheridge
6a679023d3
Fix order of .status().send()
2020-01-07 09:27:02 +00:00
Simon Detheridge
fce275e1d4
Post-decaf cleanup of app.js
2020-01-07 09:27:02 +00:00
Simon Detheridge
0329c759dc
Remove wrapper function in PersistorManager
2020-01-07 09:26:43 +00:00
Simon Detheridge
dbfacce988
Post-decaf cleanup of PersistorManager
2020-01-07 09:26:43 +00:00
Simon Detheridge
10f7087912
Post-decaf cleanup of KeyBuilderTests
2020-01-07 09:26:43 +00:00
Simon Detheridge
ec80052d9a
Decaf cleanup of FileController
...
Remove deprecated res.send
2020-01-07 09:26:23 +00:00
Simon Detheridge
37d44d5d86
Return full path in stubbed 'glob' method
2020-01-07 09:26:01 +00:00
Simon Detheridge
fccb288985
Add comment regarding deleteDirectory being internal-only
2020-01-07 09:26:01 +00:00
Simon Detheridge
f1b6b35c69
Throw errors more consistently in FSPersistorManager
2020-01-07 09:26:00 +00:00
Simon Detheridge
25f1c2bfc4
Delete temporary file when error in writing to stream
2020-01-07 09:26:00 +00:00
Simon Detheridge
039bec02f7
Use directory names in acceptance tests, and add test for getting size
2020-01-07 09:26:00 +00:00
Simon Detheridge
21ba083877
Use 'glob' to determine 'directory' size.
...
Check for filtered directory paths in all tests.
2020-01-07 09:26:00 +00:00
Simon Detheridge
ce90292394
Decaf cleanup for FSPersistorManager
2020-01-07 09:26:00 +00:00
Simon Detheridge
4315824d3c
Minor cleanup of FilestoreApp acceptance-test helper
2020-01-07 09:25:04 +00:00
Simon Detheridge
a8158d6c8c
Cleanup and promisify acceptance tests. Run tests for all backends.
2020-01-07 09:25:04 +00:00
Simon Detheridge
006f84abeb
Decaf and promisify ImageOptimiser
2020-01-07 09:24:29 +00:00
Simon Detheridge
42adc59d01
Clean up and promisify health-check controller and KeyBuilder
2020-01-07 09:23:41 +00:00
Simon Detheridge
ac2d05ecb3
Remove unnecessary logging from FileConverter
2020-01-07 09:22:53 +00:00
Simon Detheridge
b6a1ea6a30
Cleanup and promisify FileConverter
2020-01-07 09:22:53 +00:00
Simon Detheridge
a216be5fd3
Improve error handling in FileHandler
2020-01-07 09:18:59 +00:00
Simon Detheridge
27aaff7843
Decaf cleanup for FileHandler and LocalFileWriter
...
Simplified code and tests where possible
2020-01-07 09:18:56 +00:00
Simon Detheridge
eacad77112
Cleanup SafeExec
2019-12-19 15:42:46 +00:00
Simon Detheridge
f19d82369d
Bump eslintrc up again to last-known-compatible version with eslit-prettier
2019-12-18 09:52:18 +00:00
Simon Detheridge
b94df9952b
Add OError
2019-12-18 09:34:25 +00:00
Simon Detheridge
42f6d10031
Remove mongo and redis from dependencies
2019-12-17 15:19:19 +00:00
Simon Detheridge
0b3b71b857
Fall back to old AWS env vars if present
2019-12-17 09:57:51 +00:00
Simon Detheridge
b7e098208f
Update build files from build_scripts
2019-12-16 17:17:25 +00:00
Simon Detheridge
29f333d60c
Placate eslint
2019-12-16 15:38:00 +00:00
Simon Detheridge
7870c8185e
Add linting to makefile
2019-12-16 11:58:22 +00:00
Simon Detheridge
230baef8fa
Add linting tasks to package.json
2019-12-16 11:32:58 +00:00
Simon Detheridge
35d050a49c
Prettier fixes
2019-12-16 11:32:46 +00:00
Simon Detheridge
6974fd640d
Remove unusued cluster.coffee
2019-12-16 11:29:25 +00:00
Simon Detheridge
495424a94b
Fix metrics includes in SandboxedModule
2019-12-16 11:28:24 +00:00
Simon Detheridge
eae7d28495
Prettier: convert test/unit decaffeinated files to Prettier format
2019-12-16 11:20:29 +00:00
Simon Detheridge
ec60f778e6
Rename test/unit/coffee to test/unit/js
2019-12-16 11:20:27 +00:00
decaffeinate
ff04085bdd
decaffeinate: Run post-processing cleanups on AWSSDKPersistorManagerTests.coffee and 12 other files
2019-12-16 11:20:25 +00:00
decaffeinate
5cf7138ef1
decaffeinate: Convert AWSSDKPersistorManagerTests.coffee and 12 other files to JS
2019-12-16 11:20:22 +00:00
decaffeinate
72c83bdaff
decaffeinate: Rename AWSSDKPersistorManagerTests.coffee and 12 other files from .coffee to .js
2019-12-16 11:20:20 +00:00
Simon Detheridge
e06c03e536
Fix decaf error in FSPersistorManager
2019-12-16 11:19:56 +00:00
Simon Detheridge
0b6e725137
Remove coffeescript compiles from package.json
2019-12-16 11:19:56 +00:00
Simon Detheridge
9943e2cc8f
Prettier: convert individual decaffeinated files to Prettier format
2019-12-16 11:19:56 +00:00
Simon Detheridge
1d295ef7e4
Decaffeinate: convert individual files to js
2019-12-16 11:19:55 +00:00
Simon Detheridge
d70a33c4f5
Rename individual coffee files to js files
2019-12-16 11:19:43 +00:00
Simon Detheridge
e4b1106761
Prettier: convert test/acceptance decaffeinated files to Prettier format
2019-12-16 11:19:43 +00:00
Simon Detheridge
b8e7abd25e
Rename test/acceptance/coffee to test/acceptance/js
2019-12-16 11:19:43 +00:00
decaffeinate
738e1d821e
decaffeinate: Run post-processing cleanups on FilestoreApp.coffee and 1 other file
2019-12-16 11:19:43 +00:00
decaffeinate
e6e75d4e19
decaffeinate: Convert FilestoreApp.coffee and 1 other file to JS
2019-12-16 11:19:43 +00:00
decaffeinate
1c7d1af4ca
decaffeinate: Rename FilestoreApp.coffee and 1 other file from .coffee to .js
2019-12-16 11:19:43 +00:00
Simon Detheridge
6225f2f236
Prettier: convert app/js decaffeinated files to Prettier format
2019-12-16 11:19:36 +00:00
Simon Detheridge
601861ee58
Remove js from dockerignore
2019-12-16 10:54:12 +00:00
Simon Detheridge
3718c90c7a
Rename app/coffee dir to app/js
2019-12-16 10:54:08 +00:00
decaffeinate
fcbb13b5bf
decaffeinate: Run post-processing cleanups on AWSSDKPersistorManager.coffee and 13 other files
2019-12-16 10:54:06 +00:00
decaffeinate
6bd8452f19
decaffeinate: Convert AWSSDKPersistorManager.coffee and 13 other files to JS
2019-12-16 10:54:03 +00:00
decaffeinate
8d2c87420e
decaffeinate: Rename AWSSDKPersistorManager.coffee and 13 other files from .coffee to .js
2019-12-16 10:53:59 +00:00
Simon Detheridge
ae3d25c969
Decaffeinate: add eslint and prettier rc files
2019-12-16 10:53:56 +00:00
Simon Detheridge
1798efd4be
Install eslint and prettier
2019-12-16 10:53:52 +00:00
Simon Detheridge
c9106eff95
Upgrade Docker image to node 10
2019-12-16 10:53:42 +00:00
Jakob Ackermann
ed97fcfcd0
[logging] do not overwrite the logger name
...
Signed-off-by: Jakob Ackermann <das7pad@outlook.com>
2019-12-16 10:53:34 +00:00
Jakob Ackermann
c5e1584fcc
[FSPersistorManager] fix the stream opening for node10+
...
Attaching a `readable` listener causes the stream to hang otherwise.
Signed-off-by: Jakob Ackermann <das7pad@outlook.com>
2019-12-16 10:53:23 +00:00
Simon Detheridge
dab44f6fec
Merge branch 'master' into logging/keep-logger-name
2019-12-16 09:42:18 +00:00
Simon Detheridge
32b7f25404
Merge pull request #49 from das7pad/misc/node10plus-support
...
[misc] add support for node 10+
2019-12-16 09:39:00 +00:00
Simon Detheridge
56b38af678
Explicitly wait for S3 in acceptance tests
2019-12-10 17:43:34 +00:00
Simon Detheridge
237c4113cd
Ensure fakes3 is healthy before running tests
2019-12-10 16:36:33 +00:00
Simon Detheridge
cf684dcd98
Fix fakes3 configuration in ci yml
2019-12-10 11:38:19 +00:00
Simon Detheridge
6f326d5650
Use SSL setting based on url protocol
2019-12-09 17:41:20 +00:00
Simon Detheridge
48aa141591
Add metric for s3 ingress
2019-12-09 17:06:00 +00:00
Simon Detheridge
1d1106bc67
Add metric for s3 egress
2019-12-09 17:06:00 +00:00
Simon Detheridge
c01603b1e7
Support custom S3 endpoints
2019-12-09 17:06:00 +00:00
Simon Detheridge
96457597ac
Add fake s3 server and initial config
2019-12-09 17:06:00 +00:00
Simon Detheridge
86b9e4b53a
Rename request -> s3Request to prevent overwriting main import
2019-12-09 17:05:58 +00:00
Brian Gough
2ec38068aa
add comments about aws-sdk and s3 backends
2019-12-06 14:35:13 +00:00
Nate Stemen
aba0d14edd
bump build script to 1.1.24
2019-10-25 11:52:54 -04:00
Nate Stemen
4798589213
replace private link with public one
2019-10-25 11:51:58 -04:00
Shane Kilkelly
dbede8509c
Merge pull request #54 from overleaf/sk-dep-upgrades-2019-06
...
update logger and metrics
2019-07-09 10:18:16 +01:00
Eric Mc Sween
7d900b57bf
Fix createUnbufferedStream() function call
...
In 49a21155f642670dfea264ac73fb60241f37cb87, I managed to incorrectly
write the `createUnbufferedStream()` function from the AWS SDK as
`getUnbufferedStream()` and to consistently use that naming in the unit
tests.
This commit fixes that. I have tested again on S3.
2019-06-19 12:58:17 -04:00
Shane Kilkelly
013400d7a4
Re-add environment variable for conversions
2019-06-19 14:04:57 +01:00
Shane Kilkelly
f865762c29
update logger and metrics
2019-06-19 13:50:58 +01:00
Eric Mc Sween
f2521a29b9
Use AWS SDK for getFileStream()
...
The AWS SDK has a retry strategy to deal with rate limiting or transient
unavailability of S3. We hope it will reduce our error rates.
2019-06-19 08:18:11 -04:00
Eric Mc Sween
3575c89d03
Return file size on HEAD request
...
This will be used by the file preview feature when it gets partial
content.
2019-06-17 08:23:59 -04:00
Timothée Alby
f081546ec0
update Git URL in Jenkinsfile
2019-05-07 17:37:07 +01:00
Timothée Alby
32a54a7e37
Update README.md
2019-05-07 17:22:35 +02:00
Jakob Ackermann
a79adcd325
[logging] do not overwrite the logger name
...
Signed-off-by: Jakob Ackermann <das7pad@outlook.com>
2019-05-02 12:48:43 +02:00
Jakob Ackermann
c59a3db4e8
[FSPersistorManager] fix the stream opening for node10+
...
Attaching a `readable` listener causes the stream to hang otherwise.
Signed-off-by: Jakob Ackermann <das7pad@outlook.com>
2019-05-02 02:47:53 +02:00
Brian Gough
b9d3b1a842
fix beginShutdown to use server.close()
2019-03-15 15:35:28 +00:00
Henry Oswald
37b039abf0
add sentry to settings.defaults.coffee
2019-03-05 17:20:16 +00:00
Henry Oswald
2adae20368
remove a console.log
2019-02-06 11:43:11 +00:00
Henry Oswald
0f48c77e7e
enable conversions for local docker compose
2019-02-05 17:19:33 +00:00
Henry Oswald
5eec00db51
don't run conversions test when disabled
2019-02-05 14:06:51 +00:00
Henry Oswald
3fccf79ca8
cleanup
2019-02-05 12:50:08 +00:00
Henry Oswald
ca9231b04e
set enable conversions to false by default but enable in compse
2019-02-05 11:19:18 +00:00
Henry Oswald
19007befb2
Merge branch 'master' into ho-docker
2019-02-01 16:26:45 +00:00
Henry Oswald
ea845f4935
bulk upgrade to 1.1.12, moved to npm rather than git
2019-01-31 16:01:20 +00:00
Brian Gough
9d93eee3e8
return a 404 error (instead of a 500) when copying a missing file
2019-01-09 10:31:59 +00:00
Henry Oswald
3614f217e6
add enableConversions flag to disable conversions which we can't do on
...
k8
2019-01-07 15:54:24 +00:00
Christopher Hoskin
0421192449
Ignore any file ending in .map, wherever it is
2019-01-04 16:22:13 +00:00
Christopher Hoskin
c8d146b571
Bump settings to v1.1.0
2019-01-04 16:03:45 +00:00
Christopher Hoskin
b522fe1531
Remove grunt
2019-01-04 16:03:22 +00:00
Christopher Hoskin
6763809748
Bump logger to v1.5.9
2019-01-03 16:27:45 +00:00
Christopher Hoskin
594b403bd3
Move metrics init to start of app.coffee
2019-01-02 09:55:31 +00:00
Christopher Hoskin
3b6eba33ba
Remove Metrics.inc "startup"
from app.coffee
2019-01-02 09:53:53 +00:00
Christopher Hoskin
aab5a1af6a
Merge branch 'master' into csh-ho-docker
2019-01-02 09:51:35 +00:00
Christopher Hoskin
6214256568
Bump logger to v1.5.8
2019-01-02 09:47:38 +00:00
Brian Gough
cf12ec1154
use the aws sdk to copy files in S3PersistorManager
...
to work around problems with knox
2018-12-20 16:11:18 +00:00
Brian Gough
28271655e1
avoid problems with caching of apt-get operations
2018-12-06 08:19:39 +00:00
Henry Oswald
25275212af
bump metrics to 2.0.7
2018-12-05 12:36:32 +00:00
Henry Oswald
5d927d30b6
update build scripts to 1.1.10
2018-12-05 10:53:11 +00:00
Henry Oswald
8743527452
bump metrics to 2.0.4
2018-11-29 16:42:26 +00:00
Henry Oswald
e64156cd27
clean npm-shrinkwrap and package.json
2018-11-29 16:41:30 +00:00
Henry Oswald
95f0a02dee
stub out metrics in tests & inject metrics endpoint
2018-11-29 13:53:43 +00:00
Henry Oswald
ee1c14a410
bump metrics
2018-11-29 12:46:14 +00:00
Brian Gough
fdc5370f94
Merge pull request #36 from sharelatex/bg-upgrade-logger
...
upgrade logger-sharelatex to v1.5.8
2018-11-13 14:06:44 +00:00
Brian Gough
8951f297aa
Merge pull request #37 from sharelatex/bg-reduce-logging-of-not-found-errors
...
reduce logging of not found errors
2018-11-13 14:06:19 +00:00
Brian Gough
0d6de10b5c
Merge pull request #34 from sharelatex/bg-make-backends-consistent
...
make filestore backends consistent
2018-11-13 10:12:18 +00:00
Brian Gough
e930c7106b
Merge pull request #35 from sharelatex/bg-serve-converted-file-from-disk
...
serve file from disk to avoid read-after-write inconsistency
2018-11-13 10:12:00 +00:00
Brian Gough
23a6d6e81d
consider 403 and 404 as NotFound errors
2018-11-09 14:05:38 +00:00
Brian Gough
69b164092c
suppress unnecessary error logging for NotFound
2018-11-09 14:05:38 +00:00
Brian Gough
c2bfdd095c
upgrade logger-sharelatex to v1.5.8
2018-11-08 16:29:23 +00:00
Brian Gough
8744b3aa4e
serve file from disk to avoid read-after-write inconsistency
2018-11-08 15:09:34 +00:00
Brian Gough
78ccf453db
update unit tests
2018-11-07 16:51:06 +00:00
Brian Gough
40ac59090b
make backends consistent by deleting temporary files
2018-11-07 16:22:09 +00:00
Brian Gough
06bfd4c270
add dockerfile to install packages needed by filestore
2018-11-07 16:12:49 +00:00
Christopher Hoskin
038b3476fe
Add some details to the README
2018-10-22 14:09:56 +01:00
Henry Oswald
38f565d35f
logout settings
2018-09-28 15:53:57 +01:00
Henry Oswald
aacb54d54c
Merge branch 'master' into ho-docker
2018-09-14 17:16:18 +01:00
Brian Gough
96e9d8b0ec
fix typo
2018-09-11 10:51:10 +01:00
Brian Gough
f3adf5a653
fix shrinkwrap for upgraded logger-sharelatex module
2018-09-06 17:01:40 +01:00
Brian Gough
f4ec677296
add sentry to logging
2018-09-06 17:01:18 +01:00
Christopher Hoskin
53dacc04ef
Bump metrics from 1.3.0 to 1.8.1
2018-07-26 16:06:30 +01:00
Christopher Hoskin
0e9a535e1c
Bump build script to 1.1.9
2018-07-26 16:05:11 +01:00
Henry Oswald
cba013cd71
change ordering of docker rmi
2018-07-23 17:06:31 +01:00
Christopher Hoskin
f9f599e3e7
Push container images to overleaf-ops project
2018-07-18 11:30:01 +01:00
Christopher Hoskin
1044069535
Remove csh-gcdm-test and csh-staging repos
2018-07-17 12:03:14 +01:00
Christopher Hoskin
c7733d05cc
Remove trailing / from project name
2018-07-17 11:06:33 +01:00
Christopher Hoskin
d182e5329c
Add cr-test2 GCR project
2018-07-17 10:52:20 +01:00
Christopher Hoskin
4784ae33a2
Bump build scripts to 1.1.8
2018-07-17 10:51:15 +01:00
Henry Oswald
1e4e68c64a
fix settings test
2018-07-10 14:55:40 +01:00
Henry Oswald
089bf3e084
seperate the standard s3 creds from the bucket s3 creds
2018-07-10 14:17:37 +01:00
Michael Mazour
c4e3f9eb02
Amend: tests for populating S3 settings from environment variable
2018-07-06 11:05:40 +01:00
Michael Mazour
3e1ef3af63
Populate S3 settings from environment variable
2018-07-06 11:05:19 +01:00
Michael Mazour
2da15f2eb3
Amend - improve documentation of settings
2018-07-06 10:28:02 +01:00
Michael Mazour
7feafccf31
Amend: safely navigate to bucket credentials
2018-07-06 09:28:09 +01:00
Michael Mazour
03033409c6
Amend: remove unused params
2018-07-06 09:12:59 +01:00
Michael Mazour
336a38ec1e
Amend: scrub secrets from logs
...
Calls to `getFile` can now include S3 credentials in `opts`, so sanitize before writing to opts to log.
2018-07-04 16:41:31 +01:00
Michael Mazour
600ab3ce67
Amend: remove problematic ampersand
2018-07-04 16:39:41 +01:00
Michael Mazour
cfbf0d81ba
Amend: fix params retrieval
2018-07-04 12:11:09 +01:00
Michael Mazour
ece650741a
Amend per several review comments
...
- Removed unused vars
- Label the metric with the bucket name
2018-07-04 12:02:09 +01:00
Michael Mazour
feca8933f1
Add endpoint for arbitrary bucket fetch
...
Add `/bucket/:bucket/key/*`, which fetches the file from the given bucket at the given path. Uses auth stored at `settings.filestore.s3.{{bucketName}}` if present, and otherwise default auth.
2018-07-04 11:22:36 +01:00
Christopher Hoskin
f0cf8e6622
Fix typo in buildscript
2018-06-14 11:37:18 +01:00
Christopher Hoskin
46c61aa78c
Update build scripts to 1.1.7, add csh-staging GCR
2018-06-13 16:21:32 +01:00
Henry Oswald
1763715004
remove black package-lock.json
2018-05-22 13:45:32 +01:00
Henry Oswald
ba8a5cf3e1
bump to 1.1.3 scripts
2018-05-22 13:45:01 +01:00
Henry Oswald
58ab4c72dc
update build scripts to 1.1.1
2018-05-21 15:14:24 +01:00
Henry Oswald
ee81a6af00
updated mocha and added --exit back in
2018-05-21 15:00:32 +01:00
Henry Oswald
8efd562eb3
fix broken acceptence tests, bad quotes in file path
2018-05-21 13:40:03 +01:00
Henry Oswald
6657b6c1dc
make all needed directories in install deps
2018-05-21 11:31:14 +01:00
Henry Oswald
e3514e84cd
fix up mocha migration tests
2018-05-21 11:02:53 +01:00
Henry Oswald
768a4cc1ad
remove console.log from settings
2018-05-21 10:34:16 +01:00
Henry Oswald
09527d804e
ignore js acceptence tests
2018-05-21 10:21:13 +01:00
Henry Oswald
3cf2e1d7c3
add null checks and new body parser
2018-05-21 10:11:47 +01:00
Henry Oswald
ac990f2aba
remove app.configure
2018-05-21 09:49:32 +01:00
Henry Oswald
d9eac8dd8b
upgrade mocha
2018-05-21 09:41:14 +01:00
Henry Oswald
fda6cb0084
fix acceptence tests, add them back into jenkins, upgrade express
2018-05-21 09:37:54 +01:00
Henry Oswald
df6b5203a1
point ci to gcr.io/csh-gcdm-test/
2018-05-18 17:18:39 +01:00
Henry Oswald
28fc39d6c9
don't use quay.io in image name
2018-05-18 17:08:18 +01:00
Henry Oswald
bfb98050e3
don't push to quay.io
2018-05-18 16:57:53 +01:00
Henry Oswald
088d8850ba
don't run acceptence tests yet
2018-05-18 16:48:59 +01:00
Henry Oswald
fcb2d18f8b
don't set command in make test acceptence
2018-05-18 16:15:44 +01:00
Christopher Hoskin
b32c63e16c
Remove --exit option to support older mocha
2018-05-18 14:59:34 +01:00
Christopher Hoskin
7fc7252c35
Remove touch package-lock.json
2018-05-18 14:29:20 +01:00
Christopher Hoskin
8db861258a
Add files generated by bin/update_build_scripts
2018-05-18 14:27:46 +01:00
Christopher Hoskin
5d5cbbef7e
Create fake package-lock.json during build tagret
2018-05-18 14:15:02 +01:00
Christopher Hoskin
a2165d85a0
Update build script to use build-target docker
2018-05-18 13:55:33 +01:00
Christopher Hoskin
de74ec5094
Change build and publish to use the GCR of one of my projects (hard code for now)
2018-05-18 13:47:06 +01:00
Henry Oswald
46f2ff051c
cd into tmp
2018-03-23 17:29:21 +00:00
Henry Oswald
b45e2a922c
install gs correctly
2018-03-23 17:21:15 +00:00
Henry Oswald
8c782ba6b2
debuging
2018-03-23 17:12:37 +00:00
Henry Oswald
801adf8e9e
install gs manually
2018-03-23 17:06:31 +00:00
Henry Oswald
092a151d51
Use new build system
2018-03-23 16:36:41 +00:00
Henry Oswald
837a8ff35f
change uploads permission after creation of user
2018-03-23 16:26:42 +00:00
Henry Oswald
1af209cd24
chown uploads
2018-03-23 16:23:04 +00:00
Henry Oswald
2f0cc4e64a
chown uploads
2018-03-23 16:19:52 +00:00
Henry Oswald
73f77b5667
copy everything into docker container
2018-03-23 16:14:30 +00:00
Henry Oswald
9e321658ed
make s3 configurable via env vars
2018-03-23 15:29:24 +00:00
Henry Oswald
77e34d33df
hello james
2018-01-16 17:00:27 +00:00
Henry Oswald
08c734e850
another bump
...
git log
2018-01-16 14:48:21 +00:00
Henry Oswald
dbfebc0d1c
bump
2018-01-16 14:36:09 +00:00
Henry Oswald
e235651e9c
bump version
2018-01-16 12:44:26 +00:00
Henry Oswald
c23b700927
wip
2018-01-16 12:27:36 +00:00
James Allen
b834e53e13
Provide hosts as environment settings and add npm run start script
2017-12-29 08:14:23 +00:00
Joe Green
a70a216c52
Update Jenkinsfile
2017-10-16 14:10:06 +01:00
Joe Green
6dcef9fdea
alert only on master
2017-10-12 17:07:06 +01:00
Henry Oswald
0e2de1e5ad
added shrinkwrap file
2017-09-15 14:57:01 +01:00
Joe Green
bfef9a9d67
added build.txt
2017-09-04 14:49:14 +01:00
Joe Green
706dc917dd
Create Jenkinsfile
2017-08-11 11:12:32 +01:00
Shane Kilkelly
e162c779d3
Merge pull request #24 from sharelatex/sk-ignore-png-optimiser-timeout
...
If optipng is sigkilled by timeout, then ignore that error
2017-08-03 09:22:43 +01:00
Shane Kilkelly
eaa72d0747
upgrade to node 6
2017-06-27 14:53:20 +01:00
Shane Kilkelly
23fe93ccfd
Prevent double-logging
2017-06-22 15:28:08 +01:00
Shane Kilkelly
ba9477cdb3
Re-work error checking
2017-06-22 09:41:36 +01:00
Shane Kilkelly
d0bbe8beae
If optipng is sigkilled by timeout, then ignore that error
2017-06-21 15:18:36 +01:00
Shane Kilkelly
2f8c693be3
Add a .nvmrc file
2017-03-27 14:49:25 +01:00
Shane Kilkelly
b12f69722f
Begin fixing health-check
2017-03-08 14:59:34 +00:00
Henry Oswald
e0f39da616
null check fsPath on unlink file
2016-12-19 14:48:09 +00:00
Henry Oswald
9e8c27f46b
don’t shutdown filestore on uncaught exception
2016-12-19 14:47:54 +00:00
Henry Oswald
bf93fc3f50
run health check max 1 a at a time
2016-12-19 13:26:21 +00:00
Henry Oswald
fd99887753
running check in background
2016-12-19 12:29:08 +00:00
Henry Oswald
6c81d18627
bumped up rlimit-nproc to 600
2016-12-14 18:15:09 +00:00
Henry Oswald
d41bf2e974
result path not in final cb
2016-12-14 18:11:28 +00:00
Henry Oswald
4bba56ed28
added missing files
2016-12-14 17:14:15 +00:00
Henry Oswald
27139bc433
increased rlimit-nproc and added health check for converting image
2016-12-14 16:47:53 +00:00
Henry Oswald
b166c86197
added missing originalFsPath
2016-12-13 14:29:04 +00:00
Henry Oswald
b38e7d3078
* delete both file and converted file if there is an error
...
* increase timeout of optpng
2016-12-13 12:43:57 +00:00
Shane Kilkelly
4064956cea
working minimal profile
2016-09-28 10:05:27 +01:00
Shane Kilkelly
edb5987c3d
A working profile, tested.
2016-09-27 15:54:41 +01:00
Shane Kilkelly
ff39a9da22
sort the list of syscalls
2016-09-26 15:35:53 +01:00
Shane Kilkelly
21be88482b
Change the seccomp whitelist to match that of docker default list
2016-09-26 15:33:22 +01:00
Brian Gough
fd95599609
remove old firejail includes
2016-08-11 11:52:58 +01:00
James Allen
a95aae16e6
Tweak convert profile to work on api-2
2016-05-10 17:50:46 +01:00
James Allen
a41a307b58
Update firejail profile to allow pdf previews for templates to work
2016-05-10 14:34:48 +01:00
James Allen
92f5234580
Add convert firejail profile
2016-05-09 14:44:49 +01:00
James Allen
be43330208
Allow convert command to be prefixed by security commands
2016-05-09 11:37:35 +01:00
Henrique Santos
ffaac4c81b
add unit tests
2016-03-13 20:45:48 -03:00
Henrique Santos
0bb12f32fa
remove project controller / handler
...
add unit test scaffolding
2016-03-13 16:22:14 -03:00
Henrique Santos
1f20744a68
add size function for each persitor
2016-03-12 11:08:07 -03:00
Henrique Santos
5d2fc89137
scaffolding project methods
2016-03-12 04:35:49 -03:00
Henrique Santos
099d0fef2f
scaffolding project size
2016-03-11 22:31:46 -03:00
James Allen
8ddfbb99de
Only parse body when we need to for copyFile request
2015-11-25 17:01:19 +00:00
James Allen
bc6c8d6afc
Don't send multiple status codes on insertFile error
2015-11-25 16:07:51 +00:00
Xavier Trochu
81a7203315
Fix the checkIfFileExists API.
2015-11-25 15:40:35 +01:00
Xavier Trochu
3b6270236c
Replace indentation from 2 space to tabs.
2015-11-20 14:38:23 +01:00
Xavier Trochu
b6486cb825
Cleanup the dependencies
2015-11-20 14:05:54 +01:00
Xavier Trochu
0219900933
Forward the options argument to getFileStream
2015-11-20 12:07:04 +01:00
Xavier Trochu
bfd41fdaf9
Add aws-sdk unit test.
...
Fix Aws-Sdk persistor to return a correct error on file not found.
Fix FileHandler after some change were lost on a previous merge
2015-11-20 12:02:22 +01:00
Xavier Trochu
476db58c3f
Add the opts parameter to getFileStream and implement Range requests
2015-11-17 12:52:25 +01:00
Xavier Trochu
26523b011a
small fixes
2015-11-16 09:59:27 +01:00
Xavier Trochu
63aef6b832
Fix postmerge issue
2015-11-05 12:11:02 +01:00
Xavier Trochu
a367190a21
Merge remote-tracking branch 'sharelatex/master'
2015-11-04 15:20:15 +01:00
James Allen
f723b1699b
Mongo/redis no longer needed without acceptance tests
2015-10-07 18:39:47 +01:00
James Allen
9b40d34b2a
Acceptance tests no longer run on Travis
2015-10-07 18:39:24 +01:00
Henry Oswald
cf28ee4f9b
locked down packages
2015-09-17 10:34:27 +01:00
Shane Kilkelly
0d450b4e33
Tighten up error handling.
2015-09-10 11:10:30 +01:00
Shane Kilkelly
fcfc595c35
Handle potential error before calling LocalFileWriter.writeStream
.
2015-09-10 09:12:49 +01:00
Shane Kilkelly
a6fff2d4e4
Check for a large response body, rather than exact match on the first few bytes.
2015-09-08 11:50:43 +01:00
Shane Kilkelly
c6a81c94eb
Test the cacheWarm
option for preview.
2015-09-08 11:47:43 +01:00
Shane Kilkelly
b8042ae771
Add an acceptance test that uses a pdf file.
2015-09-08 11:43:19 +01:00
Shane Kilkelly
297ad78b1e
Add an acceptance test to check the preview endpoint doesn't time out.
...
This endpoint needs a lot more testing.
2015-09-08 10:32:41 +01:00
Shane Kilkelly
da5a538095
Fix calls to PersistorManager.getFileStream
which were missed in initial refactor.
...
This should fix a breakage in the template publishing workflow.
Ref: b70b837b7ba29b0631f3105ec1dd5f6cdebbd195
2015-09-08 09:23:08 +01:00
Shane Kilkelly
9b8f11f517
Fix the check for a success response from S3.
...
We need to handle both a 200 and 206 (partial content) response.
2015-09-01 12:25:10 +01:00
Shane Kilkelly
82af1be756
Improve testing of error conditions in the FSPersistorManager.
2015-09-01 12:10:30 +01:00
Shane Kilkelly
edc06c82e5
Correct usage of '==' rather than '=' when checking err.code.
2015-09-01 12:10:05 +01:00
Shane Kilkelly
8bbf81f4bc
If the S3 service produces an error, handle it.
2015-09-01 11:36:28 +01:00
Shane Kilkelly
47fdf3c5c9
Ensure that Filestore responds with 404 when a file does not exist.
2015-08-31 16:47:16 +01:00
Shane Kilkelly
4d60872782
Merge branch 'master' of github.com:sharelatex/filestore-sharelatex
2015-08-28 14:48:25 +01:00
Shane Kilkelly
959191ae50
Add an acceptance test for using range header when getting a file.
2015-08-28 14:20:14 +01:00
Shane Kilkelly
ed980a21e1
Test the FileHandler layer.
2015-08-28 13:50:40 +01:00
Shane Kilkelly
64f70eddbd
Add cluster.js to .gitignore.
2015-08-28 13:38:54 +01:00
Shane Kilkelly
8a631e70f7
Refactor, and log when we have a 'range' header.
2015-08-28 12:16:41 +01:00
Shane Kilkelly
387e96ab67
Refactor to make _get_range testable.
2015-08-28 12:02:50 +01:00
Shane Kilkelly
97cdc243d3
Parse out the range header and set start
and end
properties on options.
2015-08-28 11:45:16 +01:00
Shane Kilkelly
b905d9fba1
Add range-parser
dependency.
2015-08-28 11:44:47 +01:00
Shane Kilkelly
76506ed91e
Pass start/end options down to the S3PersistorManager.
...
Test that those options cause a 'Range' header to be attached to the
request sent to S3.
2015-08-28 10:09:41 +01:00
Shane Kilkelly
d88736e3b5
Pass the start
and end
options down to fs.createReadStream
, and test for same.
2015-08-27 16:12:11 +01:00
Shane Kilkelly
57aedefdd3
Begin work towards obeying 'Range' headers, thread the opts
down to persistence layer.
2015-08-27 15:21:33 +01:00
Brian Gough
bea9ee6b19
add gc to filestore
2015-08-18 15:36:07 +01:00
Brian Gough
2e08281b88
replace the /gc endpoint with a automatic gc every 3 minutes
2015-08-05 16:52:01 +01:00
Brian Gough
e1683b8e1c
added a /gc endpoint to debug filestore memory usage
2015-08-04 11:26:51 +01:00
Xavier Trochu
f417b9b171
Fix the AWS-SDK Persistor
2015-07-24 15:48:46 +02:00
Xavier Trochu
7ef46a79a0
Fix aws-sdk persistor.
...
Also fix some typos.
2015-07-24 09:06:52 +02:00
Xavier Trochu
41397821b1
Add a manager using the aws-sdk library
...
The knox library does not support the AWS Instance Role.
So use the official AWS SDK to connect to S3
2015-07-22 16:42:45 +02:00
Xavier Trochu
3289451ece
Revert "Make S3 Key/Secret pair optional."
...
This reverts commit 65122e603099a4f9ef5c5186b7eb8832340a993f.
knox requires the key and secret arguments.
2015-07-22 16:41:45 +02:00
Xavier Trochu
566e69c6cc
Make S3 Key/Secret pair optional.
...
When using Instance Role in AWS, the credentials are unneeded. So
make them optional to use.
2015-07-21 10:00:59 +02:00
Xavier Trochu
c9b5bb028d
Merge branch 'master' of https://github.com/sharelatex/filestore-sharelatex
2015-07-21 09:02:16 +02:00
Brian Gough
9b84422092
added event loop monitor
2015-06-23 13:41:14 +01:00
Brian Gough
8ebbacee5a
move pipe after error handler setup
2015-05-08 15:19:45 +01:00
Brian Gough
7e9f8b885b
add support for public files
2015-05-08 15:15:48 +01:00
Henry Oswald
d5f98c9800
make startup message consistent
2015-04-30 15:04:55 +01:00
Henry Oswald
33e654accf
get the logging locally to work with bunyan
2015-04-22 18:27:34 +01:00
Henry Oswald
7c2e4b00ed
no file subtype can be an issue, added a check on it in the keybuilder
2015-04-13 12:01:49 +01:00
Henry Oswald
158b640b59
Merge branch 'master' of https://github.com/sharelatex/filestore-sharelatex
2015-04-09 17:16:23 +01:00
Henry Oswald
b3432f016c
allow proxying of template images though filestore
2015-04-09 17:14:47 +01:00
Brian Gough
07e53bbb2f
handle error when listing bucket
2015-03-31 10:35:33 +01:00
Xavier Trochu
da1b7b9e82
Merge remote-tracking branch 'origin/master'
2015-03-23 16:25:18 +01:00
James Allen
801b5653e4
Fix unit test when false is not at /bin/false
2015-03-20 14:27:25 +00:00
James Allen
a370739722
Release version 0.1.4
2015-03-20 14:20:13 +00:00
Brian Gough
a5e63f7261
Merge pull request #18 from sharelatex/terminate-processes
...
Use process group to terminate processes
2015-03-13 11:32:57 +00:00
Brian Gough
143d44e54b
add tests for SafeExec module
2015-03-13 10:15:53 +00:00
Brian Gough
a7b9376919
use SIGTERM instead of SIGKILL to allow process to shut down cleanly
2015-03-13 10:15:37 +00:00
Brian Gough
9aaef729ad
cleanup and comments
2015-03-13 10:10:51 +00:00
Brian Gough
63ee4d1e7d
use close event instead of exit to capture stdout/stderr correctly
2015-03-13 10:10:21 +00:00
Brian Gough
0e5abe1ff8
update tests to use safe_exec
2015-03-13 09:31:43 +00:00
Brian Gough
33d8974d02
kill process group on timeout of convert commands
2015-03-12 17:09:27 +00:00
James Allen
eb89337ada
Release version 0.1.3
2015-02-26 11:32:21 +00:00
James Allen
45689fd2b8
Only call getFileStream callback once
2015-02-26 11:32:05 +00:00
James Allen
eb40dbbe96
Release version 0.1.2
2015-02-10 13:19:04 +00:00
Xavier Trochu
4b54372466
miscellaneous cleanup
...
_ Remove some references to scribtex
_ Fix some issue when templates are unavailable
2015-01-16 11:04:15 +01:00
Brian Gough
2c731586a8
Merge pull request #16 from sharelatex/cluster-restart
...
Graceful cluster restart on domain exception
2015-01-07 11:50:24 +00:00
Brian Gough
e03ee96216
added simple clustering master
2015-01-07 11:33:42 +00:00
Brian Gough
79292b4602
fix log message
2015-01-07 10:35:17 +00:00
Brian Gough
191d0a4986
handle SIGTERM in filestore
2015-01-07 10:35:17 +00:00
Brian Gough
02101868e1
put all shutdown logic in beginShutdown, force program to exit after 2 mins
2015-01-07 10:35:17 +00:00
Brian Gough
e8e2338c0c
move connection:close middleware ahead of other routes
2015-01-05 16:56:19 +00:00
Brian Gough
d5478a8ebb
make sure shutdown only happens once
2014-12-23 16:55:34 +00:00
Brian Gough
d3fa6b4a01
catch exceptions within the body of the domain error handler
2014-12-23 16:49:34 +00:00
Brian Gough
48f62990b4
server can exit immediately, does not need a domain
...
exit with an exception as no requests can be served
2014-12-23 16:48:11 +00:00
Brian Gough
74fc095913
close HTTP keep-alive connections when shutting down
2014-12-23 16:46:31 +00:00
Brian Gough
f4ac7d422b
run acceptance tests on travis under forever
2014-12-22 13:41:07 +00:00
Brian Gough
ad379ec103
add grunt target for test:acceptance
2014-12-22 13:13:36 +00:00
Brian Gough
d4c281e22c
add acceptance tests to travis.yml
2014-12-22 13:10:40 +00:00
Brian Gough
8b9213d09b
fix getFileStream unit test as callback no longer fires immediately
...
success callback is now called only when file is readable
2014-12-22 13:07:30 +00:00
Brian Gough
efd6035849
test against rimraf module when deleting directory recursively
2014-12-22 13:01:39 +00:00
Brian Gough
3f18f6a714
update unit tests - FileHandler now calls deleteDirectory on cache directory
2014-12-22 12:58:41 +00:00
Brian Gough
64c6341844
check delete status code in copy test
2014-12-22 11:34:13 +00:00
Brian Gough
de465aee94
change copy test to match usage in web, check http status
2014-12-22 11:27:48 +00:00
Brian Gough
dad209262f
execute success callback when filesystem copyfile completed
2014-12-22 11:27:00 +00:00
Brian Gough
72ebaea5e2
log FS delete errors only on failure
2014-12-22 11:11:31 +00:00
Brian Gough
7d96f3aadd
use 'response' to send 'NoSuchKey' when local file is not found
2014-12-22 11:11:03 +00:00
Brian Gough
8744cf94a5
delete directory with rimraf
2014-12-19 16:58:31 +00:00
Brian Gough
af4c72150f
fix path in directory deletion
2014-12-19 16:57:44 +00:00
Brian Gough
55c4e220fa
prevent double callback in getFileStream
2014-12-19 16:28:42 +00:00
Brian Gough
c74356daeb
Merge branch 'master' of github.com:sharelatex/filestore-sharelatex
...
# Please enter a commit message to explain why this merge is necessary,
# especially if it merges an updated upstream into a topic branch.
#
# Lines starting with '#' will be ignored, and an empty message aborts
# the commit.
2014-12-18 16:57:27 +00:00
Brian Gough
c2e50c619d
request restart on unhandled exceptions to avoid memory leak
2014-12-18 16:55:11 +00:00
Henry Oswald
e90eb4b3fb
bump knox
2014-11-29 13:02:06 +00:00
Henry Oswald
ca851589c9
Merge branch 'master' of https://github.com/sharelatex/filestore-sharelatex
...
Conflicts:
package.json
2014-11-27 13:49:16 +00:00
Henry Oswald
dee7799fd4
added heapdump and endpoint to make dump
2014-11-27 13:48:39 +00:00
James Allen
9d97d1dc53
Handle templates locally ok
2014-11-27 11:08:29 +00:00
James Allen
72ad81887a
Release version 0.1.0
2014-08-19 14:10:48 +01:00
James Allen
b79dfcd477
Lock down module versions
2014-08-19 14:10:37 +01:00
Henry Oswald
72df59e81e
added copy file acceptence test
2014-06-19 17:32:04 +01:00
Henry Oswald
5d00f70a31
added delete accp test
2014-06-19 17:16:45 +01:00
Henry Oswald
f83ab25b35
basic acceptence test of sending and getting file back added
2014-06-19 17:07:50 +01:00
Henry Oswald
04bffaeceb
get local file writer tests using fake settings
2014-06-17 13:33:36 +01:00
Henry Oswald
c8c758642a
removed reference to easy image
2014-06-17 12:47:53 +01:00
Henry Oswald
088e494c30
Merge branch 'master' of github.com:sharelatex/filestore-sharelatex
2014-06-17 12:44:41 +01:00