Simon Detheridge
0329c759dc
Remove wrapper function in PersistorManager
2020-01-07 09:26:43 +00:00
Simon Detheridge
dbfacce988
Post-decaf cleanup of PersistorManager
2020-01-07 09:26:43 +00:00
Simon Detheridge
ec80052d9a
Decaf cleanup of FileController
...
Remove deprecated res.send
2020-01-07 09:26:23 +00:00
Simon Detheridge
fccb288985
Add comment regarding deleteDirectory being internal-only
2020-01-07 09:26:01 +00:00
Simon Detheridge
f1b6b35c69
Throw errors more consistently in FSPersistorManager
2020-01-07 09:26:00 +00:00
Simon Detheridge
25f1c2bfc4
Delete temporary file when error in writing to stream
2020-01-07 09:26:00 +00:00
Simon Detheridge
21ba083877
Use 'glob' to determine 'directory' size.
...
Check for filtered directory paths in all tests.
2020-01-07 09:26:00 +00:00
Simon Detheridge
ce90292394
Decaf cleanup for FSPersistorManager
2020-01-07 09:26:00 +00:00
Simon Detheridge
006f84abeb
Decaf and promisify ImageOptimiser
2020-01-07 09:24:29 +00:00
Simon Detheridge
42adc59d01
Clean up and promisify health-check controller and KeyBuilder
2020-01-07 09:23:41 +00:00
Simon Detheridge
ac2d05ecb3
Remove unnecessary logging from FileConverter
2020-01-07 09:22:53 +00:00
Simon Detheridge
b6a1ea6a30
Cleanup and promisify FileConverter
2020-01-07 09:22:53 +00:00
Simon Detheridge
a216be5fd3
Improve error handling in FileHandler
2020-01-07 09:18:59 +00:00
Simon Detheridge
27aaff7843
Decaf cleanup for FileHandler and LocalFileWriter
...
Simplified code and tests where possible
2020-01-07 09:18:56 +00:00
Simon Detheridge
eacad77112
Cleanup SafeExec
2019-12-19 15:42:46 +00:00
Simon Detheridge
e06c03e536
Fix decaf error in FSPersistorManager
2019-12-16 11:19:56 +00:00
Simon Detheridge
6225f2f236
Prettier: convert app/js decaffeinated files to Prettier format
2019-12-16 11:19:36 +00:00
Simon Detheridge
3718c90c7a
Rename app/coffee dir to app/js
2019-12-16 10:54:08 +00:00
decaffeinate
fcbb13b5bf
decaffeinate: Run post-processing cleanups on AWSSDKPersistorManager.coffee and 13 other files
2019-12-16 10:54:06 +00:00
decaffeinate
6bd8452f19
decaffeinate: Convert AWSSDKPersistorManager.coffee and 13 other files to JS
2019-12-16 10:54:03 +00:00
decaffeinate
8d2c87420e
decaffeinate: Rename AWSSDKPersistorManager.coffee and 13 other files from .coffee to .js
2019-12-16 10:53:59 +00:00
Jakob Ackermann
ed97fcfcd0
[logging] do not overwrite the logger name
...
Signed-off-by: Jakob Ackermann <das7pad@outlook.com>
2019-12-16 10:53:34 +00:00
Jakob Ackermann
c5e1584fcc
[FSPersistorManager] fix the stream opening for node10+
...
Attaching a `readable` listener causes the stream to hang otherwise.
Signed-off-by: Jakob Ackermann <das7pad@outlook.com>
2019-12-16 10:53:23 +00:00
Simon Detheridge
dab44f6fec
Merge branch 'master' into logging/keep-logger-name
2019-12-16 09:42:18 +00:00
Simon Detheridge
32b7f25404
Merge pull request #49 from das7pad/misc/node10plus-support
...
[misc] add support for node 10+
2019-12-16 09:39:00 +00:00
Simon Detheridge
6f326d5650
Use SSL setting based on url protocol
2019-12-09 17:41:20 +00:00
Simon Detheridge
48aa141591
Add metric for s3 ingress
2019-12-09 17:06:00 +00:00
Simon Detheridge
1d1106bc67
Add metric for s3 egress
2019-12-09 17:06:00 +00:00
Simon Detheridge
c01603b1e7
Support custom S3 endpoints
2019-12-09 17:06:00 +00:00
Simon Detheridge
86b9e4b53a
Rename request -> s3Request to prevent overwriting main import
2019-12-09 17:05:58 +00:00
Brian Gough
2ec38068aa
add comments about aws-sdk and s3 backends
2019-12-06 14:35:13 +00:00
Eric Mc Sween
7d900b57bf
Fix createUnbufferedStream() function call
...
In 49a21155f642670dfea264ac73fb60241f37cb87, I managed to incorrectly
write the `createUnbufferedStream()` function from the AWS SDK as
`getUnbufferedStream()` and to consistently use that naming in the unit
tests.
This commit fixes that. I have tested again on S3.
2019-06-19 12:58:17 -04:00
Eric Mc Sween
f2521a29b9
Use AWS SDK for getFileStream()
...
The AWS SDK has a retry strategy to deal with rate limiting or transient
unavailability of S3. We hope it will reduce our error rates.
2019-06-19 08:18:11 -04:00
Eric Mc Sween
3575c89d03
Return file size on HEAD request
...
This will be used by the file preview feature when it gets partial
content.
2019-06-17 08:23:59 -04:00
Jakob Ackermann
a79adcd325
[logging] do not overwrite the logger name
...
Signed-off-by: Jakob Ackermann <das7pad@outlook.com>
2019-05-02 12:48:43 +02:00
Jakob Ackermann
c59a3db4e8
[FSPersistorManager] fix the stream opening for node10+
...
Attaching a `readable` listener causes the stream to hang otherwise.
Signed-off-by: Jakob Ackermann <das7pad@outlook.com>
2019-05-02 02:47:53 +02:00
Henry Oswald
2adae20368
remove a console.log
2019-02-06 11:43:11 +00:00
Henry Oswald
5eec00db51
don't run conversions test when disabled
2019-02-05 14:06:51 +00:00
Henry Oswald
3fccf79ca8
cleanup
2019-02-05 12:50:08 +00:00
Henry Oswald
19007befb2
Merge branch 'master' into ho-docker
2019-02-01 16:26:45 +00:00
Brian Gough
9d93eee3e8
return a 404 error (instead of a 500) when copying a missing file
2019-01-09 10:31:59 +00:00
Henry Oswald
3614f217e6
add enableConversions flag to disable conversions which we can't do on
...
k8
2019-01-07 15:54:24 +00:00
Christopher Hoskin
aab5a1af6a
Merge branch 'master' into csh-ho-docker
2019-01-02 09:51:35 +00:00
Brian Gough
cf12ec1154
use the aws sdk to copy files in S3PersistorManager
...
to work around problems with knox
2018-12-20 16:11:18 +00:00
Brian Gough
8951f297aa
Merge pull request #37 from sharelatex/bg-reduce-logging-of-not-found-errors
...
reduce logging of not found errors
2018-11-13 14:06:19 +00:00
Brian Gough
0d6de10b5c
Merge pull request #34 from sharelatex/bg-make-backends-consistent
...
make filestore backends consistent
2018-11-13 10:12:18 +00:00
Brian Gough
23a6d6e81d
consider 403 and 404 as NotFound errors
2018-11-09 14:05:38 +00:00
Brian Gough
69b164092c
suppress unnecessary error logging for NotFound
2018-11-09 14:05:38 +00:00
Brian Gough
8744b3aa4e
serve file from disk to avoid read-after-write inconsistency
2018-11-08 15:09:34 +00:00
Brian Gough
40ac59090b
make backends consistent by deleting temporary files
2018-11-07 16:22:09 +00:00
Henry Oswald
38f565d35f
logout settings
2018-09-28 15:53:57 +01:00
Henry Oswald
aacb54d54c
Merge branch 'master' into ho-docker
2018-09-14 17:16:18 +01:00
Henry Oswald
089bf3e084
seperate the standard s3 creds from the bucket s3 creds
2018-07-10 14:17:37 +01:00
Michael Mazour
7feafccf31
Amend: safely navigate to bucket credentials
2018-07-06 09:28:09 +01:00
Michael Mazour
03033409c6
Amend: remove unused params
2018-07-06 09:12:59 +01:00
Michael Mazour
336a38ec1e
Amend: scrub secrets from logs
...
Calls to `getFile` can now include S3 credentials in `opts`, so sanitize before writing to opts to log.
2018-07-04 16:41:31 +01:00
Michael Mazour
600ab3ce67
Amend: remove problematic ampersand
2018-07-04 16:39:41 +01:00
Michael Mazour
cfbf0d81ba
Amend: fix params retrieval
2018-07-04 12:11:09 +01:00
Michael Mazour
ece650741a
Amend per several review comments
...
- Removed unused vars
- Label the metric with the bucket name
2018-07-04 12:02:09 +01:00
Michael Mazour
feca8933f1
Add endpoint for arbitrary bucket fetch
...
Add `/bucket/:bucket/key/*`, which fetches the file from the given bucket at the given path. Uses auth stored at `settings.filestore.s3.{{bucketName}}` if present, and otherwise default auth.
2018-07-04 11:22:36 +01:00
Henry Oswald
8efd562eb3
fix broken acceptence tests, bad quotes in file path
2018-05-21 13:40:03 +01:00
Henry Oswald
e3514e84cd
fix up mocha migration tests
2018-05-21 11:02:53 +01:00
Henry Oswald
3cf2e1d7c3
add null checks and new body parser
2018-05-21 10:11:47 +01:00
Henry Oswald
fda6cb0084
fix acceptence tests, add them back into jenkins, upgrade express
2018-05-21 09:37:54 +01:00
Shane Kilkelly
23fe93ccfd
Prevent double-logging
2017-06-22 15:28:08 +01:00
Shane Kilkelly
ba9477cdb3
Re-work error checking
2017-06-22 09:41:36 +01:00
Shane Kilkelly
d0bbe8beae
If optipng is sigkilled by timeout, then ignore that error
2017-06-21 15:18:36 +01:00
Shane Kilkelly
b12f69722f
Begin fixing health-check
2017-03-08 14:59:34 +00:00
Henry Oswald
e0f39da616
null check fsPath on unlink file
2016-12-19 14:48:09 +00:00
Henry Oswald
bf93fc3f50
run health check max 1 a at a time
2016-12-19 13:26:21 +00:00
Henry Oswald
fd99887753
running check in background
2016-12-19 12:29:08 +00:00
Henry Oswald
d41bf2e974
result path not in final cb
2016-12-14 18:11:28 +00:00
Henry Oswald
4bba56ed28
added missing files
2016-12-14 17:14:15 +00:00
Henry Oswald
b166c86197
added missing originalFsPath
2016-12-13 14:29:04 +00:00
Henry Oswald
b38e7d3078
* delete both file and converted file if there is an error
...
* increase timeout of optpng
2016-12-13 12:43:57 +00:00
James Allen
be43330208
Allow convert command to be prefixed by security commands
2016-05-09 11:37:35 +01:00
Henrique Santos
0bb12f32fa
remove project controller / handler
...
add unit test scaffolding
2016-03-13 16:22:14 -03:00
Henrique Santos
1f20744a68
add size function for each persitor
2016-03-12 11:08:07 -03:00
Henrique Santos
5d2fc89137
scaffolding project methods
2016-03-12 04:35:49 -03:00
Henrique Santos
099d0fef2f
scaffolding project size
2016-03-11 22:31:46 -03:00
James Allen
bc6c8d6afc
Don't send multiple status codes on insertFile error
2015-11-25 16:07:51 +00:00
Xavier Trochu
81a7203315
Fix the checkIfFileExists API.
2015-11-25 15:40:35 +01:00
Xavier Trochu
3b6270236c
Replace indentation from 2 space to tabs.
2015-11-20 14:38:23 +01:00
Xavier Trochu
0219900933
Forward the options argument to getFileStream
2015-11-20 12:07:04 +01:00
Xavier Trochu
bfd41fdaf9
Add aws-sdk unit test.
...
Fix Aws-Sdk persistor to return a correct error on file not found.
Fix FileHandler after some change were lost on a previous merge
2015-11-20 12:02:22 +01:00
Xavier Trochu
476db58c3f
Add the opts parameter to getFileStream and implement Range requests
2015-11-17 12:52:25 +01:00
Xavier Trochu
26523b011a
small fixes
2015-11-16 09:59:27 +01:00
Xavier Trochu
63aef6b832
Fix postmerge issue
2015-11-05 12:11:02 +01:00
Xavier Trochu
a367190a21
Merge remote-tracking branch 'sharelatex/master'
2015-11-04 15:20:15 +01:00
Shane Kilkelly
0d450b4e33
Tighten up error handling.
2015-09-10 11:10:30 +01:00
Shane Kilkelly
fcfc595c35
Handle potential error before calling LocalFileWriter.writeStream.
2015-09-10 09:12:49 +01:00
Shane Kilkelly
da5a538095
Fix calls to PersistorManager.getFileStream which were missed in initial refactor.
...
This should fix a breakage in the template publishing workflow.
Ref: b70b837b7ba29b0631f3105ec1dd5f6cdebbd195
2015-09-08 09:23:08 +01:00
Shane Kilkelly
9b8f11f517
Fix the check for a success response from S3.
...
We need to handle both a 200 and 206 (partial content) response.
2015-09-01 12:25:10 +01:00
Shane Kilkelly
82af1be756
Improve testing of error conditions in the FSPersistorManager.
2015-09-01 12:10:30 +01:00
Shane Kilkelly
edc06c82e5
Correct usage of '==' rather than '=' when checking err.code.
2015-09-01 12:10:05 +01:00
Shane Kilkelly
8bbf81f4bc
If the S3 service produces an error, handle it.
2015-09-01 11:36:28 +01:00
Shane Kilkelly
47fdf3c5c9
Ensure that Filestore responds with 404 when a file does not exist.
2015-08-31 16:47:16 +01:00
Shane Kilkelly
8a631e70f7
Refactor, and log when we have a 'range' header.
2015-08-28 12:16:41 +01:00
Shane Kilkelly
387e96ab67
Refactor to make _get_range testable.
2015-08-28 12:02:50 +01:00
Shane Kilkelly
97cdc243d3
Parse out the range header and set start and end properties on options.
2015-08-28 11:45:16 +01:00
