Jakob Ackermann
2cf9ff7b41
Merge pull request #89 from overleaf/jpa-is-doc-deleted
...
[misc] add a new endpoint for querying the deleted status of a doc
2021-02-09 11:14:28 +00:00
Jakob Ackermann
3ef1f91f47
Merge pull request #86 from overleaf/jpa-drop-expensive-default
...
[misc] MongoManager: drop unused, potentially expensive default options
2021-02-09 11:14:02 +00:00
Jakob Ackermann
28b1ad3243
[misc] add a new endpoint for querying the deleted status of a doc
...
`/project/:project_id/doc/:doc_id/deleted` responds with:
- 404: the doc does not exist
- 200 and body `{"deleted":true}`: doc exists and is deleted
- 200 and body `{"deleted":false}`: doc exists and is not deleted
2021-01-15 14:49:27 +00:00
Jakob Ackermann
1edc628454
[misc] MongoManager: drop unused, potentially expensive default options
...
All call-sites specify an options object.
2021-01-04 16:07:14 +00:00
Jakob Ackermann
9298f486ad
[app] add a new route for archiving a single doc
...
There is a minor opportunity for optimizing the mongo call(s) for
archiving: we should include `{ inS3: false }` in the queries instead
of filtering locally. It's been like that for years now, so it can
stay a little longer.
2021-01-04 13:47:55 +00:00
Simon Detheridge
31c757cce2
Don't throw 404 errors when unarchiving, if the doc is already unarchived
2020-09-25 09:17:01 +01:00
Jakob Ackermann
3644db6848
[misc] mongodb: use the new db connector by default
...
mongojs was enabling it by default as well.
2020-09-17 15:40:23 +01:00
Jakob Ackermann
b37e27f023
[misc] run the codemod for moving mongo projections into options
2020-09-17 15:40:20 +01:00
Jakob Ackermann
54a82b8c62
[misc] mongodb: export a helper for adding new collections to db
...
Co-Authored-By: Eric Mc Sween <eric.mcsween@overleaf.com>
Co-Authored-By: Simon Detheridge <s@sd.ai>
2020-09-17 15:39:46 +01:00
Jakob Ackermann
1e7fa2246d
[misc] mongodb: refactor the process of setting up the db construct
...
Co-Authored-By: John Lees-Miller <jdleesmiller@gmail.com>
2020-09-17 15:39:46 +01:00
Jakob Ackermann
8a8a830ad3
[misc] simplify mongodb collection access using a shared db construct
...
Resolve the getCollection Promises once and store the result in a shared
`db` object which can get imported by all the call-sites.
The http server is starting only after a Promise of `waitForDb()`
resolves. This covers the app code and the acceptance tests.
2020-09-17 15:39:44 +01:00
Simon Detheridge
9ec93ca0d6
Merge pull request #75 from overleaf/spd-p-map
...
Replace tiny-async-pool with p-map
2020-09-17 15:37:37 +01:00
Simon Detheridge
90a6642b33
Replace tiny-async-pool with p-map
2020-09-14 16:34:34 +01:00
Simon Detheridge
ad4531ff01
Upgrade metrics to @overleaf/metrics'
2020-09-10 18:07:14 +01:00
Jakob Ackermann
2fa09928ee
[misc] migrate acceptance tests to the native mongo driver, drop mongojs
2020-08-25 09:41:23 +01:00
Jakob Ackermann
df0747ec48
[misc] migrate the app to the native mongo driver
...
acceptance tests to follow in a separate commit
2020-08-25 09:40:59 +01:00
Simon Detheridge
1575cd41eb
Don't pass empty array to asyncPool when archiving
2020-08-10 15:44:41 +01:00
Simon Detheridge
cb8c2bf4b5
AsyncPool -> asyncPool
2020-07-31 16:21:59 +01:00
Simon Detheridge
9724407e31
Export .promises from MongoManager
2020-07-31 16:20:07 +01:00
Simon Detheridge
268f436461
Don't call AsyncPool with an empty array
2020-07-24 11:51:18 +01:00
Simon Detheridge
6e04db552a
Implement object-persistor support
2020-07-23 19:42:49 +01:00
Ersun Warncke
06ec5556b0
[misc] fix express deprecations
...
Co-Authored-By: Jakob Ackermann <jakob.ackermann@overleaf.com>
2020-07-02 14:48:11 +01:00
Tim Alby
c854f1f570
prettier auto-format
2020-05-28 15:20:54 +02:00
Simon Detheridge
746791766c
Additional prettier fix
2020-02-17 10:49:32 +00:00
Simon Detheridge
fb931e206c
prettier: convert app/js decaffeinated files to Prettier format
2020-02-17 10:49:00 +00:00
Simon Detheridge
c1805978c4
decaffeinate: rename app/coffee dir to app/js
2020-02-17 10:48:58 +00:00
decaffeinate
d31a747c72
decaffeinate: Run post-processing cleanups on DocArchiveManager.coffee and 7 other files
2020-02-17 10:48:57 +00:00
decaffeinate
afa2577381
decaffeinate: Convert DocArchiveManager.coffee and 7 other files to JS
2020-02-17 10:48:55 +00:00
decaffeinate
88badb15c4
decaffeinate: Rename DocArchiveManager.coffee and 7 other files from .coffee to .js
2020-02-17 10:48:53 +00:00
Simon Detheridge
03b8a1901d
Account for ranges in large json payloads and add line-count limit
2019-08-06 09:50:04 +01:00
Simon Detheridge
5261e1b0be
Add endpoint to completely purge a document
2019-07-02 15:02:37 +01:00
Henry Oswald
7e3c6ddb3f
also delete doc ops
2018-10-19 12:10:33 +01:00
Henry Oswald
2828df74e5
hard delete a doc on health check
2018-10-18 18:06:05 +01:00
James Allen
a4cc02dbc3
Provide hosts as environment settings and add npm run start script
2017-12-29 08:12:31 +00:00
James Allen
d86a964081
Remove another verbose log object
2017-09-20 09:52:28 +02:00
James Allen
05a1939547
Don't log out doc lines and ranges
2017-09-20 09:42:08 +02:00
Shane Kilkelly
1f3b0a3713
Upgrade mongojs dependency to version 2.4.0
...
- Upgrade in package.json
- Use new connection syntax
- Fix unit tests, issues with object equality
2017-08-22 08:26:30 +01:00
Brian Gough
abe90cad10
Revert "Merge pull request #23 from sharelatex/bg-optimise-get-all-docs"
...
This reverts commit c1337cf23f37bd9ba48419accf1f54c00390e0ea, reversing
changes made to fbb2fa15bf4239db006db742f906554d61a0eac4.
2017-06-30 14:54:11 +01:00
Brian Gough
a2ac37ac99
comments about optimistic loading
2017-06-28 10:06:50 +01:00
Brian Gough
b9b0596d83
optimistically load all docs
2017-06-27 16:58:20 +01:00
James Allen
e10d09d040
Merge pull request #22 from sharelatex/ja-archive-ranges
...
Archive ranges as well as doc lines
2017-05-17 12:02:42 +01:00
Shane Kilkelly
6a6fdc13a5
Merge branch 'sk-upgrade-metrics'
2017-04-04 09:34:45 +01:00
James Allen
7a64f040f1
Test and fix _mongoDocToS3Doc
2017-04-03 16:04:54 +01:00
James Allen
f94e33a3a9
Merge branch 'master' into ja-archive-ranges
...
Conflicts:
app/coffee/DocArchiveManager.coffee
test/acceptance/coffee/ArchiveDocsTests.coffee
2017-03-30 17:19:42 +01:00
James Allen
a9ab7b739d
Archive ranges as well as doc lines
2017-03-30 17:13:43 +01:00
Shane Kilkelly
bc37c844ea
update to new metrics api
2017-03-17 15:27:36 +00:00
Shane Kilkelly
e879f9c742
Upgrade metrics
2017-03-16 16:46:58 +00:00
Henry Oswald
c0e23df82b
break getDoc into multiple helpers, stop passing filters though from other calls
2017-03-15 22:09:56 +00:00
Henry Oswald
ffd1aa0ca4
just get doc_id for quick check to see if it exists
2017-03-10 11:45:03 +00:00
Henry Oswald
7adc67d13a
add missing error handler
2017-03-10 10:37:11 +00:00
Henry Oswald
c2b96c7e1e
Merge pull request #17 from sharelatex/ho-always-check-s3
...
Always get inS3 when getting doc
2017-03-10 10:36:59 +00:00
Henry Oswald
f067c039fe
always search for inS3 when getting a doc
2017-03-09 18:51:54 +00:00
Henry Oswald
8bd0df3530
also log out lines in error message
2017-03-09 17:36:28 +00:00
Henry Oswald
dbfe5ac712
error if doc lines are not array when unarchiving
2017-03-09 17:34:33 +00:00
James Allen
315a1d5e80
Add ranges into health check
2017-01-31 10:23:51 +01:00
James Allen
fb39bb0442
Validate that ranges are present in a set document request
2017-01-19 12:15:14 +01:00
James Allen
5e730b33b2
Support new comment op format
2016-12-16 16:44:08 +00:00
James Allen
0174495f75
Add an end point to get all the ranges in a project
2016-12-09 14:37:24 +00:00
James Allen
6fa8b89154
Be explicit about the data we return from mongo
2016-12-05 17:27:31 +00:00
James Allen
a6ec672d84
Merge branch 'ja-delete-refactor' into ja-track-changes
...
Conflicts:
test/acceptance/coffee/ArchiveDocsTests.coffee
2016-12-05 16:58:29 +00:00
James Allen
3d195de3e9
Improve testing of archiving and be explicit about include_deleted parameter
2016-12-05 16:48:01 +00:00
James Allen
7717625534
Include deleted docs when archiving
2016-12-05 16:31:51 +00:00
James Allen
2985cb587a
Add tests for RangeManager
2016-12-05 15:38:38 +00:00
James Allen
b8a97731e4
Merge branch 'master' into ja-track-changes
...
Conflicts:
test/acceptance/coffee/GettingAllDocsTests.coffee
2016-12-05 14:58:57 +00:00
James Allen
bd5086c054
Don't return deleted docs with all project docs
2016-12-05 14:37:44 +00:00
James Allen
0a65a5eabf
Set and get ranges along with lines
2016-12-05 14:21:49 +00:00
James Allen
ca96d168dd
Make attributes that can be set in mongo flexible
2016-12-02 15:22:08 +00:00
James Allen
144bae3a55
Add an error guard around the version being present
2016-12-02 15:17:38 +00:00
James Allen
ab527c3c5d
remove code left over from migration
2016-12-02 15:02:54 +00:00
James Allen
e156d8017c
Be explicit in version filter
2016-11-30 15:29:43 +00:00
James Allen
8b95a04a36
Don't look up version in mongo if not needed
2016-11-30 15:08:56 +00:00
James Allen
50f2983912
Log out version when updating doc
2016-11-29 16:49:35 +00:00
James Allen
eb9a3b314a
Set/get version in docOps via docstore HTTP API
2016-11-28 14:55:16 +00:00
Brian Gough
7a28fb58e7
clean up mongo query
2016-09-05 11:14:41 +01:00
Brian Gough
0a0a8e23fe
remove unused modules
2016-09-05 11:14:41 +01:00
Henry Oswald
1541eb319c
add more null checks to health check
2015-12-02 10:10:52 +00:00
Henry Oswald
a964119b5e
check err in health check and null check res
2015-12-01 21:50:18 +00:00
Henry Oswald
29c812cece
moved the build opts down
2015-10-16 12:57:05 +01:00
Henry Oswald
4f5669591a
add timeout to health check requests and always try the delete
2015-10-16 12:30:53 +01:00
Henry Oswald
de9a24f7bb
pulled health check into its own file and use project_id from settings
2015-10-16 10:13:54 +01:00
Henry Oswald
f1bf0c97ec
make archive and unarchive parallelLimit to speed it up
2015-09-22 09:41:26 +01:00
Henry Oswald
9cfa4b3f84
Don't put docs which are already in s3 through archive job
...
This fixes the bug we saw with 'RangeError: Maximum call stack size exceeded'
if lots of docs are already in s3 cb() gets called synchronously multiple times quickly
which can cause the execption. I am not sure where the recursion is, maybe inside async.
doing setImmediate(cb) also fixes the issue as I beilve it gives the
process chance to clear the stack. Similar to process.nextTick
2015-09-07 14:11:02 +01:00
Henry Oswald
738e33a052
change crypto to always use utf-8 as it works with aws
2015-09-01 16:52:02 +01:00
Henry Oswald
599b9770f4
add un archive endpoint
2015-08-13 22:48:45 +01:00
Henry Oswald
ef4e437c3f
improve doc manager so it calls get doc recursivly
2015-08-13 22:47:33 +01:00
Henry Oswald
b8884352f6
added some better error logging
2015-08-13 13:59:22 +01:00
Henry Oswald
859e8a0e57
- deal with calling archive twice on same project
...
- clean up unit tests
- change s3 keys
- use err not error
2015-08-13 13:00:09 +01:00
Henrique Dias
42f2afc4dd
remove unarchived docs from s3
2015-06-03 20:05:15 -03:00
Henrique Dias
aa23c1da56
md5 check for doc lines
2015-06-03 11:45:47 -03:00
Henrique Dias
5ba14522da
handling error
2015-06-02 20:08:50 -03:00
Henrique Dias
a26320013d
more acceptance tests
2015-06-02 19:24:45 -03:00
Henrique Dias
0cd2120430
more unit tests for archive feature
2015-06-02 18:13:16 -03:00
Henrique Dias
5c31e80062
more refactoring
2015-06-02 16:29:32 -03:00
Henrique Dias
ddd26798f1
refactoring
...
s3 filter in router
2015-06-02 15:55:22 -03:00
Henrique Dias
86f16caeef
fix some issues
2015-06-02 15:13:28 -03:00
Henry Oswald
ea92f57acc
point where we unset doc lines also search on doc rev to make sure
...
the document has not been updated
2015-06-02 12:39:49 +01:00
Henry Oswald
60bc7bf4e1
pulled logic for arching docs arpart a bit, can be moved into its own file
...
hacked out (needs tests) the unarchiving of individual docs & doc updates
2015-06-02 12:36:38 +01:00
Henry Oswald
19f1a9ad82
changed style of async jobs to map to use underscore which is like we
...
use in the rest of the stack. Also removed a few => binds which are
not needed
2015-06-02 10:59:35 +01:00
Henrique Dias
7c7ff64904
retrieving all docs from s3
2015-06-01 19:36:26 -03:00
Henrique Dias
7f5204f0f7
initial version for archive docs in S3
2015-06-01 18:24:40 -03:00
