Eric Mc Sween
d9caced0d6
Change skip_flush option to ignore_flush_errors in delete doc
...
Instead of skipping the flush, we'll still try to flush and proceed with
the doc deletion, even when the flush fails.
2020-03-10 09:40:49 -04:00
Eric Mc Sween
9b70eb75b3
Rename flush param to skip_flush in delete doc
...
Also move it to the query string instead of the body.
2020-03-09 16:27:32 -04:00
Eric Mc Sween
c09bc0e868
Add a "flush: false" option to the doc delete endpoint
...
This will delete the document from Redis without flushing to web,
docstore or history. To be used when something is broken.
2020-03-07 08:59:15 -05:00
Brian Gough
0419039d4d
Merge branch 'master' into bg-use-separate-redis-for-project-history
2020-02-21 14:13:33 +00:00
Jakob Ackermann
82ab2f7d14
Merge pull request #65 from das7pad/hotfix-redis-config
...
[redis] use the correct instance for the real time updates
2020-02-12 13:21:04 +01:00
Jakob Ackermann
e73978410c
Merge pull request #67 from das7pad/hotfix-static-queue-check
...
[misc] tests: fix a static test
2020-02-12 13:20:55 +01:00
Brian Gough
26d8d07e7a
remove unused redis settings from rate limit unit test
2020-02-11 14:32:49 +00:00
Jakob Ackermann
4070b0ef6f
[misc] tests: fix a static test
...
`ProjectHistoryRedisManager.queueOps` is masked and can not push
changes into redis during the unittest.
Signed-off-by: Jakob Ackermann <das7pad@outlook.com>
2020-02-07 12:36:46 +01:00
Jakob Ackermann
10d177f41b
[misc] test/unit: stub the correct redis settings
2020-02-07 12:24:57 +01:00
Brian Gough
7036803acf
add missing argument to metrics.inc
...
also track retries rather than attempts (which is always 1 for a successful request)
2020-01-14 15:00:21 +00:00
Brian Gough
3caa0e7c05
add failure/retry metrics for web-api requests
2020-01-14 13:53:50 +00:00
Brian Gough
d0c5eb5698
support migration of project history keys to separate redis instance
2019-12-13 16:38:41 +00:00
Brian Gough
ad19fee667
add setting so that double flush is the default
...
can be disabled to stop flushing to track-changes
2019-11-25 13:36:25 +00:00
Brian Gough
4f6583bbf2
fix getDocVersion and add tests
2019-11-25 13:28:36 +00:00
Brian Gough
8b73bb9f13
Merge branch 'master' into bg-filter-track-changes-updates
2019-11-22 10:41:33 +00:00
Brian Gough
b7055eecee
add metrics for history flushes
2019-11-22 09:14:32 +00:00
Brian Gough
dcd7649bad
filter track-changes updates for projects using project-history
2019-11-19 10:02:56 +00:00
Brian Gough
f6b2ac7360
queue deletes for deferred processing
2019-09-25 16:42:49 +01:00
Brian Gough
912a3a7753
remove redis server-side hashing for performance
...
we still compute the document hash in node, and check it on retrieval
but we don't check the hash at the point of writing it in redis which
was previously done with a redis Lua script.
2019-09-09 15:27:58 +01:00
Brian Gough
e75f2cc325
add unit tests for skipping history flush
2019-08-15 11:01:45 +01:00
Brian Gough
97487a077e
fix cluster/sentinel connection with real-time
2019-07-10 09:42:05 +01:00
Henry Oswald
3b3b2da0f5
add pubsub redis connection and remove real time redis connection
2019-07-04 13:34:31 +01:00
Brian Gough
16fb297043
Revert "skip hash check when non-BMP characters replaced"
2019-06-27 11:39:45 +01:00
Brian Gough
f37860599d
skip hash check when non-BMP characters replaced
2019-06-25 16:36:10 +01:00
Brian Gough
984b2d38e6
add unit tests
2019-06-11 16:33:14 +01:00
Brian Gough
e95059f98e
handle non-urgent flushes in background
2019-06-03 10:01:10 +01:00
Brian Gough
0bbfa7de27
Merge branch 'master' into bg-downgrade-delete-component-error
2019-05-08 09:07:02 +01:00
Brian Gough
27a8248196
convert "Delete component" errors into warnings
2019-05-07 16:55:17 +01:00
Henry Oswald
e57741cb80
stub out project flusher for unit tests
2019-05-07 15:46:30 +01:00
Tim Alby
c1c23e4bee
record last author id on document flush
...
This is a multi-steps process:
* get a update's `user_id` from the metadata
* store the `user_id` (`lastUpdatedBy`) and current date (`lastUpdatedAt`) for
the document in Redis on every updates
* fetch `lastUpdatedAt` and `lastUpdatedBy` from Redis on document flush
* send the data to web to be persisted in Mongo
2019-05-02 11:10:02 +01:00
James Allen
52f3596e53
Review feedback
2019-04-16 11:05:17 +01:00
James Allen
3d76f4b9bf
Record a snapshot to mongo when a doc's comments/changes get collapsed
2019-04-11 13:27:46 +01:00
Brian Gough
fd1425d83f
include a unique id in every message published to redis
2019-03-21 12:10:15 +00:00
Brian Gough
8c5d74faef
use explicit json content-type to avoid security issues with text/html
2019-02-12 16:45:11 +00:00
Henry Oswald
ecaef6485b
revert the removal of realtime keyspace
2019-02-07 15:27:51 +00:00
Henry Oswald
4e1a2c787c
Revert "turn down logging, use logger.info for less important data"
...
This reverts commit c5f91428e3c7702fbbd3ffd1ef7a772d513f33f2.
2019-02-06 15:29:22 +00:00
Christopher Hoskin
1217d8a80a
Merge branch 'master' into csh-ho-docker-issue-1338-bulk-upgrade
2019-01-04 09:18:40 +00:00
Henry Oswald
2505be283a
turn down logging, use logger.info for less important data
2018-12-13 12:46:10 +00:00
Henry Oswald
25e1ad7491
remove redis.realtime
...
real time pushes data into the document updater redis instance
2018-12-05 16:17:41 +00:00
Brian Gough
aae1352519
ensure that project history is flushed when the project is deleted
2018-10-30 12:03:44 +00:00
Brian Gough
fdcb806518
set a timestamp for the first entry in the projectHistory:Ops queue
2018-07-24 09:17:31 +01:00
Brian Gough
e471730efb
Revert "Merge pull request #33 from sharelatex/bg-add-timestamp-marker-to-project-history-queue"
...
This reverts commit 8ec398b133aa75c45384dd7ceb630cd743f4a15c, reversing
changes made to e3a6c934cf53fd245d7d0df737463cc90f954885.
2018-07-23 16:05:04 +01:00
Brian Gough
6237577f87
set a timestamp for the first entry in the projectHistory:Ops queue
2018-07-20 10:43:31 +01:00
Hayden Faulds
62a10f25f2
move tk call to before sandboxed module call
2018-06-05 13:54:30 +01:00
Hayden Faulds
af92ca70a1
coerce projectHistoryId to integer after reading from Redis
2018-04-23 15:19:06 +01:00
Hayden Faulds
1b63141e49
Revert "Revert "Add projectHistoryId to updates""
2018-04-23 12:08:04 +01:00
Hayden Faulds
3cc20d6311
Revert "Add projectHistoryId to updates"
2018-04-23 11:41:29 +01:00
Hayden Faulds
25c0be4eaa
add projectHistryId to project structure updates
2018-04-13 15:22:05 +01:00
Hayden Faulds
a220794d32
add projectHistoryId to resync updates
2018-04-13 15:22:05 +01:00
Hayden Faulds
fb1852a593
cache projectHistoryId with doc in Redis
2018-04-13 15:22:05 +01:00
Brian Gough
3385d2640a
fix structure ordering bug
2018-03-15 16:27:10 +00:00
Brian Gough
b3887fd984
update unit tests for incoming project versions
2018-03-15 16:24:11 +00:00
Brian Gough
0642e3c8c9
support project version on incoming requests
2018-03-15 16:15:10 +00:00
Hayden Faulds
4a5731440a
Merge pull request #21 from sharelatex/hof-history-sync
...
resync project history endpoint
2018-03-14 10:40:59 +00:00
Hayden Faulds
1226f96fc3
extract ProjectHistoryRedisManager
2018-03-09 15:36:38 +00:00
Hayden Faulds
a72d228354
add unit tests
2018-03-07 13:29:53 +00:00
James Allen
939eaa2d4b
Don't allow a document to be loaded without a pathname
2018-03-02 10:02:49 +00:00
Hayden Faulds
5a11332aa3
remove unecessary method wrapping
2018-02-22 10:17:59 +00:00
Hayden Faulds
ea0dd9700b
reduce the number of times we flush project history
2018-02-22 10:01:05 +00:00
James Allen
3d5740fd7d
Merge pull request #18 from sharelatex/msw-flush-project-ops
...
Flush project structure changes every 500
2018-02-16 10:30:00 +00:00
James Allen
657da70d45
Fix unit test namespacing from mocha upgrade
2018-02-15 16:28:58 +00:00
James Allen
f6c79f3203
Fix and extend unit tests
2018-02-15 15:55:12 +00:00
Michael Walker
241d1b27d5
Remove _shouldFlushHistoryOps wrapper
2018-01-31 11:27:40 +00:00
Michael Walker
3028fb9c3d
Only flush project updates when crossing the threshold
2018-01-24 11:57:03 +00:00
Brian Gough
41f15c4fa8
fix unit test
2017-12-14 14:46:27 +00:00
Brian Gough
947e18eccf
Merge branch 'bg-limit-ops-in-lock'
2017-12-14 14:26:44 +00:00
James Allen
6abf9c227d
Calculate and send doc length before each update
2017-12-11 17:26:32 +00:00
Hayden Faulds
a3420b1236
version entity additions
2017-11-10 14:54:56 +00:00
Hayden Faulds
944e633bac
add acceptance test for entity renaming
2017-11-08 09:20:38 +00:00
Hayden Faulds
7e86afe55e
version file renames
2017-11-08 09:20:38 +00:00
Hayden Faulds
6d571e6d23
version document renames
2017-11-08 09:20:37 +00:00
Hayden Faulds
be41a1614d
decorate ops with pathname and doc_length
2017-10-23 15:32:25 +01:00
Hayden Faulds
962520fca8
flush track-changes and project-history in HistoryManager
2017-10-23 15:32:25 +01:00
Hayden Faulds
d003aef31c
conditionally enqueue history updates for project
2017-10-23 15:32:25 +01:00
Hayden Faulds
c27df0bfef
split apart multi and rclient in tests
2017-10-23 14:24:23 +01:00
Hayden Faulds
81f998afe9
return pathname from DocumentManager.getDoc
2017-10-23 14:23:21 +01:00
Hayden Faulds
748315aadc
handle pathname in DocumentManager.getDoc
2017-10-23 14:23:21 +01:00
Hayden Faulds
cbdace7386
store pathname in Redis cache
2017-10-23 14:23:21 +01:00
Hayden Faulds
05b93a629a
return pathname from PersistenceManager
2017-10-23 14:17:14 +01:00
Brian Gough
02d3d1bd17
fix unit tests
...
timekeeper must be called before SandboxedModule.require
2017-10-23 14:02:53 +01:00
Brian Gough
7c2a28c161
fix use of timekeeper in unit tests
2017-10-23 13:33:14 +01:00
Brian Gough
7cbb3e7af8
change getProjectDocs endpoint from GET to POST
...
also note that it flushes docs if they are considered old (i.e. not
recently flushed)
2017-10-11 15:30:29 +01:00
Brian Gough
8d3f82360c
update DocumentManager.getDoc signature
...
keep alreadyLoaded flag at the end for consistency
2017-10-11 14:25:33 +01:00
Brian Gough
2bbbf3c005
add unflushed time to doc in redis
2017-10-06 12:34:28 +01:00
Brian Gough
ad745f133d
Merge pull request #59 from sharelatex/bg-reset-project-state
...
add endpoint to clear project state
2017-09-13 10:39:47 +01:00
Brian Gough
2047bd5516
fix broken unit test
2017-09-11 09:44:05 +01:00
Brian Gough
f34c12fdc7
add endpoint to clear project state
2017-09-08 15:50:26 +01:00
Brian Gough
9d931a3fa7
fix broken unit tests
2017-08-18 12:08:39 +01:00
Brian Gough
230c93766d
fix broken RedisManager unit test
2017-08-18 11:59:31 +01:00
Brian Gough
f6be68eb08
added missing unit tests file
2017-08-11 16:49:16 +01:00
Brian Gough
f696ccb0d9
add unit tests for getProjectDocs
2017-08-10 16:41:28 +01:00
Brian Gough
ca7d9dce85
fix unit tests
2017-08-08 09:40:39 +01:00
Shane Kilkelly
c3465cf3d4
Reduce retry delay to 10ms, and refactor
2017-07-18 15:28:18 +01:00
Shane Kilkelly
0d46c3e2b3
WIP: auto-retry web requests
2017-07-17 10:08:21 +01:00
Brian Gough
d2730c8d6b
unit tests for locking timeouts
2017-07-12 10:48:32 +01:00
Brian Gough
963e513057
add a timeout for getPreviousDocOps
...
it uses several redis operations and this makes it prone to subsequent
timeouts if getDoc succeeds but is slow
2017-06-23 15:50:21 +01:00
James Allen
7d22e4422c
Merge pull request #49 from sharelatex/bg-add-redis-timeout
...
add redis timeout
2017-06-13 14:46:32 +01:00
James Allen
770b53f6af
Add missing mocks to speed up tests and clean up output
2017-06-08 16:43:27 +01:00
Brian Gough
1ef258c878
add a timeout on the redis getDoc request
2017-06-07 11:34:42 +01:00
Brian Gough
729216c9b1
add unit tests for DocsIn check
2017-05-31 16:08:45 +01:00
