Commit graph

127 commits

Author SHA1 Message Date
Henry Oswald
46b389e8b3 fix off by 1 error in key sharding 2021-02-09 10:48:40 +00:00
Henry Oswald
31e1808dd8 shard the pending-updates-list queue 2021-02-08 16:02:41 +00:00
Henry Oswald
c308e60b6b speed up DocumentUpdaterControllerTests by directly injecting redis into
tests
2021-02-08 11:13:22 +00:00
Jakob Ackermann
746c5aeb80 [misc] bump metrics module to 3.4.1
- renamed package from `metrics-sharelatex` to `@overleaf/metrics`
- drop support for statsd backend
- decaffeinate
- compress `/metrics` response using gzip
- bump debugging agents to latest versions
- expose prometheus interfaces for custom metrics (custom tags)
- cleanup of open sockets metrics
- fix deprecation warnings for header access
2020-11-25 11:57:22 +00:00
Jakob Ackermann
15af5c7977 [misc] bump @overleaf/redis-wrapper to version 2.0.0 2020-11-11 16:24:22 +00:00
Jakob Ackermann
c6f2a3b387 Merge pull request #189 from overleaf/jpa-relevel-log
[misc] re-level expected error log messages
2020-09-02 10:46:08 +02:00
Jakob Ackermann
4960bdd6fe [misc] re-level log: 404 from web -> WARN and emit 'project not found'
A stale browser tab tried to join a deleted project.
Emitting 'project not found'/'ProjectNotFound' will trigger a page
 reload in the frontend, upon web can render a 404.
See frontend: ConnectionManager.joinProject callback
2020-08-27 11:51:57 +01:00
Jakob Ackermann
55d938ba14 [misc] add tests for web replying with a 404 for joinProject 2020-08-27 11:51:56 +01:00
Jakob Ackermann
884b340c75 [misc] re-level log: 403 from web goes to WARN and emit 'not authorized'
Users will get redirected to the login page and will see a 'restricted'
 page after logging in again.
See frontend: ConnectionManager.reportConnectionError
2020-08-27 11:51:56 +01:00
Jakob Ackermann
d2a2b9d46e [misc] add tests for web replying with a 403 for joinProject 2020-08-27 11:51:56 +01:00
Jakob Ackermann
2ce7b36c95 [misc] drop duplicate log line for unauthorized applyOtUpdate calls
The violation is logged in Router._handleError.
2020-08-27 10:22:31 +01:00
Jakob Ackermann
425052ff91 Merge pull request #187 from overleaf/jpa-o-error-tagging
[misc] migrate to OError tagging/wrapping
2020-08-25 11:46:28 +02:00
Jakob Ackermann
ee3d3b09ed [misc] wrap redis errors as tagging does not work with them
ioredis may reuse the error instance for multiple callbacks. E.g. when
 the connection to redis fails, the queue is flushed with the same
 MaxRetriesPerRequestError instance.
2020-08-24 10:12:20 +01:00
Jakob Ackermann
537e97be73 [misc] OError.tag all the errors in async contexts
See the docs of OError.tag:
https://github.com/overleaf/o-error#long-stack-traces-with-oerrortag
(currently at 221dd902e7bfa0ee92de1ea5a3cbf3152c3ceeb4)

I am tagging all errors at each async hop. Most of the controller code
 will only ever see already tagged errors -- or new errors created in
 our app code. They should have enough info that we do not need to tag
 them again.
2020-08-24 10:12:06 +01:00
Jakob Ackermann
4cb8cc4a85 [DocumentUpdaterManager] use a new ClientRequestedMissingOpsError 2020-08-21 12:47:07 +01:00
Jakob Ackermann
8abfdb87ff [DocumentUpdaterManager] use a new DocumentUpdaterRequestFailedError 2020-08-21 12:47:07 +01:00
Jakob Ackermann
68bc9d0d23 [WebApiManager] use a new WebApiRequestFailedError 2020-08-21 12:47:06 +01:00
Jakob Ackermann
af50f9b02c [DocumentUpdaterManager] use a new UpdateTooLargeError 2020-08-21 12:47:05 +01:00
Jakob Ackermann
5950b26a42 [SafeJsonParse] migrate to OError and use a new DataTooLargeToParseError 2020-08-21 12:47:05 +01:00
Jakob Ackermann
f82177a46a [Errors] migrate to OError 2020-08-21 12:47:04 +01:00
Brian Gough
831d794bf4 clean up join/leave handling
Co-Authored-By: Jakob Ackermann <jakob.ackermann@overleaf.com>
2020-08-12 10:54:22 +01:00
Jakob Ackermann
8601084a2e [misc] WebsocketControllerTests: remove debugging code 2020-07-01 11:36:36 +01:00
Jakob Ackermann
a77222470b [misc] WebsocketControllerTests: always pass cb to updateClientPosition
...and fix the inconsistent async behaviour for logged-in vs anonymous
 users in the app.
2020-06-29 18:11:33 +01:00
Jakob Ackermann
4d01c02946 [misc] WebsocketControllerTests: explicitly pass a cb to leaveProject
...and fix the stubbing of `io.sockets.clients`.

We were running the assertions prior to the actual completion of the
 leaveProject. Which in turn shadowed the broken test setup --
 `io.sockets.clients()` was returning the `this.clientsInRoom=[]` from a
 previous context.
2020-06-29 17:38:22 +01:00
Jakob Ackermann
bea0e9beb7 [misc] RoomManagerTests: explicitly pass a cb to joinProject/joinDoc 2020-06-29 17:12:26 +01:00
decaffeinate
3eceb8a5f6 prettier: convert test/unit decaffeinated files to Prettier format 2020-06-23 18:30:16 +01:00
decaffeinate
68e2adebf5 decaffeinate: rename test/unit/coffee to test/unit/js 2020-06-23 18:30:06 +01:00
decaffeinate
93697a8c5c decaffeinate: Run post-processing cleanups on AuthorizationManagerTests.coffee and 13 other files 2020-06-23 18:30:03 +01:00
decaffeinate
2ca620e7a0 decaffeinate: Convert AuthorizationManagerTests.coffee and 13 other files to JS 2020-06-23 18:29:59 +01:00
decaffeinate
e5d07bd3af decaffeinate: Rename AuthorizationManagerTests.coffee and 13 other files from .coffee to .js 2020-06-23 18:29:53 +01:00
Jakob Ackermann
5282f8f531 [misc] synchronous client store using an Object at .ol_context 2020-06-22 17:04:08 +01:00
Jakob Ackermann
5f7841526f [misc] RoomManager: emitOnCompletion: properly handle Promise rejections
```
result = Promise.all([<Promise that rejects eventually>]) # rejection 1
result.then () -> RoomEvents.emit(eventName)              # rejection 2
result.catch (err) -> RoomEvents.emit(eventName, err)     # handle r1
```
As shown above, the second rejection remains unhandled. The fix is to
 chain the `.catch()` onto the `.then()` Promise.
2020-06-17 09:29:12 +01:00
Jakob Ackermann
dc553c4150 [misc] vendor a patched session.socket.io middleware 2020-06-09 15:21:33 +01:00
Jakob Ackermann
5238e6ad36 Merge pull request #145 from overleaf/jpa-backport-140
[backport] 140: stop processing requests as we detect a client disconnect
2020-06-09 14:41:12 +02:00
Jakob Ackermann
589bedc3dd Merge pull request #146 from overleaf/jpa-downgrade-logging
[misc] downgrade logging when running tests
2020-06-09 14:40:59 +02:00
Jakob Ackermann
f40241a037 [misc] downgrade logging when running tests 2020-06-05 11:38:09 +01:00
Jakob Ackermann
0b2cccf1e0 [misc] apply review feedback: adjust metric names
Co-Authored-By: Brian Gough <brian.gough@overleaf.com>

(cherry-picked from commit 67674b83efb452ece05cdc39525ee3a5eeb8a4d7)
2020-06-05 11:33:00 +01:00
Jakob Ackermann
7fa9061015 [misc] stop processing requests as we detect a client disconnect
v2 expose `client.connected`; v0 exposes `client.disconnected`

(cherry-picked from commit a9d70484343ca9be367d45bf7bb949e4be449647)
2020-06-05 11:32:13 +01:00
Jakob Ackermann
c6d08647c7 [misc] socket.io: use a secondary publicId for public facing usages 2020-06-04 17:18:07 +01:00
Jakob Ackermann
d13acb8ca3 [ChannelManager] port related and still mostly valid test from v2
I skipped the bulk of verifyConsistentBehaviour tests which are not
 valid for the new implementation -- there is no optimization and
 always cleanup.
2020-05-15 18:39:33 +02:00
Jakob Ackermann
1095851dfe [misc] test/unit: fix typos and assertion of error messages
Sinon does not check the contents of the passed error when checked via
 sinon.stub().calledWith.
```
callback = sinon.stub()
callback(new Error("some message"))
  .calledWith(new Error("completely different message"))
  === true
```

Cherry-pick plus an additional patch for the joinProject bail-out.
(cherry picked from commit d9570fee70701a5f431c39fdbec5f8bc5a7843fe)
2020-05-15 14:46:05 +02:00
Jakob Ackermann
b713beb7f0 Merge pull request #135 from overleaf/jpa-skip-leave-project-for-invalid-clients
[WebsocketController] skip leaveProject when joinProject didn't complete
2020-05-13 15:34:41 +02:00
Jakob Ackermann
55af5e502f [WebsocketController] skip leaveProject when joinProject didn't complete
Also drop dead code:
 - user_id bailout

   There is a check on a completed joinProject call now. It will always
    set a user_id, see Router.coffee which has a fallback `{_id:"..."}`.

 - late project_id bailout

   WebsocketLoadBalancer.emitToRoom will not work without a project_id.
   We have to bail out before the call.
2020-05-12 17:15:08 +02:00
Jakob Ackermann
684cb3903c [WebsocketController] handle redis subscribe error on joinProject
joinProject should not complete when the redis pub/sub subscribe request
 failed.
2020-05-12 13:03:50 +02:00
Brian Gough
5c28da1031 add metric for pendingUpdates queue 2020-04-07 11:53:54 +01:00
Jakob Ackermann
56628a16c6 [misc] track redis pub/sub payload sizes on publish 2020-03-30 11:31:44 +02:00
Jakob Ackermann
af53d3b603 [misc] skip duplicate JSON serialization for size check 2020-03-24 11:22:28 +01:00
Jakob Ackermann
cb675d38c2 [misc] SafeJsonParse: align the size limit with the frontend->rt limit
frontend -> real-time and doc-updater -> real-time should be in sync.
Otherwise we can send a payload to doc-updater, but can not receive the
 confirmation of it -- and the client will send it again in a loop.

Also log the size of the payload.
2020-03-24 09:14:15 +01:00
Jakob Ackermann
15244a54be [misc] WebsocketController: limit the update size to 7mb
bail out early on -- especially do not push the update into redis for
 doc-updater to discard it.

Confirm the update silently, otherwise the frontend will send it again.
Broadcast a 'otUpdateError' message and disconnect the client, like
 doc-updater would do.
2020-03-24 09:12:12 +01:00
Shane Kilkelly
6df88ebc49 Filter "comments" if restricted user. 2019-10-30 13:54:40 +00:00