The moderation system now uses a "pass/review" strategy instead of "pass/review/block", treating hard violations the same as review (only routing to manual review rather than auto-rejecting). Add fallback-approve behavior in the hook system so slow moderation APIs never silently block normal content when they time out or get cancelled - timed-out moderation now approves with "ReviewedBy: system" instead of leaving the zero-value result that would reject content. Update post/comment/vote services and handlers to use the new unified behavior. Simplify moderation prompts to emphasize "宁可放过，不可误伤" (prefer false positives over false negatives).
Add Session model, SessionService, and SessionRepository to track user login
sessions and enable token revocation on auth-critical events.
- Introduce explicit TokenType (access/refresh) in JWT claims to prevent
refresh token misuse via access token endpoints
- Add SessionID field to JWT claims, enabling stateless JWT validation
against revoked sessions
- Replace legacy Auth middleware with RequireAuth/OptionalAuth pipeline
that validates token type, account status, and session validity
- Implement session revocation on password change, reset, user ban/inactive,
and explicit logout
- Add Principal cache with active invalidation for banned/role-changed users
- Fix IDOR vulnerability: GetMessagesByCursor now validates currentUserID
is conversation participant via GetParticipantStrict
- Add group member visibility checks: announcements, group info, member list
now require group membership
- Simplify Casbin policy: remove g grouping, use r.sub == p.sub matcher
with globMatch; user_roles table is single source of truth for roles
- Add migration logic to clean legacy casbin g rules and migrate old p rules
from path-style to admin/<domain> resource naming
Add GET /api/v1/messages/:id endpoint to retrieve quoted/replied messages
for preview fill and navigation purposes. The endpoint includes authorization
checks ensuring only conversation participants can access the message.
Additionally improve file upload handling for native clients (expo-file-system)
by accepting explicit displayName from frontend, with sanitization to prevent
path traversal and 255-character length limit. File extension detection now
prioritizes the real display name over the multipart header filename.
Add client_request_id field to post creation endpoints (CreateVotePost and CreatePostWithSegments).
When provided, the server uses Redis SetNX-based idempotency to prevent duplicate posts
from double-clicks or network retries within a 24-hour TTL. Duplicate requests return
the originally created post; concurrent in-flight requests return 429 "正在发布中，请稍候".
Consolidate comment status changes through transitionStatus() to ensure
posts.comments_count and comments.replies_count are atomically maintained
within the same transaction, preventing inconsistencies between displayed
and stored counts.
- Remove separate ApplyPublishedStats() calls, now handled in UpdateModerationStatus
- Add cache invalidation for admin moderation operations
- Update report auto-hide to use unified status transition
Add DeviceCleanupWorker to periodically clean up stale device tokens that haven't
been used within the retention period. This prevents accumulation of invalid
registration_ids and maintains data hygiene.
- New DeviceCleanupConfig with enabled, retention_days (3), and interval_minutes (360)
- New repository methods DeleteStaleDevices and DeleteOldestActiveDevice
- Updated RegisterDevice to evict oldest active device when reaching limit
- Reduced MaxDevicesPerUser from 10 to 3 per user
- Worker integrated into app startup/shutdown lifecycle
- Add AdminDeviceTokenResponse DTO including push_token (RegistrationID) for push debugging
- Add DeviceTokenToAdminResponse and DeviceTokensToAdminResponse converters
- Inject PushService into AdminUserHandler for retrieving user devices
- Register GET /api/v1/admin/users/:id/devices route for querying target user's device tokens
- Add OPPO category/notify_level support (2024.11.20 regulation: category required when notify_level present)
- Add Honor importance field (NORMAL=service/LOW=marketing) for notification classification
- Add vivo category field (IM/ACCOUNT) for message scenario identification
- Set Xiaomi channel/template defaults in code (system=153609, chat=153608, templates P10761/M10289)
- Add classification option to JPush push payloads (0=operation, 1=system)
- Update xiaomi template keywords mapping to match actual template placeholders
- Add keyword search support for GetFollowers and GetFollowing endpoints
- Bind new config fields to environment variables for OPPO/Honor/vivo
Viper's AutomaticEnv does not bind nested keys when sections are commented out
in YAML. Add explicit BindEnv calls for APP_DATABASE_REPLICA_* variables so
they work when the replica section is not defined in config.yaml. Also bind
optional connection pool parameters (MAX_IDLE_CONNS, MAX_OPEN_CONNS).
Move the vendor switch logic into the ThirdPartyChannel struct as Set/IsEmpty/Normalize
methods to improve encapsulation and eliminate code duplication in the vendor params
builder. Also simplify messageToParams to reuse buildMessagePayload results and remove
unused notification_type extras from push notifications.
Add FileCleanupWorker for automatic expiration of chat files with configurable retention period and batch processing. Files uploaded via `/api/v1/uploads/files` are tracked in `uploaded_files` table with expiration timestamps, then deleted from S3 and database upon expiry. Expired file URLs are injected as `"expired": true` in message responses.
Extend JPush push notification configuration with vendor-specific channel_id support (xiaomi, huawei, oppo, vivo, meizu, honor, fcm) for differentiating system vs chat notifications. Add iOS APNs thread-id grouping configuration for notification categorization.
- Introduce interfaces for all major services (JWT, PostAI, Comment, Message, Notification, QRCodeLogin, Upload, Vote, etc.) to support dependency inversion.
- Move query parameters from `internal/dto` to a new `internal/query` package to separate request payloads from data transfer objects.
- Refactor `internal/router` to use embedded `RouterDeps` for cleaner dependency management.
- Decouple handlers from repositories by injecting services instead of direct repository access, ensuring proper layering.
- Improve database initialization by moving it from `internal/model` to `internal/database`.
- Optimize message decryption by implementing a more efficient `BatchDecrypt` method in `MessageEncryptor` using a worker pool.
- Enhance error handling and security by implementing fail-fast checks for encryption key length during startup.
- Clean up unused code, including the `avatar` package and several unused DTOs.
Introduces group calling capabilities, including group invites and participant
lifecycle management. Enhances the existing call system with support for
media types (voice/video) and improved participant tracking.
- Add `GroupInvite` and `ParticipantJoin/Leave` to `CallService`.
- Implement WebSocket handlers for `call_group_invite` and
`call_participant_join`.
- Update `CallSession` model to support group IDs, media types, and
tracking who ended the call.
- Integrate `GroupService` into `CallService` via dependency injection.
- Add error constants for group call limitations and participant capacity.
- Update LiveKit webhook handling to process participant leave events.
Refactor grade synchronization to remove semester-based filtering in favor
of a more flexible user-based approach and improve GPA summary updates.
Replace the manual WebRTC signaling implementation with LiveKit SFU. This includes:
- Adding LiveKit service, handler, and configuration.
- Updating Docker Compose to include LiveKit server, Redis, and PostgreSQL.
- Refactoring `CallService` and `WSHandler` to support LiveKit room readiness instead of raw SDP/ICE relaying.
- Adding new API endpoints for LiveKit token generation and webhooks.
- Removing deprecated WebRTC configuration and manual signaling DTOs.
Add support for fetching and synchronizing empty classroom data. This includes new protobuf definitions for task payloads and results, database models, repository, service, and HTTP handlers.
- Add `TASK_TYPE_GET_EMPTY_CLASSROOM` to `TaskType` enum
- Implement `EmptyClassroom` model and auto-migration
- Add `EmptyClassroomHandler` with `GET` and `POST /sync` routes
- Implement `EmptyClassroomSyncService` and `EmptyClassroomRepository`
- Update dependency injection with wire sets
Move the responsibility of sequence (`seq`) allocation from the application/cache layer to the database layer to ensure absolute consistency. Previously, sequences were pre-allocated via Redis/Lua, which introduced complexity in managing synchronization between the cache and the database.
Key changes:
- Implement `CreateMessageWithSeq` in `message_repo.go` using `SELECT FOR UPDATE` to lock the conversation row and atomically increment the `last_seq` within a single transaction.
- Simplify `SeqBufferManager` by removing the complex local buffering and Lua-based pre-allocation logic, reverting to a simpler model.
- Update `chat_service.go`, `group_service.go`, and `message_service.go` to remove manual sequence retrieval, relying instead on the repository's transactional allocation.
- Introduce `SyncConvSeq` in `conversation_cache.go` to perform "write-through" updates to Redis, ensuring the cache remains synchronized with the database's source of truth.
- Improve cold-start handling in `conversation_cache.go` with a new `syncSeqLua` script to prevent stale sequence reads.
Implement Lua scripts for atomic sequence incrementing and cold-start initialization in Redis to prevent race conditions. Improve the sequence buffer manager with CAS-like logic to prevent interval leakage during concurrent allocations. Update the message repository to use `SELECT ... FOR UPDATE` and conditional updates for `last_seq` to ensure database consistency during high concurrency.
- Add `nextSeqLua` and `initSeqLua` for atomic Redis operations
- Implement thread-safe buffer management in `seq_buffer.go`
- Add row-level locking in `message_repo.go`
- Update service layers to support group-aware sequence retrieval
Update the `Seq` field in `WSEventResponse` and its usages across message and websocket handlers to use `int64` instead of `string`. This aligns the data transfer object with the underlying message sequence type and removes unnecessary string conversions.
Introduce several performance and synchronization enhancements:
- Implement `SeqBufferManager` to allow sequence number pre-allocation via Redis Lua scripts, reducing atomic increment overhead.
- Add `PushWorker` to handle asynchronous message pushing using Redis Streams.
- Implement incremental conversation synchronization via `ConversationVersionLog` to allow clients to fetch only recent changes.
- Add support for Gzip compression in WebSocket communications to reduce bandwidth usage.
- Update dependency injection and configuration to support these new components.
Add support for database read replicas using the GORM dbresolver plugin. This allows for scaling read operations by distributing them across one or multiple replica nodes.
- Update `DatabaseConfig` to support single and multiple replica configurations.
- Add configuration options for replica connection pooling and selection policy.
- Integrate `dbresolver` in the database initialization process for PostgreSQL.
- Add helper methods to aggregate replica configurations.
Remove the fallback mechanism that queried the database when Redis keys were missing or unavailable. The `GetConvMaxSeq` and `GetConvMaxSeqBatch` methods now strictly rely on the Redis cache and return `ErrKeyNotFound` if the data is not present.
Improve the reliability of conversation sequence generation by implementing a fallback mechanism to the database when Redis is unavailable or when keys are missing.
- Add `GetNextSeq` to `MessageRepository` and implement it in `MessageRepositoryAdapter`.
- Update `ConversationCache` to fallback to DB in `GetConvMaxSeq`, `GetConvMaxSeqBatch`, and `GetNextSeq`.
- Ensure Redis TTL is refreshed after every `INCR` operation to prevent sequence counter expiration.
- Refactor service layers to handle sequence generation errors more explicitly.
- Add `IsEnabled` helper to `LayeredCache` for cleaner availability checks.
This commit performs a significant cleanup of the codebase by removing unused functions, methods, and entire files across various internal modules. This reduces technical debt and simplifies the project structure.
Key changes include:
- **cache**: Removed unused cache key generators and metrics snapshots.
- **dto**: Removed redundant converter functions and segment creation helpers.
- **middleware**: Deleted the unused `logger.go` middleware and simplified `ratelimit.go` and `casbin.go`.
- **model**: Removed unused ID helpers, database closing functions, and batch decryption logic.
- **pkg**: Cleaned up unused utility functions in `circuitbreaker`, `crypto`, `cursor`, `hook`, and `utils`.
- **service**: Deleted `account_cleanup_service.go` and removed unused helper functions in `log_cleanup_service.go` and `sensitive_service.go`.
- **repository**: Removed unused private loading methods in `comment_repo.go`.
Update the chat message push mechanism to include group ID and group avatar
when sending messages in group conversations. This ensures that push
notifications for group chats display the group's identity and icon
rather than the sender's personal avatar.
- Update `PushChatMessage` interface and implementation to accept `groupID` and `groupAvatar`.
- Modify `chat_service.go` to extract group metadata and pass it to the push service.
- Update `push_service.go` to include group details in the push payload extras and set the notification icon to the group avatar for group conversations.
Clean up the codebase by removing unused and deprecated unread count
functionality in `ConversationCache` and associated Redis key
generators. This follows the transition to sequence-based unread
calculation.
- Remove `IncrementUnread`, `ClearUnread`, `GetUnreadCountFromHash`,
`GetAllUnreadCounts`, and `GetTotalUnread` from `ConversationCache`.
- Remove `PrefixUnreadHash` and `PrefixUnreadTotal` constants and
`UnreadHashKey`/`UnreadTotalKey` helper functions.
- Remove unused `isValidContentType` helper in `MessageHandler`.
Refactor unread count logic to use arithmetic calculation (maxSeq - readSeq)
instead of manual incrementing/decrementing. This simplifies the cache
management and improves consistency.
New features:
- Batch mark-as-read functionality for multiple conversations.
- Lightweight sync data endpoint to retrieve conversation metadata
(maxSeq and last message timestamp) for client synchronization.
- Optimized batch retrieval of conversation max sequences using Redis MGet.
Technical changes:
- Deprecated `IncrementUnread` and `ClearUnread` in `ConversationCache`.
- Added `GetConvMaxSeqBatch` to reduce network round-trips.
- Added `HandleMarkReadAll` and `HandleGetSyncData` handlers.
- Updated `ChatService` to support batch operations and sync data retrieval.
Add new functionality for managing and synchronizing academic grades and exam schedules. This includes new models, repositories, services, and HTTP handlers, along with updated gRPC definitions and dependency injection configuration.
Additionally, improve the reliability of unread message count tracking by implementing an atomic Lua script for Redis operations in the conversation cache.
Introduce a new mechanism for tracking read positions using message
sequences (hasReadSeq), similar to OpenIM, to allow for O(1) unread
count calculations.
- Implement `UserReadSeq` caching in Redis with Lua scripts to prevent
sequence regression (ensuring updates only occur if the new sequence
is greater than the current one).
- Update `ConversationCache` to support sequence-based unread count
computation (`maxSeq - hasReadSeq`).
- Refactor `MessageRepository` to use conditional updates for
`last_read_seq` in the database.
- Update `ChatService` to automatically update the sender's read
sequence when sending a message.
- Optimize `MarkAsRead` logic to update both database and Redis
sequence caches and refine notification targets for private vs
group chats.
- Update `jpush` client to use pointer types for boolean fields to
properly handle optional values in JSON.
- fix unread count calculation logic in message repository
- implement logic to retrieve other participant's last read sequence in conversation handlers
- mask real names in verification record responses for privacy
- improve error reporting in main entry point by using stderr instead of zap logger during initialization
Introduce a distributed task execution system for gRPC runners. This includes:
- A `TaskBus` for cluster-mode task dispatching via Redis Pub/Sub.
- A `RunnerRegistry` to track active runner instances in Redis.
- Support for both standalone and cluster modes via configuration.
- Refactored `RunnerHub` and `TaskManager` to use a `TaskDispatcher` interface, decoupling task submission from local execution.
- Added distributed locking to `HotRankWorker` to ensure only one instance performs periodic hot rank recalculations in a multi-instance deployment.
- Updated dependency injection (Wire) to support the new runner components and registry.
Introduce a new WebSocket messaging architecture that supports both standalone and cluster modes. This allows for horizontal scaling of WebSocket servers by using Redis Pub/Sub to synchronize messages across multiple instances.
Key changes:
- Added `ws.MessagePublisher` interface to abstract message distribution.
- Implemented `ws.Bus` to handle cluster-mode messaging via Redis.
- Added `ws.OnlineTracker` to manage user online status across the cluster.
- Refactored multiple services (Chat, Group, Push, Call, etc.) to use the new `MessagePublisher` instead of a concrete `ws.Hub`.
- Added WebSocket configuration options (mode, instance ID, channel, TTL, heartbeat) to `config.yaml` and `config.go`.
- Updated dependency injection with Wire to support the new publisher and Redis client.
- Improved logging by replacing standard `log` with `zap` in several service components.
Allow image segments with URLs matching the local S3 prefix to bypass
SSRF validation, as internal S3 endpoints may resolve to private IP
addresses that would otherwise trigger security errors.
GetLastMessagesBatch was returning empty segments because it skipped
the decryption step that all other message query methods perform.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit introduces several significant architectural improvements to enhance system performance, scalability, and reliability:
- **Performance Optimization (N+1 Problem Resolution)**: Implemented batching mechanisms across `MessageHandler`, `ChatService`, `GroupService`, `MessageService`, and `UserService`. This replaces multiple individual database/cache queries with single batch operations for fetching unread counts, last messages, participants, and user information.
- **Enhanced Unread Count Management**: Migrated unread count tracking to a Redis Hash-based approach (`unread#️⃣{userID}`). This allows for atomic increments/decrements and efficient retrieval of both individual conversation unread counts and total unread counts for a user.
- **Improved Message Sequencing**: Integrated Redis-based sequence (`seq`) pre-allocation for messages to ensure strict ordering and reduce database contention during high-concurrency message creation.
- **Push Service Reliability**: Refactored the push notification system to include persistent `PushRecord` tracking. Added a recovery mechanism (`recoverPendingPushes`) to reload pending notifications from the database upon service startup, ensuring better delivery guarantees.
- **WebSocket Reliability**: Updated the WebSocket hub to move away from in-memory history replay in favor of a client-driven synchronization model (`sync_required` event and `ack` handling), reducing memory overhead and improving connection stability.
- **Cache Layer Enhancements**: Added `HIncrBy` and `IncrBySeq` to the `Cache` interface and its implementations (`RedisCache`, `LayeredCache`) to support the new unread and sequence management logic.
This commit introduces several architectural improvements and optimizations across the codebase:
- **Performance & Reliability**:
- Implemented Redis pipelining in `ConversationCache.CacheMessage` to reduce network round-trips.
- Added a circuit breaker to the JPush client to prevent cascading failures.
- Introduced batch deletion and batch member addition capabilities in repositories.
- Added message idempotency support using `client_msg_id` and a Redis-based cache.
- Optimized WebSocket handling with connection limits (total and per-user) and improved error logging.
- **Code Refactoring**:
- Refactored `Router` to use a `RouterDeps` struct, simplifying the constructor and improving maintainability.
- Unified model ID generation logic using new `id_helper.go` (supporting UUID and Snowflake).
- Standardized JSON serialization/deserialization in models using `json_helper.go`.
- Refactored DTO conversion logic, specifically for `UserResponse` (using functional options) and `Report` responses.
- Removed redundant/deprecated DTOs like `PostDetailResponse` and `TradeItemDetailResponse`.
- **Cache Improvements**:
- Enhanced `LayeredCache` with `SetRaw` to avoid double-encoding when promoting values from Redis to local cache.
- Added `DeleteBatch` support to the cache interface.
- **Other Changes**:
- Cleaned up `config.go` by removing redundant default values and explicit environment variable overrides.
- Improved WebSocket registration flow to handle connection limits gracefully.
Add comprehensive security improvements across the application:
- **IP-based login protection**: Implement IP ban system tracking login failures, auto-banning after threshold exceeded
- **Ownership verification**: Add userID parameter to Delete/Update operations for posts and comments to prevent unauthorized modifications
- **SSRF protection**: Add URL and resolved host validation for image URLs in chat and OpenAI client
- **SQL injection prevention**: Add EscapeLikeWildcard utility to escape special characters in LIKE queries
- **HTTP security**: Configure server timeouts and add security headers middleware
- **Rate limiting**: Refactor to support configurable duration and per-endpoint rate limits for auth routes
- **Error handling**: Standardize error responses using HandleError and proper error types
- Remove BOM from all Go source files
- Standardize import ordering and whitespace across handlers and services
- Replace PostgreSQL full-text search with ILIKE pattern matching in post and user repositories
- Enhance JPush client with TLS transport, rate limit monitoring, and simplified API
- Refactor PushService to include userID in device management methods
- Add MaxDevicesPerUser limit and extract helper functions for push payload construction
The JPush API returns sendno as a string value in the JSON response, but the
struct field was incorrectly typed as int64. This fix updates both the struct
field type and the corresponding zap logger call to use string type, ensuring
proper JSON parsing and logging.
Add zap logger calls to Track, PushByRegistrationIDs, PushByAliases, PushAll,
and GetDeviceInfo methods for improved observability. Logs include request
debug info, response parsing status, success confirmations, and error
tracking with relevant context like message IDs, counts, and registration IDs.
Add JPush (极光推送) integration to enable push notifications for offline users. This includes:
- New jpush client package with API for batch push notifications
- Configuration for jpush (enabled, app_key, master_secret, production mode)
- Updated push service to send messages via JPush when users are offline
- Added PushChatMessage method with do-not-disturb checking
- Integrated push service with chat service to notify offline users of new messages
- Updated deployment workflow with JPush and related environment variables
Add VerificationStatusNone constant and update converter to return
general identity with none status instead of empty strings when user
has not applied for verification.
Add Identity and VerificationStatus fields to UserResponse, UserSelfResponse,
and UserDetailResponse. Only expose identity information when verification
status is approved to protect privacy of pending/rejected applicants.
Add complete trade functionality including:
- TradeItem, TradeImage, TradeFavorite models with auto-migration
- TradeRepository with CRUD and favorite operations
- TradeService with business logic for listing, creating, updating, status management
- TradeHandler with RESTful endpoints for trade operations
- DTOs and converters for request/response handling
Routes include: list, get by id, create, update, delete, status update, view recording, favorite/unfavorite.
Move the scoped 'scored' struct definition to package level and rename
to hotRankScored to enable reuse across Refresh and refreshChannelRanks
methods, eliminating duplicate type definitions.
Add channel-specific hot post filtering and ranking across the posts
listing flow. Introduce refreshChannelRanks to compute per-channel hot
scores, new Redis cache keys for channel hot ranks, and update
repository/service/handler layers to accept channelID filtering
parameters. Update HotRankWorker dependency injection to include
ChannelRepository.
