public inbox for [email protected]help / color / mirror / Atom feed
[PATCH v2] Fix NaN handling of some geometric operators and functions 66+ messages / 9 participants [nested] [flat]
* [PATCH v2] Fix NaN handling of some geometric operators and functions @ 2020-08-27 05:49 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 66+ messages in thread From: Kyotaro Horiguchi @ 2020-08-27 05:49 UTC (permalink / raw) Some geometric operators shows somewhat odd behavior comes from NaN handling and that leads to inconsistency between index scan and seq scan on geometric conditions. For example: point '(NaN,NaN)' <-> polygon '((0,0),(0,1),(1,1))' => 0, not NaN point '(0.3,0.5)' <-> polygon '((0,0),(0,NaN),(1,1))' => 1.14, not NaN point '(NaN,NaN)' <@ polygon '((0,0),(0,1),(1,1))' => true, not false Some other functions returned totally wrong result like this: point '(1e+300,Infinity)' ## box '(2,2),(0,0)' => '(0,2)' Fix NaN and Infinity handling of geo_ops.c so that these generates saner results. However, the behavioral inconsistency that comes from Infinity cannot be eliminated with a moderate amount of additional code against the benefit so they are left alone and added documentation instead. --- doc/src/sgml/func.sgml | 16 + src/backend/utils/adt/geo_ops.c | 215 ++++++++-- src/include/utils/float.h | 8 +- src/test/regress/expected/create_index.out | 24 +- src/test/regress/expected/geometry.out | 458 ++++++++++++++------- src/test/regress/expected/point.out | 138 +++++-- src/test/regress/sql/point.sql | 2 + 7 files changed, 633 insertions(+), 228 deletions(-) diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index e2e618791e..d8638c06b8 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -10924,6 +10924,22 @@ CREATE TYPE rainbow AS ENUM ('red', 'orange', 'yellow', 'green', 'blue', 'purple on), where available for these types, likewise compare areas. </para> </caution> + <caution> + <para> + NaN and Infinity make geometric functions and operators behave + inconsistently. Geometric operators or functions that return a boolean + return false for operands that contain NaNs. Number-returning functions + and operators return the NaN in most cases but sometimes return a valid + value if no NaNs are met while calculation. Object-returning ones + yield an object that contain NaNs depending to the operation. Likewise + the objects containing Infinity can make geometric operators and + functions behave inconsistently. For example (point + '(Infinity,Infinity)' <-> line '{-1,0,5}') is Infinity but (point + '(Infinity,Infinity)' <-> line '{0,-1,5}') is NaN. It can never + be a value other than these, but you should consider it uncertain how + geometric operators behave for objects containing Infinity. + </para> + </caution> <note> <para> diff --git a/src/backend/utils/adt/geo_ops.c b/src/backend/utils/adt/geo_ops.c index a7db783958..06deeb6d12 100644 --- a/src/backend/utils/adt/geo_ops.c +++ b/src/backend/utils/adt/geo_ops.c @@ -904,9 +904,19 @@ box_intersect(PG_FUNCTION_ARGS) result = (BOX *) palloc(sizeof(BOX)); - result->high.x = float8_min(box1->high.x, box2->high.x); + /* float8_min conceals NaN, check separately for NaNs */ + if (unlikely(isnan(box1->high.x) || isnan(box2->high.x))) + result->high.x = get_float8_nan(); + else + result->high.x = float8_min(box1->high.x, box2->high.x); + result->low.x = float8_max(box1->low.x, box2->low.x); - result->high.y = float8_min(box1->high.y, box2->high.y); + + if (unlikely(isnan(box1->high.y) || isnan(box2->high.y))) + result->high.x = get_float8_nan(); + else + result->high.y = float8_min(box1->high.y, box2->high.y); + result->low.y = float8_max(box1->low.y, box2->low.y); PG_RETURN_BOX_P(result); @@ -1061,6 +1071,21 @@ line_construct(LINE *result, Point *pt, float8 m) result->A = -1.0; result->B = 0.0; result->C = pt->x; + + /* Avoid creating a valid line from an invalid point */ + if (unlikely(isnan(pt->y))) + result->C = get_float8_nan(); + } + else if (m == 0.0) + { + /* use "mx - y + yinter = 0" */ + result->A = 0.0; + result->B = -1.0; + result->C = pt->y; + + /* Avoid creating a valid line from an invalid point */ + if (unlikely(isnan(pt->x))) + result->C = get_float8_nan(); } else { @@ -1084,6 +1109,7 @@ line_construct_pp(PG_FUNCTION_ARGS) Point *pt2 = PG_GETARG_POINT_P(1); LINE *result = (LINE *) palloc(sizeof(LINE)); + /* NaNs are considered to be equal by point_eq_point */ if (point_eq_point(pt1, pt2)) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), @@ -1104,8 +1130,12 @@ line_intersect(PG_FUNCTION_ARGS) { LINE *l1 = PG_GETARG_LINE_P(0); LINE *l2 = PG_GETARG_LINE_P(1); + Point xp; - PG_RETURN_BOOL(line_interpt_line(NULL, l1, l2)); + if (line_interpt_line(&xp, l1, l2) && !isnan(xp.x) && !isnan(xp.y)) + PG_RETURN_BOOL(true); + else + PG_RETURN_BOOL(false); } Datum @@ -1123,14 +1153,17 @@ line_perp(PG_FUNCTION_ARGS) LINE *l1 = PG_GETARG_LINE_P(0); LINE *l2 = PG_GETARG_LINE_P(1); + if (unlikely(isnan(l1->C) || isnan(l2->C))) + return false; + if (FPzero(l1->A)) - PG_RETURN_BOOL(FPzero(l2->B)); + PG_RETURN_BOOL(FPzero(l2->B) && !isnan(l1->B) && !isnan(l2->A)); if (FPzero(l2->A)) - PG_RETURN_BOOL(FPzero(l1->B)); + PG_RETURN_BOOL(FPzero(l1->B) && !isnan(l2->B) && !isnan(l1->A)); if (FPzero(l1->B)) - PG_RETURN_BOOL(FPzero(l2->A)); + PG_RETURN_BOOL(FPzero(l2->A) && !isnan(l1->A) && !isnan(l2->B)); if (FPzero(l2->B)) - PG_RETURN_BOOL(FPzero(l1->A)); + PG_RETURN_BOOL(FPzero(l1->A) && !isnan(l2->A) && !isnan(l1->B)); PG_RETURN_BOOL(FPeq(float8_div(float8_mul(l1->A, l2->A), float8_mul(l1->B, l2->B)), -1.0)); @@ -1141,7 +1174,7 @@ line_vertical(PG_FUNCTION_ARGS) { LINE *line = PG_GETARG_LINE_P(0); - PG_RETURN_BOOL(FPzero(line->B)); + PG_RETURN_BOOL(FPzero(line->B) && !isnan(line->A) && !isnan(line->C)); } Datum @@ -1149,7 +1182,7 @@ line_horizontal(PG_FUNCTION_ARGS) { LINE *line = PG_GETARG_LINE_P(0); - PG_RETURN_BOOL(FPzero(line->A)); + PG_RETURN_BOOL(FPzero(line->A) && !isnan(line->B) && !isnan(line->C)); } @@ -1195,9 +1228,19 @@ static inline float8 line_sl(LINE *line) { if (FPzero(line->A)) + { + /* C is likely to be NaN than B */ + if (unlikely(isnan(line->C) || isnan(line->B))) + return get_float8_nan(); return 0.0; + } if (FPzero(line->B)) + { + /* C is likely to be NaN than A */ + if (unlikely(isnan(line->C) || isnan(line->A))) + return get_float8_nan(); return DBL_MAX; + } return float8_div(line->A, -line->B); } @@ -1209,9 +1252,19 @@ static inline float8 line_invsl(LINE *line) { if (FPzero(line->A)) + { + /* C is likely to be NaN than B */ + if (unlikely(isnan(line->C) || isnan(line->B))) + return get_float8_nan(); return DBL_MAX; + } if (FPzero(line->B)) + { + /* C is likely to be NaN than A */ + if (unlikely(isnan(line->C) || isnan(line->A))) + return get_float8_nan(); return 0.0; + } return float8_div(line->B, line->A); } @@ -1224,14 +1277,23 @@ line_distance(PG_FUNCTION_ARGS) { LINE *l1 = PG_GETARG_LINE_P(0); LINE *l2 = PG_GETARG_LINE_P(1); + Point xp; float8 ratio; - if (line_interpt_line(NULL, l1, l2)) /* intersecting? */ + if (line_interpt_line(&xp, l1, l2)) /* intersecting? */ + { + /* return NaN if NaN was involved */ + if (isnan(xp.x) || isnan(xp.y)) + PG_RETURN_FLOAT8(get_float8_nan()); + PG_RETURN_FLOAT8(0.0); + } - if (!FPzero(l1->A) && !isnan(l1->A) && !FPzero(l2->A) && !isnan(l2->A)) + if (unlikely(isnan(l1->A) || isnan(l1->B) || isnan(l2->A) || isnan(l2->B))) + ratio = get_float8_nan(); + else if (!FPzero(l1->A) && !FPzero(l2->A)) ratio = float8_div(l1->A, l2->A); - else if (!FPzero(l1->B) && !isnan(l1->B) && !FPzero(l2->B) && !isnan(l2->B)) + else if (!FPzero(l1->B) && !FPzero(l2->B)) ratio = float8_div(l1->B, l2->B); else ratio = 1.0; @@ -1291,14 +1353,19 @@ line_interpt_line(Point *result, LINE *l1, LINE *l2) } else if (!FPzero(l2->B)) { - if (FPeq(l1->A, float8_mul(l2->A, float8_div(l1->B, l2->B)))) - return false; - - x = float8_div(float8_mi(float8_mul(l2->B, l1->C), - float8_mul(l1->B, l2->C)), - float8_mi(float8_mul(l2->A, l1->B), - float8_mul(l1->A, l2->B))); - y = float8_div(-float8_pl(float8_mul(l2->A, x), l2->C), l2->B); + /* + * We know that l1->B is zero, which means l1 is vertical. The lines + * cannot be parallel and the x-coord of the cross point is -C/A of l1. + */ + x = -float8_div(l1->C, l1->A); + /* + * When l2->A is zero, y is determined independently from x even if it + * is Inf. + */ + if (FPzero(l2->A)) + y = -float8_div(l2->C, l2->B); + else + y = float8_div(-float8_pl(float8_mul(l2->A, x), l2->C), l2->B); } else return false; @@ -1626,18 +1693,32 @@ path_inter(PG_FUNCTION_ARGS) b1.high.y = b1.low.y = p1->p[0].y; for (i = 1; i < p1->npts; i++) { + /* float8_min conceals NaN, check separately for NaNs */ b1.high.x = float8_max(p1->p[i].x, b1.high.x); b1.high.y = float8_max(p1->p[i].y, b1.high.y); - b1.low.x = float8_min(p1->p[i].x, b1.low.x); + if (unlikely(isnan(p1->p[i].x))) + b1.low.x = p1->p[i].x; + else + b1.low.x = float8_min(p1->p[i].x, b1.low.x); + if (unlikely(isnan(p1->p[i].y))) + b1.low.x = p1->p[i].y; + else b1.low.y = float8_min(p1->p[i].y, b1.low.y); } b2.high.x = b2.low.x = p2->p[0].x; b2.high.y = b2.low.y = p2->p[0].y; for (i = 1; i < p2->npts; i++) { + /* float8_min conceals NaN, check separately for NaNs */ b2.high.x = float8_max(p2->p[i].x, b2.high.x); b2.high.y = float8_max(p2->p[i].y, b2.high.y); - b2.low.x = float8_min(p2->p[i].x, b2.low.x); + if (unlikely(isnan(p2->p[i].x))) + b2.low.x = p2->p[i].x; + else + b2.low.x = float8_min(p2->p[i].x, b2.low.x); + if (unlikely(isnan(p2->p[i].y))) + b2.low.y = p1->p[i].y; + else b2.low.y = float8_min(p2->p[i].y, b2.low.y); } if (!box_ov(&b1, &b2)) @@ -1728,6 +1809,11 @@ path_distance(PG_FUNCTION_ARGS) statlseg_construct(&seg2, &p2->p[jprev], &p2->p[j]); tmp = lseg_closept_lseg(NULL, &seg1, &seg2); + + /* return NULL immediately if NaN is involved */ + if (isnan(tmp)) + PG_RETURN_NULL(); + if (!have_min || float8_lt(tmp, min)) { min = tmp; @@ -1980,9 +2066,17 @@ static inline float8 point_sl(Point *pt1, Point *pt2) { if (FPeq(pt1->x, pt2->x)) + { + if (unlikely(isnan(pt1->y) || isnan(pt2->y))) + return get_float8_nan(); return DBL_MAX; + } if (FPeq(pt1->y, pt2->y)) + { + if (unlikely(isnan(pt1->x) || isnan(pt2->x))) + return get_float8_nan(); return 0.0; + } return float8_div(float8_mi(pt1->y, pt2->y), float8_mi(pt1->x, pt2->x)); } @@ -1996,9 +2090,17 @@ static inline float8 point_invsl(Point *pt1, Point *pt2) { if (FPeq(pt1->x, pt2->x)) + { + if (unlikely(isnan(pt1->y) || isnan(pt2->y))) + return get_float8_nan(); return 0.0; + } if (FPeq(pt1->y, pt2->y)) + { + if (unlikely(isnan(pt1->x) || isnan(pt2->x))) + return get_float8_nan(); return DBL_MAX; + } return float8_div(float8_mi(pt1->x, pt2->x), float8_mi(pt2->y, pt1->y)); } @@ -2414,6 +2516,11 @@ dist_ppath_internal(Point *pt, PATH *path) statlseg_construct(&lseg, &path->p[iprev], &path->p[i]); tmp = lseg_closept_point(NULL, &lseg, pt); + + /* return NaN if NaN is involved */ + if (unlikely(isnan(tmp))) + return tmp; + if (!have_min || float8_lt(tmp, result)) { result = tmp; @@ -2645,6 +2752,8 @@ dist_ppoly_internal(Point *pt, POLYGON *poly) d = lseg_closept_point(NULL, &seg, pt); if (float8_lt(d, result)) result = d; + else if (unlikely(isnan(d))) + return get_float8_nan(); } return result; @@ -2674,7 +2783,8 @@ lseg_interpt_line(Point *result, LSEG *lseg, LINE *line) * intersection point, we are done. */ line_construct(&tmp, &lseg->p[0], lseg_sl(lseg)); - if (!line_interpt_line(&interpt, &tmp, line)) + if (!line_interpt_line(&interpt, &tmp, line) || + unlikely(isnan(interpt.x) || isnan(interpt.y))) return false; /* @@ -2803,6 +2913,7 @@ lseg_closept_lseg(Point *result, LSEG *on_lseg, LSEG *to_lseg) Point point; float8 dist, d; + bool isnan = false; /* First, we handle the case when the line segments are intersecting. */ if (lseg_interpt_lseg(result, on_lseg, to_lseg)) @@ -2814,6 +2925,7 @@ lseg_closept_lseg(Point *result, LSEG *on_lseg, LSEG *to_lseg) */ dist = lseg_closept_point(result, on_lseg, &to_lseg->p[0]); d = lseg_closept_point(&point, on_lseg, &to_lseg->p[1]); + isnan |= (isnan(dist) || isnan(d)); if (float8_lt(d, dist)) { dist = d; @@ -2823,6 +2935,7 @@ lseg_closept_lseg(Point *result, LSEG *on_lseg, LSEG *to_lseg) /* The closest point can still be one of the endpoints, so we test them. */ d = lseg_closept_point(NULL, to_lseg, &on_lseg->p[0]); + isnan |= isnan(d); if (float8_lt(d, dist)) { dist = d; @@ -2830,6 +2943,7 @@ lseg_closept_lseg(Point *result, LSEG *on_lseg, LSEG *to_lseg) *result = on_lseg->p[0]; } d = lseg_closept_point(NULL, to_lseg, &on_lseg->p[1]); + isnan |= isnan(d); if (float8_lt(d, dist)) { dist = d; @@ -2837,6 +2951,12 @@ lseg_closept_lseg(Point *result, LSEG *on_lseg, LSEG *to_lseg) *result = on_lseg->p[1]; } + if (unlikely(isnan)) + { + if (result != NULL) + result->x = result->y = get_float8_nan(); + return get_float8_nan(); + } return dist; } @@ -2873,6 +2993,7 @@ box_closept_point(Point *result, BOX *box, Point *pt) Point point, closept; LSEG lseg; + bool isnan = false; if (box_contain_point(box, pt)) { @@ -2887,9 +3008,10 @@ box_closept_point(Point *result, BOX *box, Point *pt) point.y = box->high.y; statlseg_construct(&lseg, &box->low, &point); dist = lseg_closept_point(result, &lseg, pt); - + isnan |= isnan(dist); statlseg_construct(&lseg, &box->high, &point); d = lseg_closept_point(&closept, &lseg, pt); + isnan |= isnan(d); if (float8_lt(d, dist)) { dist = d; @@ -2901,6 +3023,7 @@ box_closept_point(Point *result, BOX *box, Point *pt) point.y = box->low.y; statlseg_construct(&lseg, &box->low, &point); d = lseg_closept_point(&closept, &lseg, pt); + isnan |= isnan(d); if (float8_lt(d, dist)) { dist = d; @@ -2910,6 +3033,7 @@ box_closept_point(Point *result, BOX *box, Point *pt) statlseg_construct(&lseg, &box->high, &point); d = lseg_closept_point(&closept, &lseg, pt); + isnan |= isnan(d); if (float8_lt(d, dist)) { dist = d; @@ -2917,6 +3041,13 @@ box_closept_point(Point *result, BOX *box, Point *pt) *result = closept; } + if (unlikely(isnan)) + { + if (result != NULL) + result->x = result->y = get_float8_nan(); + return get_float8_nan(); + } + return dist; } @@ -2988,6 +3119,7 @@ close_sl(PG_FUNCTION_ARGS) * even because of simple roundoff issues, there may not be a single closest * point. We are likely to set the result to the second endpoint in these * cases. + * Returns Nan and stores {NaN,NaN} to result if NaN is involved, */ static float8 lseg_closept_line(Point *result, LSEG *lseg, LINE *line) @@ -3010,6 +3142,14 @@ lseg_closept_line(Point *result, LSEG *lseg, LINE *line) } else { + /* return NaN if any of the two is NaN */ + if (unlikely(isnan(dist1) || isnan(dist2))) + { + if (result != NULL) + result->x = result->y = get_float8_nan(); + return get_float8_nan(); + } + if (result != NULL) *result = lseg->p[1]; @@ -3436,6 +3576,12 @@ make_bound_box(POLYGON *poly) y2 = y1 = poly->p[0].y; for (i = 1; i < poly->npts; i++) { + /* if NaN found, make an invalid boundbox */ + if (unlikely(isnan(poly->p[i].x) || isnan(poly->p[i].y))) + { + x1 = x2 = y1 = y2 = get_float8_nan(); + break; + } if (float8_lt(poly->p[i].x, x1)) x1 = poly->p[i].x; if (float8_gt(poly->p[i].x, x2)) @@ -3911,6 +4057,11 @@ lseg_inside_poly(Point *a, Point *b, POLYGON *poly, int start) t.p[1] = *b; s.p[0] = poly->p[(start == 0) ? (poly->npts - 1) : (start - 1)]; + /* Fast path. Check against boundbox. Also checks NaNs. */ + if (!box_contain_point(&poly->boundbox, a) || + !box_contain_point(&poly->boundbox, b)) + return false; + for (i = start; i < poly->npts && res; i++) { Point interpt; @@ -5350,6 +5501,10 @@ point_inside(Point *p, int npts, Point *plist) x0 = float8_mi(plist[0].x, p->x); y0 = float8_mi(plist[0].y, p->y); + /* NaN makes the point cannot be inside the polygon */ + if (unlikely(isnan(x0) || isnan(y0) || isnan(p->x) || isnan(p->y))) + return 0; + prev_x = x0; prev_y = y0; /* loop over polygon points and aggregate total_cross */ @@ -5359,6 +5514,10 @@ point_inside(Point *p, int npts, Point *plist) x = float8_mi(plist[i].x, p->x); y = float8_mi(plist[i].y, p->y); + /* NaN makes the point cannot be inside the polygon */ + if (unlikely(isnan(x) || isnan(y))) + return 0; + /* compute previous to current point crossing */ if ((cross = lseg_crossing(x, y, prev_x, prev_y)) == POINT_ON_POLYGON) return 2; @@ -5517,12 +5676,12 @@ pg_hypot(float8 x, float8 y) result; /* Handle INF and NaN properly */ - if (isinf(x) || isinf(y)) + if (unlikely(isnan(x) || isnan(y))) + return get_float8_nan(); + + if (unlikely(isinf(x) || isinf(y))) return get_float8_infinity(); - if (isnan(x) || isnan(y)) - return get_float8_nan(); - /* Else, drop any minus signs */ x = fabs(x); y = fabs(y); diff --git a/src/include/utils/float.h b/src/include/utils/float.h index e2aae8ef17..79bf8daca8 100644 --- a/src/include/utils/float.h +++ b/src/include/utils/float.h @@ -225,9 +225,9 @@ float4_div(const float4 val1, const float4 val2) if (unlikely(val2 == 0.0f) && !isnan(val1)) float_zero_divide_error(); result = val1 / val2; - if (unlikely(isinf(result)) && !isinf(val1) && !isinf(val2)) + if (unlikely(isinf(result)) && !isinf(val1)) float_overflow_error(); - if (unlikely(result == 0.0f) && val1 != 0.0f) + if (unlikely(result == 0.0f) && val1 != 0.0f && !isinf(val2)) float_underflow_error(); return result; @@ -241,9 +241,9 @@ float8_div(const float8 val1, const float8 val2) if (unlikely(val2 == 0.0) && !isnan(val1)) float_zero_divide_error(); result = val1 / val2; - if (unlikely(isinf(result)) && !isinf(val1) && !isinf(val2)) + if (unlikely(isinf(result)) && !isinf(val1)) float_overflow_error(); - if (unlikely(result == 0.0) && val1 != 0.0) + if (unlikely(result == 0.0) && val1 != 0.0 && !isinf(val2)) float_underflow_error(); return result; diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out index 64c0c66859..dd521923dd 100644 --- a/src/test/regress/expected/create_index.out +++ b/src/test/regress/expected/create_index.out @@ -139,7 +139,7 @@ SELECT count(*) FROM point_tbl WHERE box '(0,0,100,100)' @> f1; SELECT count(*) FROM point_tbl WHERE f1 <@ polygon '(0,0),(0,100),(100,100),(50,50),(100,0),(0,0)'; count ------- - 5 + 4 (1 row) SELECT count(*) FROM point_tbl WHERE f1 <@ circle '<(50,50),50>'; @@ -157,7 +157,7 @@ SELECT count(*) FROM point_tbl p WHERE p.f1 << '(0.0, 0.0)'; SELECT count(*) FROM point_tbl p WHERE p.f1 >> '(0.0, 0.0)'; count ------- - 3 + 4 (1 row) SELECT count(*) FROM point_tbl p WHERE p.f1 <^ '(0.0, 0.0)'; @@ -169,7 +169,7 @@ SELECT count(*) FROM point_tbl p WHERE p.f1 <^ '(0.0, 0.0)'; SELECT count(*) FROM point_tbl p WHERE p.f1 >^ '(0.0, 0.0)'; count ------- - 4 + 5 (1 row) SELECT count(*) FROM point_tbl p WHERE p.f1 ~= '(-5, -12)'; @@ -188,10 +188,11 @@ SELECT * FROM point_tbl ORDER BY f1 <-> '0,1'; (10,10) (-5,-12) (5.1,34.5) + (Infinity,1e+300) (1e+300,Infinity) (NaN,NaN) -(10 rows) +(11 rows) SELECT * FROM point_tbl WHERE f1 IS NULL; f1 @@ -202,16 +203,17 @@ SELECT * FROM point_tbl WHERE f1 IS NULL; SELECT * FROM point_tbl WHERE f1 IS NOT NULL ORDER BY f1 <-> '0,1'; f1 ------------------- - (1e-300,-1e-300) (0,0) + (1e-300,-1e-300) (-3,4) (-10,0) (10,10) (-5,-12) (5.1,34.5) (1e+300,Infinity) + (Infinity,1e+300) (NaN,NaN) -(9 rows) +(10 rows) SELECT * FROM point_tbl WHERE f1 <@ '(-10,-10),(10,10)':: box ORDER BY f1 <-> '0,1'; f1 @@ -464,7 +466,7 @@ SELECT count(*) FROM point_tbl p WHERE p.f1 >> '(0.0, 0.0)'; SELECT count(*) FROM point_tbl p WHERE p.f1 >> '(0.0, 0.0)'; count ------- - 3 + 4 (1 row) EXPLAIN (COSTS OFF) @@ -494,7 +496,7 @@ SELECT count(*) FROM point_tbl p WHERE p.f1 >^ '(0.0, 0.0)'; SELECT count(*) FROM point_tbl p WHERE p.f1 >^ '(0.0, 0.0)'; count ------- - 4 + 5 (1 row) EXPLAIN (COSTS OFF) @@ -530,10 +532,11 @@ SELECT * FROM point_tbl ORDER BY f1 <-> '0,1'; (10,10) (-5,-12) (5.1,34.5) + (Infinity,1e+300) (1e+300,Infinity) (NaN,NaN) -(10 rows) +(11 rows) EXPLAIN (COSTS OFF) SELECT * FROM point_tbl WHERE f1 IS NULL; @@ -568,9 +571,10 @@ SELECT * FROM point_tbl WHERE f1 IS NOT NULL ORDER BY f1 <-> '0,1'; (10,10) (-5,-12) (5.1,34.5) + (Infinity,1e+300) (1e+300,Infinity) (NaN,NaN) -(9 rows) +(10 rows) EXPLAIN (COSTS OFF) SELECT * FROM point_tbl WHERE f1 <@ '(-10,-10),(10,10)':: box ORDER BY f1 <-> '0,1'; diff --git a/src/test/regress/expected/geometry.out b/src/test/regress/expected/geometry.out index 5b9d37030f..20acb0b0b4 100644 --- a/src/test/regress/expected/geometry.out +++ b/src/test/regress/expected/geometry.out @@ -120,6 +120,7 @@ SELECT p1.f1, p2.f1, slope(p1.f1, p2.f1) FROM POINT_TBL p1, POINT_TBL p2; (0,0) | (-5,-12) | 2.4 (0,0) | (1e-300,-1e-300) | 1.79769313486e+308 (0,0) | (1e+300,Infinity) | Infinity + (0,0) | (Infinity,1e+300) | 0 (0,0) | (NaN,NaN) | NaN (0,0) | (10,10) | 1 (-10,0) | (0,0) | 0 @@ -129,6 +130,7 @@ SELECT p1.f1, p2.f1, slope(p1.f1, p2.f1) FROM POINT_TBL p1, POINT_TBL p2; (-10,0) | (-5,-12) | -2.4 (-10,0) | (1e-300,-1e-300) | 0 (-10,0) | (1e+300,Infinity) | Infinity + (-10,0) | (Infinity,1e+300) | 0 (-10,0) | (NaN,NaN) | NaN (-10,0) | (10,10) | 0.5 (-3,4) | (0,0) | -1.33333333333 @@ -138,6 +140,7 @@ SELECT p1.f1, p2.f1, slope(p1.f1, p2.f1) FROM POINT_TBL p1, POINT_TBL p2; (-3,4) | (-5,-12) | 8 (-3,4) | (1e-300,-1e-300) | -1.33333333333 (-3,4) | (1e+300,Infinity) | Infinity + (-3,4) | (Infinity,1e+300) | 0 (-3,4) | (NaN,NaN) | NaN (-3,4) | (10,10) | 0.461538461538 (5.1,34.5) | (0,0) | 6.76470588235 @@ -147,6 +150,7 @@ SELECT p1.f1, p2.f1, slope(p1.f1, p2.f1) FROM POINT_TBL p1, POINT_TBL p2; (5.1,34.5) | (-5,-12) | 4.60396039604 (5.1,34.5) | (1e-300,-1e-300) | 6.76470588235 (5.1,34.5) | (1e+300,Infinity) | Infinity + (5.1,34.5) | (Infinity,1e+300) | 0 (5.1,34.5) | (NaN,NaN) | NaN (5.1,34.5) | (10,10) | -5 (-5,-12) | (0,0) | 2.4 @@ -156,6 +160,7 @@ SELECT p1.f1, p2.f1, slope(p1.f1, p2.f1) FROM POINT_TBL p1, POINT_TBL p2; (-5,-12) | (-5,-12) | 1.79769313486e+308 (-5,-12) | (1e-300,-1e-300) | 2.4 (-5,-12) | (1e+300,Infinity) | Infinity + (-5,-12) | (Infinity,1e+300) | 0 (-5,-12) | (NaN,NaN) | NaN (-5,-12) | (10,10) | 1.46666666667 (1e-300,-1e-300) | (0,0) | 1.79769313486e+308 @@ -165,6 +170,7 @@ SELECT p1.f1, p2.f1, slope(p1.f1, p2.f1) FROM POINT_TBL p1, POINT_TBL p2; (1e-300,-1e-300) | (-5,-12) | 2.4 (1e-300,-1e-300) | (1e-300,-1e-300) | 1.79769313486e+308 (1e-300,-1e-300) | (1e+300,Infinity) | Infinity + (1e-300,-1e-300) | (Infinity,1e+300) | 0 (1e-300,-1e-300) | (NaN,NaN) | NaN (1e-300,-1e-300) | (10,10) | 1 (1e+300,Infinity) | (0,0) | Infinity @@ -174,8 +180,19 @@ SELECT p1.f1, p2.f1, slope(p1.f1, p2.f1) FROM POINT_TBL p1, POINT_TBL p2; (1e+300,Infinity) | (-5,-12) | Infinity (1e+300,Infinity) | (1e-300,-1e-300) | Infinity (1e+300,Infinity) | (1e+300,Infinity) | 1.79769313486e+308 + (1e+300,Infinity) | (Infinity,1e+300) | NaN (1e+300,Infinity) | (NaN,NaN) | NaN (1e+300,Infinity) | (10,10) | Infinity + (Infinity,1e+300) | (0,0) | 0 + (Infinity,1e+300) | (-10,0) | 0 + (Infinity,1e+300) | (-3,4) | 0 + (Infinity,1e+300) | (5.1,34.5) | 0 + (Infinity,1e+300) | (-5,-12) | 0 + (Infinity,1e+300) | (1e-300,-1e-300) | 0 + (Infinity,1e+300) | (1e+300,Infinity) | NaN + (Infinity,1e+300) | (Infinity,1e+300) | 0 + (Infinity,1e+300) | (NaN,NaN) | NaN + (Infinity,1e+300) | (10,10) | 0 (NaN,NaN) | (0,0) | NaN (NaN,NaN) | (-10,0) | NaN (NaN,NaN) | (-3,4) | NaN @@ -183,6 +200,7 @@ SELECT p1.f1, p2.f1, slope(p1.f1, p2.f1) FROM POINT_TBL p1, POINT_TBL p2; (NaN,NaN) | (-5,-12) | NaN (NaN,NaN) | (1e-300,-1e-300) | NaN (NaN,NaN) | (1e+300,Infinity) | NaN + (NaN,NaN) | (Infinity,1e+300) | NaN (NaN,NaN) | (NaN,NaN) | NaN (NaN,NaN) | (10,10) | NaN (10,10) | (0,0) | 1 @@ -192,14 +210,15 @@ SELECT p1.f1, p2.f1, slope(p1.f1, p2.f1) FROM POINT_TBL p1, POINT_TBL p2; (10,10) | (-5,-12) | 1.46666666667 (10,10) | (1e-300,-1e-300) | 1 (10,10) | (1e+300,Infinity) | Infinity + (10,10) | (Infinity,1e+300) | 0 (10,10) | (NaN,NaN) | NaN (10,10) | (10,10) | 1.79769313486e+308 -(81 rows) +(100 rows) -- Add point SELECT p1.f1, p2.f1, p1.f1 + p2.f1 FROM POINT_TBL p1, POINT_TBL p2; - f1 | f1 | ?column? --------------------+-------------------+------------------- + f1 | f1 | ?column? +-------------------+-------------------+--------------------- (0,0) | (0,0) | (0,0) (0,0) | (-10,0) | (-10,0) (0,0) | (-3,4) | (-3,4) @@ -207,6 +226,7 @@ SELECT p1.f1, p2.f1, p1.f1 + p2.f1 FROM POINT_TBL p1, POINT_TBL p2; (0,0) | (-5,-12) | (-5,-12) (0,0) | (1e-300,-1e-300) | (1e-300,-1e-300) (0,0) | (1e+300,Infinity) | (1e+300,Infinity) + (0,0) | (Infinity,1e+300) | (Infinity,1e+300) (0,0) | (NaN,NaN) | (NaN,NaN) (0,0) | (10,10) | (10,10) (-10,0) | (0,0) | (-10,0) @@ -216,6 +236,7 @@ SELECT p1.f1, p2.f1, p1.f1 + p2.f1 FROM POINT_TBL p1, POINT_TBL p2; (-10,0) | (-5,-12) | (-15,-12) (-10,0) | (1e-300,-1e-300) | (-10,-1e-300) (-10,0) | (1e+300,Infinity) | (1e+300,Infinity) + (-10,0) | (Infinity,1e+300) | (Infinity,1e+300) (-10,0) | (NaN,NaN) | (NaN,NaN) (-10,0) | (10,10) | (0,10) (-3,4) | (0,0) | (-3,4) @@ -225,6 +246,7 @@ SELECT p1.f1, p2.f1, p1.f1 + p2.f1 FROM POINT_TBL p1, POINT_TBL p2; (-3,4) | (-5,-12) | (-8,-8) (-3,4) | (1e-300,-1e-300) | (-3,4) (-3,4) | (1e+300,Infinity) | (1e+300,Infinity) + (-3,4) | (Infinity,1e+300) | (Infinity,1e+300) (-3,4) | (NaN,NaN) | (NaN,NaN) (-3,4) | (10,10) | (7,14) (5.1,34.5) | (0,0) | (5.1,34.5) @@ -234,6 +256,7 @@ SELECT p1.f1, p2.f1, p1.f1 + p2.f1 FROM POINT_TBL p1, POINT_TBL p2; (5.1,34.5) | (-5,-12) | (0.1,22.5) (5.1,34.5) | (1e-300,-1e-300) | (5.1,34.5) (5.1,34.5) | (1e+300,Infinity) | (1e+300,Infinity) + (5.1,34.5) | (Infinity,1e+300) | (Infinity,1e+300) (5.1,34.5) | (NaN,NaN) | (NaN,NaN) (5.1,34.5) | (10,10) | (15.1,44.5) (-5,-12) | (0,0) | (-5,-12) @@ -243,6 +266,7 @@ SELECT p1.f1, p2.f1, p1.f1 + p2.f1 FROM POINT_TBL p1, POINT_TBL p2; (-5,-12) | (-5,-12) | (-10,-24) (-5,-12) | (1e-300,-1e-300) | (-5,-12) (-5,-12) | (1e+300,Infinity) | (1e+300,Infinity) + (-5,-12) | (Infinity,1e+300) | (Infinity,1e+300) (-5,-12) | (NaN,NaN) | (NaN,NaN) (-5,-12) | (10,10) | (5,-2) (1e-300,-1e-300) | (0,0) | (1e-300,-1e-300) @@ -252,6 +276,7 @@ SELECT p1.f1, p2.f1, p1.f1 + p2.f1 FROM POINT_TBL p1, POINT_TBL p2; (1e-300,-1e-300) | (-5,-12) | (-5,-12) (1e-300,-1e-300) | (1e-300,-1e-300) | (2e-300,-2e-300) (1e-300,-1e-300) | (1e+300,Infinity) | (1e+300,Infinity) + (1e-300,-1e-300) | (Infinity,1e+300) | (Infinity,1e+300) (1e-300,-1e-300) | (NaN,NaN) | (NaN,NaN) (1e-300,-1e-300) | (10,10) | (10,10) (1e+300,Infinity) | (0,0) | (1e+300,Infinity) @@ -261,8 +286,19 @@ SELECT p1.f1, p2.f1, p1.f1 + p2.f1 FROM POINT_TBL p1, POINT_TBL p2; (1e+300,Infinity) | (-5,-12) | (1e+300,Infinity) (1e+300,Infinity) | (1e-300,-1e-300) | (1e+300,Infinity) (1e+300,Infinity) | (1e+300,Infinity) | (2e+300,Infinity) + (1e+300,Infinity) | (Infinity,1e+300) | (Infinity,Infinity) (1e+300,Infinity) | (NaN,NaN) | (NaN,NaN) (1e+300,Infinity) | (10,10) | (1e+300,Infinity) + (Infinity,1e+300) | (0,0) | (Infinity,1e+300) + (Infinity,1e+300) | (-10,0) | (Infinity,1e+300) + (Infinity,1e+300) | (-3,4) | (Infinity,1e+300) + (Infinity,1e+300) | (5.1,34.5) | (Infinity,1e+300) + (Infinity,1e+300) | (-5,-12) | (Infinity,1e+300) + (Infinity,1e+300) | (1e-300,-1e-300) | (Infinity,1e+300) + (Infinity,1e+300) | (1e+300,Infinity) | (Infinity,Infinity) + (Infinity,1e+300) | (Infinity,1e+300) | (Infinity,2e+300) + (Infinity,1e+300) | (NaN,NaN) | (NaN,NaN) + (Infinity,1e+300) | (10,10) | (Infinity,1e+300) (NaN,NaN) | (0,0) | (NaN,NaN) (NaN,NaN) | (-10,0) | (NaN,NaN) (NaN,NaN) | (-3,4) | (NaN,NaN) @@ -270,6 +306,7 @@ SELECT p1.f1, p2.f1, p1.f1 + p2.f1 FROM POINT_TBL p1, POINT_TBL p2; (NaN,NaN) | (-5,-12) | (NaN,NaN) (NaN,NaN) | (1e-300,-1e-300) | (NaN,NaN) (NaN,NaN) | (1e+300,Infinity) | (NaN,NaN) + (NaN,NaN) | (Infinity,1e+300) | (NaN,NaN) (NaN,NaN) | (NaN,NaN) | (NaN,NaN) (NaN,NaN) | (10,10) | (NaN,NaN) (10,10) | (0,0) | (10,10) @@ -279,14 +316,15 @@ SELECT p1.f1, p2.f1, p1.f1 + p2.f1 FROM POINT_TBL p1, POINT_TBL p2; (10,10) | (-5,-12) | (5,-2) (10,10) | (1e-300,-1e-300) | (10,10) (10,10) | (1e+300,Infinity) | (1e+300,Infinity) + (10,10) | (Infinity,1e+300) | (Infinity,1e+300) (10,10) | (NaN,NaN) | (NaN,NaN) (10,10) | (10,10) | (20,20) -(81 rows) +(100 rows) -- Subtract point SELECT p1.f1, p2.f1, p1.f1 - p2.f1 FROM POINT_TBL p1, POINT_TBL p2; - f1 | f1 | ?column? --------------------+-------------------+--------------------- + f1 | f1 | ?column? +-------------------+-------------------+---------------------- (0,0) | (0,0) | (0,0) (0,0) | (-10,0) | (10,0) (0,0) | (-3,4) | (3,-4) @@ -294,6 +332,7 @@ SELECT p1.f1, p2.f1, p1.f1 - p2.f1 FROM POINT_TBL p1, POINT_TBL p2; (0,0) | (-5,-12) | (5,12) (0,0) | (1e-300,-1e-300) | (-1e-300,1e-300) (0,0) | (1e+300,Infinity) | (-1e+300,-Infinity) + (0,0) | (Infinity,1e+300) | (-Infinity,-1e+300) (0,0) | (NaN,NaN) | (NaN,NaN) (0,0) | (10,10) | (-10,-10) (-10,0) | (0,0) | (-10,0) @@ -303,6 +342,7 @@ SELECT p1.f1, p2.f1, p1.f1 - p2.f1 FROM POINT_TBL p1, POINT_TBL p2; (-10,0) | (-5,-12) | (-5,12) (-10,0) | (1e-300,-1e-300) | (-10,1e-300) (-10,0) | (1e+300,Infinity) | (-1e+300,-Infinity) + (-10,0) | (Infinity,1e+300) | (-Infinity,-1e+300) (-10,0) | (NaN,NaN) | (NaN,NaN) (-10,0) | (10,10) | (-20,-10) (-3,4) | (0,0) | (-3,4) @@ -312,6 +352,7 @@ SELECT p1.f1, p2.f1, p1.f1 - p2.f1 FROM POINT_TBL p1, POINT_TBL p2; (-3,4) | (-5,-12) | (2,16) (-3,4) | (1e-300,-1e-300) | (-3,4) (-3,4) | (1e+300,Infinity) | (-1e+300,-Infinity) + (-3,4) | (Infinity,1e+300) | (-Infinity,-1e+300) (-3,4) | (NaN,NaN) | (NaN,NaN) (-3,4) | (10,10) | (-13,-6) (5.1,34.5) | (0,0) | (5.1,34.5) @@ -321,6 +362,7 @@ SELECT p1.f1, p2.f1, p1.f1 - p2.f1 FROM POINT_TBL p1, POINT_TBL p2; (5.1,34.5) | (-5,-12) | (10.1,46.5) (5.1,34.5) | (1e-300,-1e-300) | (5.1,34.5) (5.1,34.5) | (1e+300,Infinity) | (-1e+300,-Infinity) + (5.1,34.5) | (Infinity,1e+300) | (-Infinity,-1e+300) (5.1,34.5) | (NaN,NaN) | (NaN,NaN) (5.1,34.5) | (10,10) | (-4.9,24.5) (-5,-12) | (0,0) | (-5,-12) @@ -330,6 +372,7 @@ SELECT p1.f1, p2.f1, p1.f1 - p2.f1 FROM POINT_TBL p1, POINT_TBL p2; (-5,-12) | (-5,-12) | (0,0) (-5,-12) | (1e-300,-1e-300) | (-5,-12) (-5,-12) | (1e+300,Infinity) | (-1e+300,-Infinity) + (-5,-12) | (Infinity,1e+300) | (-Infinity,-1e+300) (-5,-12) | (NaN,NaN) | (NaN,NaN) (-5,-12) | (10,10) | (-15,-22) (1e-300,-1e-300) | (0,0) | (1e-300,-1e-300) @@ -339,6 +382,7 @@ SELECT p1.f1, p2.f1, p1.f1 - p2.f1 FROM POINT_TBL p1, POINT_TBL p2; (1e-300,-1e-300) | (-5,-12) | (5,12) (1e-300,-1e-300) | (1e-300,-1e-300) | (0,0) (1e-300,-1e-300) | (1e+300,Infinity) | (-1e+300,-Infinity) + (1e-300,-1e-300) | (Infinity,1e+300) | (-Infinity,-1e+300) (1e-300,-1e-300) | (NaN,NaN) | (NaN,NaN) (1e-300,-1e-300) | (10,10) | (-10,-10) (1e+300,Infinity) | (0,0) | (1e+300,Infinity) @@ -348,8 +392,19 @@ SELECT p1.f1, p2.f1, p1.f1 - p2.f1 FROM POINT_TBL p1, POINT_TBL p2; (1e+300,Infinity) | (-5,-12) | (1e+300,Infinity) (1e+300,Infinity) | (1e-300,-1e-300) | (1e+300,Infinity) (1e+300,Infinity) | (1e+300,Infinity) | (0,NaN) + (1e+300,Infinity) | (Infinity,1e+300) | (-Infinity,Infinity) (1e+300,Infinity) | (NaN,NaN) | (NaN,NaN) (1e+300,Infinity) | (10,10) | (1e+300,Infinity) + (Infinity,1e+300) | (0,0) | (Infinity,1e+300) + (Infinity,1e+300) | (-10,0) | (Infinity,1e+300) + (Infinity,1e+300) | (-3,4) | (Infinity,1e+300) + (Infinity,1e+300) | (5.1,34.5) | (Infinity,1e+300) + (Infinity,1e+300) | (-5,-12) | (Infinity,1e+300) + (Infinity,1e+300) | (1e-300,-1e-300) | (Infinity,1e+300) + (Infinity,1e+300) | (1e+300,Infinity) | (Infinity,-Infinity) + (Infinity,1e+300) | (Infinity,1e+300) | (NaN,0) + (Infinity,1e+300) | (NaN,NaN) | (NaN,NaN) + (Infinity,1e+300) | (10,10) | (Infinity,1e+300) (NaN,NaN) | (0,0) | (NaN,NaN) (NaN,NaN) | (-10,0) | (NaN,NaN) (NaN,NaN) | (-3,4) | (NaN,NaN) @@ -357,6 +412,7 @@ SELECT p1.f1, p2.f1, p1.f1 - p2.f1 FROM POINT_TBL p1, POINT_TBL p2; (NaN,NaN) | (-5,-12) | (NaN,NaN) (NaN,NaN) | (1e-300,-1e-300) | (NaN,NaN) (NaN,NaN) | (1e+300,Infinity) | (NaN,NaN) + (NaN,NaN) | (Infinity,1e+300) | (NaN,NaN) (NaN,NaN) | (NaN,NaN) | (NaN,NaN) (NaN,NaN) | (10,10) | (NaN,NaN) (10,10) | (0,0) | (10,10) @@ -366,9 +422,10 @@ SELECT p1.f1, p2.f1, p1.f1 - p2.f1 FROM POINT_TBL p1, POINT_TBL p2; (10,10) | (-5,-12) | (15,22) (10,10) | (1e-300,-1e-300) | (10,10) (10,10) | (1e+300,Infinity) | (-1e+300,-Infinity) + (10,10) | (Infinity,1e+300) | (-Infinity,-1e+300) (10,10) | (NaN,NaN) | (NaN,NaN) (10,10) | (10,10) | (0,0) -(81 rows) +(100 rows) -- Multiply with point SELECT p1.f1, p2.f1, p1.f1 * p2.f1 FROM POINT_TBL p1, POINT_TBL p2 WHERE p1.f1[0] BETWEEN 1 AND 1000; @@ -388,11 +445,13 @@ SELECT p1.f1, p2.f1, p1.f1 * p2.f1 FROM POINT_TBL p1, POINT_TBL p2 WHERE p1.f1[0 (10,10) | (1e-300,-1e-300) | (2e-299,0) (5.1,34.5) | (1e+300,Infinity) | (-Infinity,Infinity) (10,10) | (1e+300,Infinity) | (-Infinity,Infinity) + (5.1,34.5) | (Infinity,1e+300) | (Infinity,Infinity) + (10,10) | (Infinity,1e+300) | (Infinity,Infinity) (5.1,34.5) | (NaN,NaN) | (NaN,NaN) (10,10) | (NaN,NaN) | (NaN,NaN) (5.1,34.5) | (10,10) | (-294,396) (10,10) | (10,10) | (0,200) -(18 rows) +(20 rows) -- Underflow error SELECT p1.f1, p2.f1, p1.f1 * p2.f1 FROM POINT_TBL p1, POINT_TBL p2 WHERE p1.f1[0] < 1; @@ -415,11 +474,13 @@ SELECT p1.f1, p2.f1, p1.f1 / p2.f1 FROM POINT_TBL p1, POINT_TBL p2 WHERE p2.f1[0 (1e-300,-1e-300) | (10,10) | (0,-1e-301) (1e+300,Infinity) | (5.1,34.5) | (Infinity,Infinity) (1e+300,Infinity) | (10,10) | (Infinity,Infinity) + (Infinity,1e+300) | (5.1,34.5) | (Infinity,-Infinity) + (Infinity,1e+300) | (10,10) | (Infinity,-Infinity) (NaN,NaN) | (5.1,34.5) | (NaN,NaN) (NaN,NaN) | (10,10) | (NaN,NaN) (10,10) | (5.1,34.5) | (0.325588278822,-0.241724631247) (10,10) | (10,10) | (1,0) -(18 rows) +(20 rows) -- Overflow error SELECT p1.f1, p2.f1, p1.f1 / p2.f1 FROM POINT_TBL p1, POINT_TBL p2 WHERE p2.f1[0] > 1000; @@ -494,13 +555,23 @@ SELECT p.f1, l.s, p.f1 <-> l.s AS dist_pl, l.s <-> p.f1 AS dist_lp FROM POINT_TB (1e+300,Infinity) | {0,-1,5} | Infinity | Infinity (1e+300,Infinity) | {1,0,5} | NaN | NaN (1e+300,Infinity) | {0,3,0} | Infinity | Infinity - (1e+300,Infinity) | {1,-1,0} | Infinity | Infinity - (1e+300,Infinity) | {-0.4,-1,-6} | Infinity | Infinity - (1e+300,Infinity) | {-0.000184615384615,-1,15.3846153846} | Infinity | Infinity + (1e+300,Infinity) | {1,-1,0} | NaN | NaN + (1e+300,Infinity) | {-0.4,-1,-6} | NaN | NaN + (1e+300,Infinity) | {-0.000184615384615,-1,15.3846153846} | NaN | NaN (1e+300,Infinity) | {3,NaN,5} | NaN | NaN (1e+300,Infinity) | {NaN,NaN,NaN} | NaN | NaN (1e+300,Infinity) | {0,-1,3} | Infinity | Infinity (1e+300,Infinity) | {-1,0,3} | NaN | NaN + (Infinity,1e+300) | {0,-1,5} | NaN | NaN + (Infinity,1e+300) | {1,0,5} | Infinity | Infinity + (Infinity,1e+300) | {0,3,0} | NaN | NaN + (Infinity,1e+300) | {1,-1,0} | NaN | NaN + (Infinity,1e+300) | {-0.4,-1,-6} | NaN | NaN + (Infinity,1e+300) | {-0.000184615384615,-1,15.3846153846} | NaN | NaN + (Infinity,1e+300) | {3,NaN,5} | NaN | NaN + (Infinity,1e+300) | {NaN,NaN,NaN} | NaN | NaN + (Infinity,1e+300) | {0,-1,3} | NaN | NaN + (Infinity,1e+300) | {-1,0,3} | Infinity | Infinity (NaN,NaN) | {0,-1,5} | NaN | NaN (NaN,NaN) | {1,0,5} | NaN | NaN (NaN,NaN) | {0,3,0} | NaN | NaN @@ -521,7 +592,7 @@ SELECT p.f1, l.s, p.f1 <-> l.s AS dist_pl, l.s <-> p.f1 AS dist_lp FROM POINT_TB (10,10) | {NaN,NaN,NaN} | NaN | NaN (10,10) | {0,-1,3} | 7 | 7 (10,10) | {-1,0,3} | 7 | 7 -(90 rows) +(100 rows) -- Distance to line segment SELECT p.f1, l.s, p.f1 <-> l.s AS dist_ps, l.s <-> p.f1 AS dist_sp FROM POINT_TBL p, LSEG_TBL l; @@ -582,7 +653,15 @@ SELECT p.f1, l.s, p.f1 <-> l.s AS dist_ps, l.s <-> p.f1 AS dist_sp FROM POINT_TB (1e+300,Infinity) | [(11,22),(33,44)] | Infinity | Infinity (1e+300,Infinity) | [(-10,2),(-10,3)] | Infinity | Infinity (1e+300,Infinity) | [(0,-20),(30,-20)] | Infinity | Infinity - (1e+300,Infinity) | [(NaN,1),(NaN,90)] | Infinity | Infinity + (1e+300,Infinity) | [(NaN,1),(NaN,90)] | NaN | NaN + (Infinity,1e+300) | [(1,2),(3,4)] | Infinity | Infinity + (Infinity,1e+300) | [(0,0),(6,6)] | Infinity | Infinity + (Infinity,1e+300) | [(10,-10),(-3,-4)] | Infinity | Infinity + (Infinity,1e+300) | [(-1000000,200),(300000,-40)] | Infinity | Infinity + (Infinity,1e+300) | [(11,22),(33,44)] | Infinity | Infinity + (Infinity,1e+300) | [(-10,2),(-10,3)] | Infinity | Infinity + (Infinity,1e+300) | [(0,-20),(30,-20)] | NaN | NaN + (Infinity,1e+300) | [(NaN,1),(NaN,90)] | NaN | NaN (NaN,NaN) | [(1,2),(3,4)] | NaN | NaN (NaN,NaN) | [(0,0),(6,6)] | NaN | NaN (NaN,NaN) | [(10,-10),(-3,-4)] | NaN | NaN @@ -599,7 +678,7 @@ SELECT p.f1, l.s, p.f1 <-> l.s AS dist_ps, l.s <-> p.f1 AS dist_sp FROM POINT_TB (10,10) | [(-10,2),(-10,3)] | 21.1896201004 | 21.1896201004 (10,10) | [(0,-20),(30,-20)] | 30 | 30 (10,10) | [(NaN,1),(NaN,90)] | NaN | NaN -(72 rows) +(80 rows) -- Distance to box SELECT p.f1, b.f1, p.f1 <-> b.f1 AS dist_pb, b.f1 <-> p.f1 AS dist_bp FROM POINT_TBL p, BOX_TBL b; @@ -640,6 +719,11 @@ SELECT p.f1, b.f1, p.f1 <-> b.f1 AS dist_pb, b.f1 <-> p.f1 AS dist_bp FROM POINT (1e+300,Infinity) | (-2,2),(-8,-10) | Infinity | Infinity (1e+300,Infinity) | (2.5,3.5),(2.5,2.5) | Infinity | Infinity (1e+300,Infinity) | (3,3),(3,3) | Infinity | Infinity + (Infinity,1e+300) | (2,2),(0,0) | NaN | NaN + (Infinity,1e+300) | (3,3),(1,1) | NaN | NaN + (Infinity,1e+300) | (-2,2),(-8,-10) | NaN | NaN + (Infinity,1e+300) | (2.5,3.5),(2.5,2.5) | Infinity | Infinity + (Infinity,1e+300) | (3,3),(3,3) | Infinity | Infinity (NaN,NaN) | (2,2),(0,0) | NaN | NaN (NaN,NaN) | (3,3),(1,1) | NaN | NaN (NaN,NaN) | (-2,2),(-8,-10) | NaN | NaN @@ -650,7 +734,7 @@ SELECT p.f1, b.f1, p.f1 <-> b.f1 AS dist_pb, b.f1 <-> p.f1 AS dist_bp FROM POINT (10,10) | (-2,2),(-8,-10) | 14.4222051019 | 14.4222051019 (10,10) | (2.5,3.5),(2.5,2.5) | 9.92471662064 | 9.92471662064 (10,10) | (3,3),(3,3) | 9.89949493661 | 9.89949493661 -(45 rows) +(50 rows) -- Distance to path SELECT p.f1, p1.f1, p.f1 <-> p1.f1 AS dist_ppath, p1.f1 <-> p.f1 AS dist_pathp FROM POINT_TBL p, PATH_TBL p1; @@ -719,6 +803,15 @@ SELECT p.f1, p1.f1, p.f1 <-> p1.f1 AS dist_ppath, p1.f1 <-> p.f1 AS dist_pathp F (1e+300,Infinity) | ((10,20)) | Infinity | Infinity (1e+300,Infinity) | [(11,12),(13,14)] | Infinity | Infinity (1e+300,Infinity) | ((11,12),(13,14)) | Infinity | Infinity + (Infinity,1e+300) | [(1,2),(3,4)] | Infinity | Infinity + (Infinity,1e+300) | ((1,2),(3,4)) | Infinity | Infinity + (Infinity,1e+300) | [(0,0),(3,0),(4,5),(1,6)] | NaN | NaN + (Infinity,1e+300) | ((1,2),(3,4)) | Infinity | Infinity + (Infinity,1e+300) | ((1,2),(3,4)) | Infinity | Infinity + (Infinity,1e+300) | [(1,2),(3,4)] | Infinity | Infinity + (Infinity,1e+300) | ((10,20)) | Infinity | Infinity + (Infinity,1e+300) | [(11,12),(13,14)] | Infinity | Infinity + (Infinity,1e+300) | ((11,12),(13,14)) | Infinity | Infinity (NaN,NaN) | [(1,2),(3,4)] | NaN | NaN (NaN,NaN) | ((1,2),(3,4)) | NaN | NaN (NaN,NaN) | [(0,0),(3,0),(4,5),(1,6)] | NaN | NaN @@ -737,7 +830,7 @@ SELECT p.f1, p1.f1, p.f1 <-> p1.f1 AS dist_ppath, p1.f1 <-> p.f1 AS dist_pathp F (10,10) | ((10,20)) | 10 | 10 (10,10) | [(11,12),(13,14)] | 2.2360679775 | 2.2360679775 (10,10) | ((11,12),(13,14)) | 2.2360679775 | 2.2360679775 -(81 rows) +(90 rows) -- Distance to polygon SELECT p.f1, p1.f1, p.f1 <-> p1.f1 AS dist_ppoly, p1.f1 <-> p.f1 AS dist_polyp FROM POINT_TBL p, POLYGON_TBL p1; @@ -792,13 +885,20 @@ SELECT p.f1, p1.f1, p.f1 <-> p1.f1 AS dist_ppoly, p1.f1 <-> p.f1 AS dist_polyp F (1e+300,Infinity) | ((1,2),(7,8),(5,6),(3,-4)) | Infinity | Infinity (1e+300,Infinity) | ((0,0)) | Infinity | Infinity (1e+300,Infinity) | ((0,1),(0,1)) | Infinity | Infinity - (NaN,NaN) | ((2,0),(2,4),(0,0)) | 0 | 0 - (NaN,NaN) | ((3,1),(3,3),(1,0)) | 0 | 0 - (NaN,NaN) | ((1,2),(3,4),(5,6),(7,8)) | 0 | 0 - (NaN,NaN) | ((7,8),(5,6),(3,4),(1,2)) | 0 | 0 - (NaN,NaN) | ((1,2),(7,8),(5,6),(3,-4)) | 0 | 0 - (NaN,NaN) | ((0,0)) | 0 | 0 - (NaN,NaN) | ((0,1),(0,1)) | 0 | 0 + (Infinity,1e+300) | ((2,0),(2,4),(0,0)) | Infinity | Infinity + (Infinity,1e+300) | ((3,1),(3,3),(1,0)) | Infinity | Infinity + (Infinity,1e+300) | ((1,2),(3,4),(5,6),(7,8)) | Infinity | Infinity + (Infinity,1e+300) | ((7,8),(5,6),(3,4),(1,2)) | Infinity | Infinity + (Infinity,1e+300) | ((1,2),(7,8),(5,6),(3,-4)) | Infinity | Infinity + (Infinity,1e+300) | ((0,0)) | Infinity | Infinity + (Infinity,1e+300) | ((0,1),(0,1)) | Infinity | Infinity + (NaN,NaN) | ((2,0),(2,4),(0,0)) | NaN | NaN + (NaN,NaN) | ((3,1),(3,3),(1,0)) | NaN | NaN + (NaN,NaN) | ((1,2),(3,4),(5,6),(7,8)) | NaN | NaN + (NaN,NaN) | ((7,8),(5,6),(3,4),(1,2)) | NaN | NaN + (NaN,NaN) | ((1,2),(7,8),(5,6),(3,-4)) | NaN | NaN + (NaN,NaN) | ((0,0)) | NaN | NaN + (NaN,NaN) | ((0,1),(0,1)) | NaN | NaN (10,10) | ((2,0),(2,4),(0,0)) | 10 | 10 (10,10) | ((3,1),(3,3),(1,0)) | 9.89949493661 | 9.89949493661 (10,10) | ((1,2),(3,4),(5,6),(7,8)) | 3.60555127546 | 3.60555127546 @@ -806,7 +906,7 @@ SELECT p.f1, p1.f1, p.f1 <-> p1.f1 AS dist_ppoly, p1.f1 <-> p.f1 AS dist_polyp F (10,10) | ((1,2),(7,8),(5,6),(3,-4)) | 3.60555127546 | 3.60555127546 (10,10) | ((0,0)) | 14.1421356237 | 14.1421356237 (10,10) | ((0,1),(0,1)) | 13.4536240471 | 13.4536240471 -(63 rows) +(70 rows) -- Closest point to line SELECT p.f1, l.s, p.f1 ## l.s FROM POINT_TBL p, LINE_TBL l; @@ -875,13 +975,23 @@ SELECT p.f1, l.s, p.f1 ## l.s FROM POINT_TBL p, LINE_TBL l; (1e+300,Infinity) | {0,-1,5} | (1e+300,5) (1e+300,Infinity) | {1,0,5} | (1e+300,Infinity) | {0,3,0} | (1e+300,0) - (1e+300,Infinity) | {1,-1,0} | (Infinity,NaN) - (1e+300,Infinity) | {-0.4,-1,-6} | (-Infinity,NaN) - (1e+300,Infinity) | {-0.000184615384615,-1,15.3846153846} | (-Infinity,NaN) + (1e+300,Infinity) | {1,-1,0} | + (1e+300,Infinity) | {-0.4,-1,-6} | + (1e+300,Infinity) | {-0.000184615384615,-1,15.3846153846} | (1e+300,Infinity) | {3,NaN,5} | (1e+300,Infinity) | {NaN,NaN,NaN} | (1e+300,Infinity) | {0,-1,3} | (1e+300,3) (1e+300,Infinity) | {-1,0,3} | + (Infinity,1e+300) | {0,-1,5} | + (Infinity,1e+300) | {1,0,5} | (-5,1e+300) + (Infinity,1e+300) | {0,3,0} | + (Infinity,1e+300) | {1,-1,0} | + (Infinity,1e+300) | {-0.4,-1,-6} | + (Infinity,1e+300) | {-0.000184615384615,-1,15.3846153846} | + (Infinity,1e+300) | {3,NaN,5} | + (Infinity,1e+300) | {NaN,NaN,NaN} | + (Infinity,1e+300) | {0,-1,3} | + (Infinity,1e+300) | {-1,0,3} | (3,1e+300) (NaN,NaN) | {0,-1,5} | (NaN,NaN) | {1,0,5} | (NaN,NaN) | {0,3,0} | @@ -902,7 +1012,7 @@ SELECT p.f1, l.s, p.f1 ## l.s FROM POINT_TBL p, LINE_TBL l; (10,10) | {NaN,NaN,NaN} | (10,10) | {0,-1,3} | (10,3) (10,10) | {-1,0,3} | (3,10) -(90 rows) +(100 rows) -- Closest point to line segment SELECT p.f1, l.s, p.f1 ## l.s FROM POINT_TBL p, LSEG_TBL l; @@ -963,7 +1073,15 @@ SELECT p.f1, l.s, p.f1 ## l.s FROM POINT_TBL p, LSEG_TBL l; (1e+300,Infinity) | [(11,22),(33,44)] | (33,44) (1e+300,Infinity) | [(-10,2),(-10,3)] | (-10,3) (1e+300,Infinity) | [(0,-20),(30,-20)] | (30,-20) - (1e+300,Infinity) | [(NaN,1),(NaN,90)] | (NaN,90) + (1e+300,Infinity) | [(NaN,1),(NaN,90)] | + (Infinity,1e+300) | [(1,2),(3,4)] | (3,4) + (Infinity,1e+300) | [(0,0),(6,6)] | (6,6) + (Infinity,1e+300) | [(10,-10),(-3,-4)] | (-3,-4) + (Infinity,1e+300) | [(-1000000,200),(300000,-40)] | (300000,-40) + (Infinity,1e+300) | [(11,22),(33,44)] | (33,44) + (Infinity,1e+300) | [(-10,2),(-10,3)] | (-10,3) + (Infinity,1e+300) | [(0,-20),(30,-20)] | + (Infinity,1e+300) | [(NaN,1),(NaN,90)] | (NaN,NaN) | [(1,2),(3,4)] | (NaN,NaN) | [(0,0),(6,6)] | (NaN,NaN) | [(10,-10),(-3,-4)] | @@ -980,7 +1098,7 @@ SELECT p.f1, l.s, p.f1 ## l.s FROM POINT_TBL p, LSEG_TBL l; (10,10) | [(-10,2),(-10,3)] | (-10,3) (10,10) | [(0,-20),(30,-20)] | (10,-20) (10,10) | [(NaN,1),(NaN,90)] | -(72 rows) +(80 rows) -- Closest point to box SELECT p.f1, b.f1, p.f1 ## b.f1 FROM POINT_TBL p, BOX_TBL b; @@ -1021,6 +1139,11 @@ SELECT p.f1, b.f1, p.f1 ## b.f1 FROM POINT_TBL p, BOX_TBL b; (1e+300,Infinity) | (-2,2),(-8,-10) | (-8,2) (1e+300,Infinity) | (2.5,3.5),(2.5,2.5) | (2.5,3.5) (1e+300,Infinity) | (3,3),(3,3) | (3,3) + (Infinity,1e+300) | (2,2),(0,0) | + (Infinity,1e+300) | (3,3),(1,1) | + (Infinity,1e+300) | (-2,2),(-8,-10) | + (Infinity,1e+300) | (2.5,3.5),(2.5,2.5) | (2.5,3.5) + (Infinity,1e+300) | (3,3),(3,3) | (3,3) (NaN,NaN) | (2,2),(0,0) | (NaN,NaN) | (3,3),(1,1) | (NaN,NaN) | (-2,2),(-8,-10) | @@ -1031,7 +1154,7 @@ SELECT p.f1, b.f1, p.f1 ## b.f1 FROM POINT_TBL p, BOX_TBL b; (10,10) | (-2,2),(-8,-10) | (-2,2) (10,10) | (2.5,3.5),(2.5,2.5) | (2.5,3.5) (10,10) | (3,3),(3,3) | (3,3) -(45 rows) +(50 rows) -- On line SELECT p.f1, l.s FROM POINT_TBL p, LINE_TBL l WHERE p.f1 <@ l.s; @@ -1060,12 +1183,7 @@ SELECT p.f1, p1.f1 FROM POINT_TBL p, PATH_TBL p1 WHERE p.f1 <@ p1.f1; ------------------+--------------------------- (0,0) | [(0,0),(3,0),(4,5),(1,6)] (1e-300,-1e-300) | [(0,0),(3,0),(4,5),(1,6)] - (NaN,NaN) | ((1,2),(3,4)) - (NaN,NaN) | ((1,2),(3,4)) - (NaN,NaN) | ((1,2),(3,4)) - (NaN,NaN) | ((10,20)) - (NaN,NaN) | ((11,12),(13,14)) -(7 rows) +(2 rows) -- -- Lines @@ -1153,8 +1271,8 @@ SELECT l1.s, l2.s, l1.s <-> l2.s FROM LINE_TBL l1, LINE_TBL l2; {0,-1,5} | {1,-1,0} | 0 {0,-1,5} | {-0.4,-1,-6} | 0 {0,-1,5} | {-0.000184615384615,-1,15.3846153846} | 0 - {0,-1,5} | {3,NaN,5} | 0 - {0,-1,5} | {NaN,NaN,NaN} | 0 + {0,-1,5} | {3,NaN,5} | NaN + {0,-1,5} | {NaN,NaN,NaN} | NaN {0,-1,5} | {0,-1,3} | 2 {0,-1,5} | {-1,0,3} | 0 {1,0,5} | {0,-1,5} | 0 @@ -1163,8 +1281,8 @@ SELECT l1.s, l2.s, l1.s <-> l2.s FROM LINE_TBL l1, LINE_TBL l2; {1,0,5} | {1,-1,0} | 0 {1,0,5} | {-0.4,-1,-6} | 0 {1,0,5} | {-0.000184615384615,-1,15.3846153846} | 0 - {1,0,5} | {3,NaN,5} | 0 - {1,0,5} | {NaN,NaN,NaN} | 0 + {1,0,5} | {3,NaN,5} | NaN + {1,0,5} | {NaN,NaN,NaN} | NaN {1,0,5} | {0,-1,3} | 0 {1,0,5} | {-1,0,3} | 8 {0,3,0} | {0,-1,5} | 5 @@ -1173,8 +1291,8 @@ SELECT l1.s, l2.s, l1.s <-> l2.s FROM LINE_TBL l1, LINE_TBL l2; {0,3,0} | {1,-1,0} | 0 {0,3,0} | {-0.4,-1,-6} | 0 {0,3,0} | {-0.000184615384615,-1,15.3846153846} | 0 - {0,3,0} | {3,NaN,5} | 0 - {0,3,0} | {NaN,NaN,NaN} | 0 + {0,3,0} | {3,NaN,5} | NaN + {0,3,0} | {NaN,NaN,NaN} | NaN {0,3,0} | {0,-1,3} | 3 {0,3,0} | {-1,0,3} | 0 {1,-1,0} | {0,-1,5} | 0 @@ -1183,8 +1301,8 @@ SELECT l1.s, l2.s, l1.s <-> l2.s FROM LINE_TBL l1, LINE_TBL l2; {1,-1,0} | {1,-1,0} | 0 {1,-1,0} | {-0.4,-1,-6} | 0 {1,-1,0} | {-0.000184615384615,-1,15.3846153846} | 0 - {1,-1,0} | {3,NaN,5} | 0 - {1,-1,0} | {NaN,NaN,NaN} | 0 + {1,-1,0} | {3,NaN,5} | NaN + {1,-1,0} | {NaN,NaN,NaN} | NaN {1,-1,0} | {0,-1,3} | 0 {1,-1,0} | {-1,0,3} | 0 {-0.4,-1,-6} | {0,-1,5} | 0 @@ -1193,8 +1311,8 @@ SELECT l1.s, l2.s, l1.s <-> l2.s FROM LINE_TBL l1, LINE_TBL l2; {-0.4,-1,-6} | {1,-1,0} | 0 {-0.4,-1,-6} | {-0.4,-1,-6} | 0 {-0.4,-1,-6} | {-0.000184615384615,-1,15.3846153846} | 0 - {-0.4,-1,-6} | {3,NaN,5} | 0 - {-0.4,-1,-6} | {NaN,NaN,NaN} | 0 + {-0.4,-1,-6} | {3,NaN,5} | NaN + {-0.4,-1,-6} | {NaN,NaN,NaN} | NaN {-0.4,-1,-6} | {0,-1,3} | 0 {-0.4,-1,-6} | {-1,0,3} | 0 {-0.000184615384615,-1,15.3846153846} | {0,-1,5} | 0 @@ -1203,38 +1321,38 @@ SELECT l1.s, l2.s, l1.s <-> l2.s FROM LINE_TBL l1, LINE_TBL l2; {-0.000184615384615,-1,15.3846153846} | {1,-1,0} | 0 {-0.000184615384615,-1,15.3846153846} | {-0.4,-1,-6} | 0 {-0.000184615384615,-1,15.3846153846} | {-0.000184615384615,-1,15.3846153846} | 0 - {-0.000184615384615,-1,15.3846153846} | {3,NaN,5} | 0 - {-0.000184615384615,-1,15.3846153846} | {NaN,NaN,NaN} | 0 + {-0.000184615384615,-1,15.3846153846} | {3,NaN,5} | NaN + {-0.000184615384615,-1,15.3846153846} | {NaN,NaN,NaN} | NaN {-0.000184615384615,-1,15.3846153846} | {0,-1,3} | 0 {-0.000184615384615,-1,15.3846153846} | {-1,0,3} | 0 - {3,NaN,5} | {0,-1,5} | 0 - {3,NaN,5} | {1,0,5} | 0 - {3,NaN,5} | {0,3,0} | 0 - {3,NaN,5} | {1,-1,0} | 0 - {3,NaN,5} | {-0.4,-1,-6} | 0 - {3,NaN,5} | {-0.000184615384615,-1,15.3846153846} | 0 - {3,NaN,5} | {3,NaN,5} | 0 - {3,NaN,5} | {NaN,NaN,NaN} | 0 - {3,NaN,5} | {0,-1,3} | 0 - {3,NaN,5} | {-1,0,3} | 0 - {NaN,NaN,NaN} | {0,-1,5} | 0 - {NaN,NaN,NaN} | {1,0,5} | 0 - {NaN,NaN,NaN} | {0,3,0} | 0 - {NaN,NaN,NaN} | {1,-1,0} | 0 - {NaN,NaN,NaN} | {-0.4,-1,-6} | 0 - {NaN,NaN,NaN} | {-0.000184615384615,-1,15.3846153846} | 0 - {NaN,NaN,NaN} | {3,NaN,5} | 0 - {NaN,NaN,NaN} | {NaN,NaN,NaN} | 0 - {NaN,NaN,NaN} | {0,-1,3} | 0 - {NaN,NaN,NaN} | {-1,0,3} | 0 + {3,NaN,5} | {0,-1,5} | NaN + {3,NaN,5} | {1,0,5} | NaN + {3,NaN,5} | {0,3,0} | NaN + {3,NaN,5} | {1,-1,0} | NaN + {3,NaN,5} | {-0.4,-1,-6} | NaN + {3,NaN,5} | {-0.000184615384615,-1,15.3846153846} | NaN + {3,NaN,5} | {3,NaN,5} | NaN + {3,NaN,5} | {NaN,NaN,NaN} | NaN + {3,NaN,5} | {0,-1,3} | NaN + {3,NaN,5} | {-1,0,3} | NaN + {NaN,NaN,NaN} | {0,-1,5} | NaN + {NaN,NaN,NaN} | {1,0,5} | NaN + {NaN,NaN,NaN} | {0,3,0} | NaN + {NaN,NaN,NaN} | {1,-1,0} | NaN + {NaN,NaN,NaN} | {-0.4,-1,-6} | NaN + {NaN,NaN,NaN} | {-0.000184615384615,-1,15.3846153846} | NaN + {NaN,NaN,NaN} | {3,NaN,5} | NaN + {NaN,NaN,NaN} | {NaN,NaN,NaN} | NaN + {NaN,NaN,NaN} | {0,-1,3} | NaN + {NaN,NaN,NaN} | {-1,0,3} | NaN {0,-1,3} | {0,-1,5} | 2 {0,-1,3} | {1,0,5} | 0 {0,-1,3} | {0,3,0} | 3 {0,-1,3} | {1,-1,0} | 0 {0,-1,3} | {-0.4,-1,-6} | 0 {0,-1,3} | {-0.000184615384615,-1,15.3846153846} | 0 - {0,-1,3} | {3,NaN,5} | 0 - {0,-1,3} | {NaN,NaN,NaN} | 0 + {0,-1,3} | {3,NaN,5} | NaN + {0,-1,3} | {NaN,NaN,NaN} | NaN {0,-1,3} | {0,-1,3} | 0 {0,-1,3} | {-1,0,3} | 0 {-1,0,3} | {0,-1,5} | 0 @@ -1243,8 +1361,8 @@ SELECT l1.s, l2.s, l1.s <-> l2.s FROM LINE_TBL l1, LINE_TBL l2; {-1,0,3} | {1,-1,0} | 0 {-1,0,3} | {-0.4,-1,-6} | 0 {-1,0,3} | {-0.000184615384615,-1,15.3846153846} | 0 - {-1,0,3} | {3,NaN,5} | 0 - {-1,0,3} | {NaN,NaN,NaN} | 0 + {-1,0,3} | {3,NaN,5} | NaN + {-1,0,3} | {NaN,NaN,NaN} | NaN {-1,0,3} | {0,-1,3} | 0 {-1,0,3} | {-1,0,3} | 0 (100 rows) @@ -1262,31 +1380,23 @@ SELECT l1.s, l2.s FROM LINE_TBL l1, LINE_TBL l2 WHERE l1.s ?# l2.s; {0,-1,5} | {1,-1,0} {0,-1,5} | {-0.4,-1,-6} {0,-1,5} | {-0.000184615384615,-1,15.3846153846} - {0,-1,5} | {3,NaN,5} - {0,-1,5} | {NaN,NaN,NaN} {0,-1,5} | {-1,0,3} {1,0,5} | {0,-1,5} {1,0,5} | {0,3,0} {1,0,5} | {1,-1,0} {1,0,5} | {-0.4,-1,-6} {1,0,5} | {-0.000184615384615,-1,15.3846153846} - {1,0,5} | {3,NaN,5} - {1,0,5} | {NaN,NaN,NaN} {1,0,5} | {0,-1,3} {0,3,0} | {1,0,5} {0,3,0} | {1,-1,0} {0,3,0} | {-0.4,-1,-6} {0,3,0} | {-0.000184615384615,-1,15.3846153846} - {0,3,0} | {3,NaN,5} - {0,3,0} | {NaN,NaN,NaN} {0,3,0} | {-1,0,3} {1,-1,0} | {0,-1,5} {1,-1,0} | {1,0,5} {1,-1,0} | {0,3,0} {1,-1,0} | {-0.4,-1,-6} {1,-1,0} | {-0.000184615384615,-1,15.3846153846} - {1,-1,0} | {3,NaN,5} - {1,-1,0} | {NaN,NaN,NaN} {1,-1,0} | {0,-1,3} {1,-1,0} | {-1,0,3} {-0.4,-1,-6} | {0,-1,5} @@ -1294,8 +1404,6 @@ SELECT l1.s, l2.s FROM LINE_TBL l1, LINE_TBL l2 WHERE l1.s ?# l2.s; {-0.4,-1,-6} | {0,3,0} {-0.4,-1,-6} | {1,-1,0} {-0.4,-1,-6} | {-0.000184615384615,-1,15.3846153846} - {-0.4,-1,-6} | {3,NaN,5} - {-0.4,-1,-6} | {NaN,NaN,NaN} {-0.4,-1,-6} | {0,-1,3} {-0.4,-1,-6} | {-1,0,3} {-0.000184615384615,-1,15.3846153846} | {0,-1,5} @@ -1303,46 +1411,20 @@ SELECT l1.s, l2.s FROM LINE_TBL l1, LINE_TBL l2 WHERE l1.s ?# l2.s; {-0.000184615384615,-1,15.3846153846} | {0,3,0} {-0.000184615384615,-1,15.3846153846} | {1,-1,0} {-0.000184615384615,-1,15.3846153846} | {-0.4,-1,-6} - {-0.000184615384615,-1,15.3846153846} | {3,NaN,5} - {-0.000184615384615,-1,15.3846153846} | {NaN,NaN,NaN} {-0.000184615384615,-1,15.3846153846} | {0,-1,3} {-0.000184615384615,-1,15.3846153846} | {-1,0,3} - {3,NaN,5} | {0,-1,5} - {3,NaN,5} | {1,0,5} - {3,NaN,5} | {0,3,0} - {3,NaN,5} | {1,-1,0} - {3,NaN,5} | {-0.4,-1,-6} - {3,NaN,5} | {-0.000184615384615,-1,15.3846153846} - {3,NaN,5} | {3,NaN,5} - {3,NaN,5} | {NaN,NaN,NaN} - {3,NaN,5} | {0,-1,3} - {3,NaN,5} | {-1,0,3} - {NaN,NaN,NaN} | {0,-1,5} - {NaN,NaN,NaN} | {1,0,5} - {NaN,NaN,NaN} | {0,3,0} - {NaN,NaN,NaN} | {1,-1,0} - {NaN,NaN,NaN} | {-0.4,-1,-6} - {NaN,NaN,NaN} | {-0.000184615384615,-1,15.3846153846} - {NaN,NaN,NaN} | {3,NaN,5} - {NaN,NaN,NaN} | {NaN,NaN,NaN} - {NaN,NaN,NaN} | {0,-1,3} - {NaN,NaN,NaN} | {-1,0,3} {0,-1,3} | {1,0,5} {0,-1,3} | {1,-1,0} {0,-1,3} | {-0.4,-1,-6} {0,-1,3} | {-0.000184615384615,-1,15.3846153846} - {0,-1,3} | {3,NaN,5} - {0,-1,3} | {NaN,NaN,NaN} {0,-1,3} | {-1,0,3} {-1,0,3} | {0,-1,5} {-1,0,3} | {0,3,0} {-1,0,3} | {1,-1,0} {-1,0,3} | {-0.4,-1,-6} {-1,0,3} | {-0.000184615384615,-1,15.3846153846} - {-1,0,3} | {3,NaN,5} - {-1,0,3} | {NaN,NaN,NaN} {-1,0,3} | {0,-1,3} -(84 rows) +(48 rows) -- Intersect with box SELECT l.s, b.f1 FROM LINE_TBL l, BOX_TBL b WHERE l.s ?# b.f1; @@ -1383,8 +1465,8 @@ SELECT l1.s, l2.s, l1.s # l2.s FROM LINE_TBL l1, LINE_TBL l2; {1,0,5} | {1,-1,0} | (-5,-5) {1,0,5} | {-0.4,-1,-6} | (-5,-4) {1,0,5} | {-0.000184615384615,-1,15.3846153846} | (-5,15.3855384615) - {1,0,5} | {3,NaN,5} | (NaN,NaN) - {1,0,5} | {NaN,NaN,NaN} | (NaN,NaN) + {1,0,5} | {3,NaN,5} | (-5,NaN) + {1,0,5} | {NaN,NaN,NaN} | (-5,NaN) {1,0,5} | {0,-1,3} | (-5,3) {1,0,5} | {-1,0,3} | {0,3,0} | {0,-1,5} | @@ -1463,8 +1545,8 @@ SELECT l1.s, l2.s, l1.s # l2.s FROM LINE_TBL l1, LINE_TBL l2; {-1,0,3} | {1,-1,0} | (3,3) {-1,0,3} | {-0.4,-1,-6} | (3,-7.2) {-1,0,3} | {-0.000184615384615,-1,15.3846153846} | (3,15.3840615385) - {-1,0,3} | {3,NaN,5} | (NaN,NaN) - {-1,0,3} | {NaN,NaN,NaN} | (NaN,NaN) + {-1,0,3} | {3,NaN,5} | (3,NaN) + {-1,0,3} | {NaN,NaN,NaN} | (3,NaN) {-1,0,3} | {0,-1,3} | (3,3) {-1,0,3} | {-1,0,3} | (100 rows) @@ -2347,6 +2429,11 @@ SELECT '' AS twentyfour, b.f1 + p.f1 AS translation | (1e+300,Infinity),(1e+300,Infinity) | (1e+300,Infinity),(1e+300,Infinity) | (1e+300,Infinity),(1e+300,Infinity) + | (Infinity,1e+300),(Infinity,1e+300) + | (Infinity,1e+300),(Infinity,1e+300) + | (Infinity,1e+300),(Infinity,1e+300) + | (Infinity,1e+300),(Infinity,1e+300) + | (Infinity,1e+300),(Infinity,1e+300) | (NaN,NaN),(NaN,NaN) | (NaN,NaN),(NaN,NaN) | (NaN,NaN),(NaN,NaN) @@ -2357,7 +2444,7 @@ SELECT '' AS twentyfour, b.f1 + p.f1 AS translation | (8,12),(2,0) | (12.5,13.5),(12.5,12.5) | (13,13),(13,13) -(45 rows) +(50 rows) SELECT '' AS twentyfour, b.f1 - p.f1 AS translation FROM BOX_TBL b, POINT_TBL p; @@ -2398,6 +2485,11 @@ SELECT '' AS twentyfour, b.f1 - p.f1 AS translation | (-1e+300,-Infinity),(-1e+300,-Infinity) | (-1e+300,-Infinity),(-1e+300,-Infinity) | (-1e+300,-Infinity),(-1e+300,-Infinity) + | (-Infinity,-1e+300),(-Infinity,-1e+300) + | (-Infinity,-1e+300),(-Infinity,-1e+300) + | (-Infinity,-1e+300),(-Infinity,-1e+300) + | (-Infinity,-1e+300),(-Infinity,-1e+300) + | (-Infinity,-1e+300),(-Infinity,-1e+300) | (NaN,NaN),(NaN,NaN) | (NaN,NaN),(NaN,NaN) | (NaN,NaN),(NaN,NaN) @@ -2408,7 +2500,7 @@ SELECT '' AS twentyfour, b.f1 - p.f1 AS translation | (-12,-8),(-18,-20) | (-7.5,-6.5),(-7.5,-7.5) | (-7,-7),(-7,-7) -(45 rows) +(50 rows) -- Multiply with point SELECT b.f1, p.f1, b.f1 * p.f1 FROM BOX_TBL b, POINT_TBL p WHERE p.f1[0] BETWEEN 1 AND 1000; @@ -2431,16 +2523,21 @@ SELECT b.f1, p.f1, b.f1 * p.f1 FROM BOX_TBL b, POINT_TBL p WHERE p.f1[0] > 1000; f1 | f1 | ?column? ---------------------+-------------------+-------------------------------------------- (2,2),(0,0) | (1e+300,Infinity) | (NaN,NaN),(-Infinity,Infinity) + (2,2),(0,0) | (Infinity,1e+300) | (NaN,NaN),(Infinity,Infinity) (2,2),(0,0) | (NaN,NaN) | (NaN,NaN),(NaN,NaN) (3,3),(1,1) | (1e+300,Infinity) | (-Infinity,Infinity),(-Infinity,Infinity) + (3,3),(1,1) | (Infinity,1e+300) | (Infinity,Infinity),(Infinity,Infinity) (3,3),(1,1) | (NaN,NaN) | (NaN,NaN),(NaN,NaN) (-2,2),(-8,-10) | (1e+300,Infinity) | (Infinity,-Infinity),(-Infinity,-Infinity) + (-2,2),(-8,-10) | (Infinity,1e+300) | (-Infinity,Infinity),(-Infinity,-Infinity) (-2,2),(-8,-10) | (NaN,NaN) | (NaN,NaN),(NaN,NaN) (2.5,3.5),(2.5,2.5) | (1e+300,Infinity) | (-Infinity,Infinity),(-Infinity,Infinity) + (2.5,3.5),(2.5,2.5) | (Infinity,1e+300) | (Infinity,Infinity),(Infinity,Infinity) (2.5,3.5),(2.5,2.5) | (NaN,NaN) | (NaN,NaN),(NaN,NaN) (3,3),(3,3) | (1e+300,Infinity) | (-Infinity,Infinity),(-Infinity,Infinity) + (3,3),(3,3) | (Infinity,1e+300) | (Infinity,Infinity),(Infinity,Infinity) (3,3),(3,3) | (NaN,NaN) | (NaN,NaN),(NaN,NaN) -(10 rows) +(15 rows) -- Divide by point SELECT b.f1, p.f1, b.f1 / p.f1 FROM BOX_TBL b, POINT_TBL p WHERE p.f1[0] BETWEEN 1 AND 1000; @@ -2470,9 +2567,10 @@ SELECT f1::box (-5,-12),(-5,-12) (1e-300,-1e-300),(1e-300,-1e-300) (1e+300,Infinity),(1e+300,Infinity) + (Infinity,1e+300),(Infinity,1e+300) (NaN,NaN),(NaN,NaN) (10,10),(10,10) -(9 rows) +(10 rows) SELECT bound_box(a.f1, b.f1) FROM BOX_TBL a, BOX_TBL b; @@ -3102,6 +3200,15 @@ SELECT p.f1, p1.f1, p.f1 + p1.f1 FROM PATH_TBL p, POINT_TBL p1; ((10,20)) | (1e+300,Infinity) | ((1e+300,Infinity)) [(11,12),(13,14)] | (1e+300,Infinity) | [(1e+300,Infinity),(1e+300,Infinity)] ((11,12),(13,14)) | (1e+300,Infinity) | ((1e+300,Infinity),(1e+300,Infinity)) + [(1,2),(3,4)] | (Infinity,1e+300) | [(Infinity,1e+300),(Infinity,1e+300)] + ((1,2),(3,4)) | (Infinity,1e+300) | ((Infinity,1e+300),(Infinity,1e+300)) + [(0,0),(3,0),(4,5),(1,6)] | (Infinity,1e+300) | [(Infinity,1e+300),(Infinity,1e+300),(Infinity,1e+300),(Infinity,1e+300)] + ((1,2),(3,4)) | (Infinity,1e+300) | ((Infinity,1e+300),(Infinity,1e+300)) + ((1,2),(3,4)) | (Infinity,1e+300) | ((Infinity,1e+300),(Infinity,1e+300)) + [(1,2),(3,4)] | (Infinity,1e+300) | [(Infinity,1e+300),(Infinity,1e+300)] + ((10,20)) | (Infinity,1e+300) | ((Infinity,1e+300)) + [(11,12),(13,14)] | (Infinity,1e+300) | [(Infinity,1e+300),(Infinity,1e+300)] + ((11,12),(13,14)) | (Infinity,1e+300) | ((Infinity,1e+300),(Infinity,1e+300)) [(1,2),(3,4)] | (NaN,NaN) | [(NaN,NaN),(NaN,NaN)] ((1,2),(3,4)) | (NaN,NaN) | ((NaN,NaN),(NaN,NaN)) [(0,0),(3,0),(4,5),(1,6)] | (NaN,NaN) | [(NaN,NaN),(NaN,NaN),(NaN,NaN),(NaN,NaN)] @@ -3120,7 +3227,7 @@ SELECT p.f1, p1.f1, p.f1 + p1.f1 FROM PATH_TBL p, POINT_TBL p1; ((10,20)) | (10,10) | ((20,30)) [(11,12),(13,14)] | (10,10) | [(21,22),(23,24)] ((11,12),(13,14)) | (10,10) | ((21,22),(23,24)) -(81 rows) +(90 rows) -- Subtract point SELECT p.f1, p1.f1, p.f1 - p1.f1 FROM PATH_TBL p, POINT_TBL p1; @@ -3189,6 +3296,15 @@ SELECT p.f1, p1.f1, p.f1 - p1.f1 FROM PATH_TBL p, POINT_TBL p1; ((10,20)) | (1e+300,Infinity) | ((-1e+300,-Infinity)) [(11,12),(13,14)] | (1e+300,Infinity) | [(-1e+300,-Infinity),(-1e+300,-Infinity)] ((11,12),(13,14)) | (1e+300,Infinity) | ((-1e+300,-Infinity),(-1e+300,-Infinity)) + [(1,2),(3,4)] | (Infinity,1e+300) | [(-Infinity,-1e+300),(-Infinity,-1e+300)] + ((1,2),(3,4)) | (Infinity,1e+300) | ((-Infinity,-1e+300),(-Infinity,-1e+300)) + [(0,0),(3,0),(4,5),(1,6)] | (Infinity,1e+300) | [(-Infinity,-1e+300),(-Infinity,-1e+300),(-Infinity,-1e+300),(-Infinity,-1e+300)] + ((1,2),(3,4)) | (Infinity,1e+300) | ((-Infinity,-1e+300),(-Infinity,-1e+300)) + ((1,2),(3,4)) | (Infinity,1e+300) | ((-Infinity,-1e+300),(-Infinity,-1e+300)) + [(1,2),(3,4)] | (Infinity,1e+300) | [(-Infinity,-1e+300),(-Infinity,-1e+300)] + ((10,20)) | (Infinity,1e+300) | ((-Infinity,-1e+300)) + [(11,12),(13,14)] | (Infinity,1e+300) | [(-Infinity,-1e+300),(-Infinity,-1e+300)] + ((11,12),(13,14)) | (Infinity,1e+300) | ((-Infinity,-1e+300),(-Infinity,-1e+300)) [(1,2),(3,4)] | (NaN,NaN) | [(NaN,NaN),(NaN,NaN)] ((1,2),(3,4)) | (NaN,NaN) | ((NaN,NaN),(NaN,NaN)) [(0,0),(3,0),(4,5),(1,6)] | (NaN,NaN) | [(NaN,NaN),(NaN,NaN),(NaN,NaN),(NaN,NaN)] @@ -3207,7 +3323,7 @@ SELECT p.f1, p1.f1, p.f1 - p1.f1 FROM PATH_TBL p, POINT_TBL p1; ((10,20)) | (10,10) | ((0,10)) [(11,12),(13,14)] | (10,10) | [(1,2),(3,4)] ((11,12),(13,14)) | (10,10) | ((1,2),(3,4)) -(81 rows) +(90 rows) -- Multiply with point SELECT p.f1, p1.f1, p.f1 * p1.f1 FROM PATH_TBL p, POINT_TBL p1; @@ -3276,6 +3392,15 @@ SELECT p.f1, p1.f1, p.f1 * p1.f1 FROM PATH_TBL p, POINT_TBL p1; ((10,20)) | (1e+300,Infinity) | ((-Infinity,Infinity)) [(11,12),(13,14)] | (1e+300,Infinity) | [(-Infinity,Infinity),(-Infinity,Infinity)] ((11,12),(13,14)) | (1e+300,Infinity) | ((-Infinity,Infinity),(-Infinity,Infinity)) + [(1,2),(3,4)] | (Infinity,1e+300) | [(Infinity,Infinity),(Infinity,Infinity)] + ((1,2),(3,4)) | (Infinity,1e+300) | ((Infinity,Infinity),(Infinity,Infinity)) + [(0,0),(3,0),(4,5),(1,6)] | (Infinity,1e+300) | [(NaN,NaN),(Infinity,NaN),(Infinity,Infinity),(Infinity,Infinity)] + ((1,2),(3,4)) | (Infinity,1e+300) | ((Infinity,Infinity),(Infinity,Infinity)) + ((1,2),(3,4)) | (Infinity,1e+300) | ((Infinity,Infinity),(Infinity,Infinity)) + [(1,2),(3,4)] | (Infinity,1e+300) | [(Infinity,Infinity),(Infinity,Infinity)] + ((10,20)) | (Infinity,1e+300) | ((Infinity,Infinity)) + [(11,12),(13,14)] | (Infinity,1e+300) | [(Infinity,Infinity),(Infinity,Infinity)] + ((11,12),(13,14)) | (Infinity,1e+300) | ((Infinity,Infinity),(Infinity,Infinity)) [(1,2),(3,4)] | (NaN,NaN) | [(NaN,NaN),(NaN,NaN)] ((1,2),(3,4)) | (NaN,NaN) | ((NaN,NaN),(NaN,NaN)) [(0,0),(3,0),(4,5),(1,6)] | (NaN,NaN) | [(NaN,NaN),(NaN,NaN),(NaN,NaN),(NaN,NaN)] @@ -3294,7 +3419,7 @@ SELECT p.f1, p1.f1, p.f1 * p1.f1 FROM PATH_TBL p, POINT_TBL p1; ((10,20)) | (10,10) | ((-100,300)) [(11,12),(13,14)] | (10,10) | [(-10,230),(-10,270)] ((11,12),(13,14)) | (10,10) | ((-10,230),(-10,270)) -(81 rows) +(90 rows) -- Divide by point SELECT p.f1, p1.f1, p.f1 / p1.f1 FROM PATH_TBL p, POINT_TBL p1 WHERE p1.f1[0] BETWEEN 1 AND 1000; @@ -3467,13 +3592,20 @@ SELECT '' AS twentyfour, p.f1, poly.f1, poly.f1 @> p.f1 AS contains | (1e+300,Infinity) | ((1,2),(7,8),(5,6),(3,-4)) | f | (1e+300,Infinity) | ((0,0)) | f | (1e+300,Infinity) | ((0,1),(0,1)) | f - | (NaN,NaN) | ((2,0),(2,4),(0,0)) | t - | (NaN,NaN) | ((3,1),(3,3),(1,0)) | t - | (NaN,NaN) | ((1,2),(3,4),(5,6),(7,8)) | t - | (NaN,NaN) | ((7,8),(5,6),(3,4),(1,2)) | t - | (NaN,NaN) | ((1,2),(7,8),(5,6),(3,-4)) | t - | (NaN,NaN) | ((0,0)) | t - | (NaN,NaN) | ((0,1),(0,1)) | t + | (Infinity,1e+300) | ((2,0),(2,4),(0,0)) | f + | (Infinity,1e+300) | ((3,1),(3,3),(1,0)) | f + | (Infinity,1e+300) | ((1,2),(3,4),(5,6),(7,8)) | f + | (Infinity,1e+300) | ((7,8),(5,6),(3,4),(1,2)) | f + | (Infinity,1e+300) | ((1,2),(7,8),(5,6),(3,-4)) | f + | (Infinity,1e+300) | ((0,0)) | f + | (Infinity,1e+300) | ((0,1),(0,1)) | f + | (NaN,NaN) | ((2,0),(2,4),(0,0)) | f + | (NaN,NaN) | ((3,1),(3,3),(1,0)) | f + | (NaN,NaN) | ((1,2),(3,4),(5,6),(7,8)) | f + | (NaN,NaN) | ((7,8),(5,6),(3,4),(1,2)) | f + | (NaN,NaN) | ((1,2),(7,8),(5,6),(3,-4)) | f + | (NaN,NaN) | ((0,0)) | f + | (NaN,NaN) | ((0,1),(0,1)) | f | (10,10) | ((2,0),(2,4),(0,0)) | f | (10,10) | ((3,1),(3,3),(1,0)) | f | (10,10) | ((1,2),(3,4),(5,6),(7,8)) | f @@ -3481,7 +3613,7 @@ SELECT '' AS twentyfour, p.f1, poly.f1, poly.f1 @> p.f1 AS contains | (10,10) | ((1,2),(7,8),(5,6),(3,-4)) | f | (10,10) | ((0,0)) | f | (10,10) | ((0,1),(0,1)) | f -(63 rows) +(70 rows) SELECT '' AS twentyfour, p.f1, poly.f1, p.f1 <@ poly.f1 AS contained FROM POLYGON_TBL poly, POINT_TBL p; @@ -3536,13 +3668,20 @@ SELECT '' AS twentyfour, p.f1, poly.f1, p.f1 <@ poly.f1 AS contained | (1e+300,Infinity) | ((1,2),(7,8),(5,6),(3,-4)) | f | (1e+300,Infinity) | ((0,0)) | f | (1e+300,Infinity) | ((0,1),(0,1)) | f - | (NaN,NaN) | ((2,0),(2,4),(0,0)) | t - | (NaN,NaN) | ((3,1),(3,3),(1,0)) | t - | (NaN,NaN) | ((1,2),(3,4),(5,6),(7,8)) | t - | (NaN,NaN) | ((7,8),(5,6),(3,4),(1,2)) | t - | (NaN,NaN) | ((1,2),(7,8),(5,6),(3,-4)) | t - | (NaN,NaN) | ((0,0)) | t - | (NaN,NaN) | ((0,1),(0,1)) | t + | (Infinity,1e+300) | ((2,0),(2,4),(0,0)) | f + | (Infinity,1e+300) | ((3,1),(3,3),(1,0)) | f + | (Infinity,1e+300) | ((1,2),(3,4),(5,6),(7,8)) | f + | (Infinity,1e+300) | ((7,8),(5,6),(3,4),(1,2)) | f + | (Infinity,1e+300) | ((1,2),(7,8),(5,6),(3,-4)) | f + | (Infinity,1e+300) | ((0,0)) | f + | (Infinity,1e+300) | ((0,1),(0,1)) | f + | (NaN,NaN) | ((2,0),(2,4),(0,0)) | f + | (NaN,NaN) | ((3,1),(3,3),(1,0)) | f + | (NaN,NaN) | ((1,2),(3,4),(5,6),(7,8)) | f + | (NaN,NaN) | ((7,8),(5,6),(3,4),(1,2)) | f + | (NaN,NaN) | ((1,2),(7,8),(5,6),(3,-4)) | f + | (NaN,NaN) | ((0,0)) | f + | (NaN,NaN) | ((0,1),(0,1)) | f | (10,10) | ((2,0),(2,4),(0,0)) | f | (10,10) | ((3,1),(3,3),(1,0)) | f | (10,10) | ((1,2),(3,4),(5,6),(7,8)) | f @@ -3550,7 +3689,7 @@ SELECT '' AS twentyfour, p.f1, poly.f1, p.f1 <@ poly.f1 AS contained | (10,10) | ((1,2),(7,8),(5,6),(3,-4)) | f | (10,10) | ((0,0)) | f | (10,10) | ((0,1),(0,1)) | f -(63 rows) +(70 rows) SELECT '' AS four, npoints(f1) AS npoints, f1 AS polygon FROM POLYGON_TBL; @@ -3929,9 +4068,10 @@ SELECT '' AS six, circle(f1, 50.0) | <(-5,-12),50> | <(1e-300,-1e-300),50> | <(1e+300,Infinity),50> + | <(Infinity,1e+300),50> | <(NaN,NaN),50> | <(10,10),50> -(9 rows) +(10 rows) SELECT '' AS four, circle(f1) FROM BOX_TBL; @@ -3993,12 +4133,19 @@ SELECT '' AS twentyfour, c1.f1 AS circle, p1.f1 AS point, (p1.f1 <-> c1.f1) AS d | <(100,200),10> | (-10,0) | 218.25424421 | <(100,200),10> | (-5,-12) | 226.577682802 | <(3,5),0> | (1e+300,Infinity) | Infinity + | <(3,5),0> | (Infinity,1e+300) | Infinity | <(1,2),3> | (1e+300,Infinity) | Infinity | <(5,1),3> | (1e+300,Infinity) | Infinity + | <(5,1),3> | (Infinity,1e+300) | Infinity + | <(1,2),3> | (Infinity,1e+300) | Infinity | <(1,3),5> | (1e+300,Infinity) | Infinity + | <(1,3),5> | (Infinity,1e+300) | Infinity | <(100,200),10> | (1e+300,Infinity) | Infinity + | <(100,200),10> | (Infinity,1e+300) | Infinity | <(1,2),100> | (1e+300,Infinity) | Infinity + | <(1,2),100> | (Infinity,1e+300) | Infinity | <(100,1),115> | (1e+300,Infinity) | Infinity + | <(100,1),115> | (Infinity,1e+300) | Infinity | <(3,5),0> | (NaN,NaN) | NaN | <(1,2),3> | (NaN,NaN) | NaN | <(5,1),3> | (NaN,NaN) | NaN @@ -4014,8 +4161,9 @@ SELECT '' AS twentyfour, c1.f1 AS circle, p1.f1 AS point, (p1.f1 <-> c1.f1) AS d | <(3,5),NaN> | (5.1,34.5) | NaN | <(3,5),NaN> | (10,10) | NaN | <(3,5),NaN> | (1e+300,Infinity) | NaN + | <(3,5),NaN> | (Infinity,1e+300) | NaN | <(3,5),NaN> | (NaN,NaN) | NaN -(53 rows) +(61 rows) -- To polygon SELECT f1, f1::polygon FROM CIRCLE_TBL WHERE f1 >= '<(0,0),1>'; @@ -4626,6 +4774,14 @@ SELECT c.f1, p.f1, c.f1 + p.f1 FROM CIRCLE_TBL c, POINT_TBL p; <(100,1),115> | (1e+300,Infinity) | <(1e+300,Infinity),115> <(3,5),0> | (1e+300,Infinity) | <(1e+300,Infinity),0> <(3,5),NaN> | (1e+300,Infinity) | <(1e+300,Infinity),NaN> + <(5,1),3> | (Infinity,1e+300) | <(Infinity,1e+300),3> + <(1,2),100> | (Infinity,1e+300) | <(Infinity,1e+300),100> + <(1,3),5> | (Infinity,1e+300) | <(Infinity,1e+300),5> + <(1,2),3> | (Infinity,1e+300) | <(Infinity,1e+300),3> + <(100,200),10> | (Infinity,1e+300) | <(Infinity,1e+300),10> + <(100,1),115> | (Infinity,1e+300) | <(Infinity,1e+300),115> + <(3,5),0> | (Infinity,1e+300) | <(Infinity,1e+300),0> + <(3,5),NaN> | (Infinity,1e+300) | <(Infinity,1e+300),NaN> <(5,1),3> | (NaN,NaN) | <(NaN,NaN),3> <(1,2),100> | (NaN,NaN) | <(NaN,NaN),100> <(1,3),5> | (NaN,NaN) | <(NaN,NaN),5> @@ -4642,7 +4798,7 @@ SELECT c.f1, p.f1, c.f1 + p.f1 FROM CIRCLE_TBL c, POINT_TBL p; <(100,1),115> | (10,10) | <(110,11),115> <(3,5),0> | (10,10) | <(13,15),0> <(3,5),NaN> | (10,10) | <(13,15),NaN> -(72 rows) +(80 rows) -- Subtract point SELECT c.f1, p.f1, c.f1 - p.f1 FROM CIRCLE_TBL c, POINT_TBL p; @@ -4704,6 +4860,14 @@ SELECT c.f1, p.f1, c.f1 - p.f1 FROM CIRCLE_TBL c, POINT_TBL p; <(100,1),115> | (1e+300,Infinity) | <(-1e+300,-Infinity),115> <(3,5),0> | (1e+300,Infinity) | <(-1e+300,-Infinity),0> <(3,5),NaN> | (1e+300,Infinity) | <(-1e+300,-Infinity),NaN> + <(5,1),3> | (Infinity,1e+300) | <(-Infinity,-1e+300),3> + <(1,2),100> | (Infinity,1e+300) | <(-Infinity,-1e+300),100> + <(1,3),5> | (Infinity,1e+300) | <(-Infinity,-1e+300),5> + <(1,2),3> | (Infinity,1e+300) | <(-Infinity,-1e+300),3> + <(100,200),10> | (Infinity,1e+300) | <(-Infinity,-1e+300),10> + <(100,1),115> | (Infinity,1e+300) | <(-Infinity,-1e+300),115> + <(3,5),0> | (Infinity,1e+300) | <(-Infinity,-1e+300),0> + <(3,5),NaN> | (Infinity,1e+300) | <(-Infinity,-1e+300),NaN> <(5,1),3> | (NaN,NaN) | <(NaN,NaN),3> <(1,2),100> | (NaN,NaN) | <(NaN,NaN),100> <(1,3),5> | (NaN,NaN) | <(NaN,NaN),5> @@ -4720,7 +4884,7 @@ SELECT c.f1, p.f1, c.f1 - p.f1 FROM CIRCLE_TBL c, POINT_TBL p; <(100,1),115> | (10,10) | <(90,-9),115> <(3,5),0> | (10,10) | <(-7,-5),0> <(3,5),NaN> | (10,10) | <(-7,-5),NaN> -(72 rows) +(80 rows) -- Multiply with point SELECT c.f1, p.f1, c.f1 * p.f1 FROM CIRCLE_TBL c, POINT_TBL p; @@ -4782,6 +4946,14 @@ SELECT c.f1, p.f1, c.f1 * p.f1 FROM CIRCLE_TBL c, POINT_TBL p; <(100,1),115> | (1e+300,Infinity) | <(-Infinity,Infinity),Infinity> <(3,5),0> | (1e+300,Infinity) | <(-Infinity,Infinity),NaN> <(3,5),NaN> | (1e+300,Infinity) | <(-Infinity,Infinity),NaN> + <(5,1),3> | (Infinity,1e+300) | <(Infinity,Infinity),Infinity> + <(1,2),100> | (Infinity,1e+300) | <(Infinity,Infinity),Infinity> + <(1,3),5> | (Infinity,1e+300) | <(Infinity,Infinity),Infinity> + <(1,2),3> | (Infinity,1e+300) | <(Infinity,Infinity),Infinity> + <(100,200),10> | (Infinity,1e+300) | <(Infinity,Infinity),Infinity> + <(100,1),115> | (Infinity,1e+300) | <(Infinity,Infinity),Infinity> + <(3,5),0> | (Infinity,1e+300) | <(Infinity,Infinity),NaN> + <(3,5),NaN> | (Infinity,1e+300) | <(Infinity,Infinity),NaN> <(5,1),3> | (NaN,NaN) | <(NaN,NaN),NaN> <(1,2),100> | (NaN,NaN) | <(NaN,NaN),NaN> <(1,3),5> | (NaN,NaN) | <(NaN,NaN),NaN> @@ -4798,7 +4970,7 @@ SELECT c.f1, p.f1, c.f1 * p.f1 FROM CIRCLE_TBL c, POINT_TBL p; <(100,1),115> | (10,10) | <(990,1010),1626.34559673> <(3,5),0> | (10,10) | <(-20,80),0> <(3,5),NaN> | (10,10) | <(-20,80),NaN> -(72 rows) +(80 rows) -- Divide by point SELECT c.f1, p.f1, c.f1 / p.f1 FROM CIRCLE_TBL c, POINT_TBL p WHERE p.f1[0] BETWEEN 1 AND 1000; diff --git a/src/test/regress/expected/point.out b/src/test/regress/expected/point.out index 15e3b83b37..77e250fc3e 100644 --- a/src/test/regress/expected/point.out +++ b/src/test/regress/expected/point.out @@ -11,6 +11,7 @@ INSERT INTO POINT_TBL(f1) VALUES ('(5.1, 34.5)'); INSERT INTO POINT_TBL(f1) VALUES ('(-5.0,-12.0)'); INSERT INTO POINT_TBL(f1) VALUES ('(1e-300,-1e-300)'); -- To underflow INSERT INTO POINT_TBL(f1) VALUES ('(1e+300,Inf)'); -- To overflow +INSERT INTO POINT_TBL(f1) VALUES ('(Inf,1e+300)'); -- Transposed INSERT INTO POINT_TBL(f1) VALUES (' ( Nan , NaN ) '); -- bad format points INSERT INTO POINT_TBL(f1) VALUES ('asdfasdf'); @@ -44,9 +45,10 @@ SELECT '' AS six, * FROM POINT_TBL; | (-5,-12) | (1e-300,-1e-300) | (1e+300,Infinity) + | (Infinity,1e+300) | (NaN,NaN) | (10,10) -(9 rows) +(10 rows) -- left of SELECT '' AS three, p.* FROM POINT_TBL p WHERE p.f1 << '(0.0, 0.0)'; @@ -115,8 +117,9 @@ SELECT '' AS three, p.* FROM POINT_TBL p | (-5,-12) | (1e-300,-1e-300) | (1e+300,Infinity) + | (Infinity,1e+300) | (NaN,NaN) -(6 rows) +(7 rows) SELECT '' AS two, p.* FROM POINT_TBL p WHERE p.f1 <@ path '[(0,0),(-10,0),(-10,10)]'; @@ -136,8 +139,9 @@ SELECT '' AS three, p.* FROM POINT_TBL p | (-5,-12) | (1e-300,-1e-300) | (1e+300,Infinity) + | (Infinity,1e+300) | (NaN,NaN) -(6 rows) +(7 rows) SELECT '' AS six, p.f1, p.f1 <-> point '(0,0)' AS dist FROM POINT_TBL p @@ -152,8 +156,9 @@ SELECT '' AS six, p.f1, p.f1 <-> point '(0,0)' AS dist | (10,10) | 14.142135623731 | (5.1,34.5) | 34.8749193547455 | (1e+300,Infinity) | Infinity + | (Infinity,1e+300) | Infinity | (NaN,NaN) | NaN -(9 rows) +(10 rows) SELECT '' AS thirtysix, p1.f1 AS point1, p2.f1 AS point2, p1.f1 <-> p2.f1 AS dist FROM POINT_TBL p1, POINT_TBL p2 @@ -210,12 +215,19 @@ SELECT '' AS thirtysix, p1.f1 AS point1, p2.f1 AS point2, p1.f1 <-> p2.f1 AS dis | (-5,-12) | (5.1,34.5) | 47.5842410888311 | (5.1,34.5) | (-5,-12) | 47.5842410888311 | (-10,0) | (1e+300,Infinity) | Infinity + | (-10,0) | (Infinity,1e+300) | Infinity | (-5,-12) | (1e+300,Infinity) | Infinity + | (-5,-12) | (Infinity,1e+300) | Infinity | (-3,4) | (1e+300,Infinity) | Infinity + | (-3,4) | (Infinity,1e+300) | Infinity | (0,0) | (1e+300,Infinity) | Infinity + | (0,0) | (Infinity,1e+300) | Infinity | (1e-300,-1e-300) | (1e+300,Infinity) | Infinity + | (1e-300,-1e-300) | (Infinity,1e+300) | Infinity | (5.1,34.5) | (1e+300,Infinity) | Infinity + | (5.1,34.5) | (Infinity,1e+300) | Infinity | (10,10) | (1e+300,Infinity) | Infinity + | (10,10) | (Infinity,1e+300) | Infinity | (1e+300,Infinity) | (-10,0) | Infinity | (1e+300,Infinity) | (-5,-12) | Infinity | (1e+300,Infinity) | (-3,4) | Infinity @@ -223,6 +235,15 @@ SELECT '' AS thirtysix, p1.f1 AS point1, p2.f1 AS point2, p1.f1 <-> p2.f1 AS dis | (1e+300,Infinity) | (1e-300,-1e-300) | Infinity | (1e+300,Infinity) | (5.1,34.5) | Infinity | (1e+300,Infinity) | (10,10) | Infinity + | (1e+300,Infinity) | (Infinity,1e+300) | Infinity + | (Infinity,1e+300) | (-10,0) | Infinity + | (Infinity,1e+300) | (-5,-12) | Infinity + | (Infinity,1e+300) | (-3,4) | Infinity + | (Infinity,1e+300) | (0,0) | Infinity + | (Infinity,1e+300) | (1e-300,-1e-300) | Infinity + | (Infinity,1e+300) | (5.1,34.5) | Infinity + | (Infinity,1e+300) | (10,10) | Infinity + | (Infinity,1e+300) | (1e+300,Infinity) | Infinity | (-10,0) | (NaN,NaN) | NaN | (-5,-12) | (NaN,NaN) | NaN | (-3,4) | (NaN,NaN) | NaN @@ -232,6 +253,8 @@ SELECT '' AS thirtysix, p1.f1 AS point1, p2.f1 AS point2, p1.f1 <-> p2.f1 AS dis | (10,10) | (NaN,NaN) | NaN | (1e+300,Infinity) | (1e+300,Infinity) | NaN | (1e+300,Infinity) | (NaN,NaN) | NaN + | (Infinity,1e+300) | (Infinity,1e+300) | NaN + | (Infinity,1e+300) | (NaN,NaN) | NaN | (NaN,NaN) | (-10,0) | NaN | (NaN,NaN) | (-5,-12) | NaN | (NaN,NaN) | (-3,4) | NaN @@ -240,8 +263,9 @@ SELECT '' AS thirtysix, p1.f1 AS point1, p2.f1 AS point2, p1.f1 <-> p2.f1 AS dis | (NaN,NaN) | (5.1,34.5) | NaN | (NaN,NaN) | (10,10) | NaN | (NaN,NaN) | (1e+300,Infinity) | NaN + | (NaN,NaN) | (Infinity,1e+300) | NaN | (NaN,NaN) | (NaN,NaN) | NaN -(81 rows) +(100 rows) SELECT '' AS thirty, p1.f1 AS point1, p2.f1 AS point2 FROM POINT_TBL p1, POINT_TBL p2 @@ -253,6 +277,7 @@ SELECT '' AS thirty, p1.f1 AS point1, p2.f1 AS point2 | (0,0) | (5.1,34.5) | (0,0) | (-5,-12) | (0,0) | (1e+300,Infinity) + | (0,0) | (Infinity,1e+300) | (0,0) | (NaN,NaN) | (0,0) | (10,10) | (-10,0) | (0,0) @@ -261,6 +286,7 @@ SELECT '' AS thirty, p1.f1 AS point1, p2.f1 AS point2 | (-10,0) | (-5,-12) | (-10,0) | (1e-300,-1e-300) | (-10,0) | (1e+300,Infinity) + | (-10,0) | (Infinity,1e+300) | (-10,0) | (NaN,NaN) | (-10,0) | (10,10) | (-3,4) | (0,0) @@ -269,6 +295,7 @@ SELECT '' AS thirty, p1.f1 AS point1, p2.f1 AS point2 | (-3,4) | (-5,-12) | (-3,4) | (1e-300,-1e-300) | (-3,4) | (1e+300,Infinity) + | (-3,4) | (Infinity,1e+300) | (-3,4) | (NaN,NaN) | (-3,4) | (10,10) | (5.1,34.5) | (0,0) @@ -277,6 +304,7 @@ SELECT '' AS thirty, p1.f1 AS point1, p2.f1 AS point2 | (5.1,34.5) | (-5,-12) | (5.1,34.5) | (1e-300,-1e-300) | (5.1,34.5) | (1e+300,Infinity) + | (5.1,34.5) | (Infinity,1e+300) | (5.1,34.5) | (NaN,NaN) | (5.1,34.5) | (10,10) | (-5,-12) | (0,0) @@ -285,6 +313,7 @@ SELECT '' AS thirty, p1.f1 AS point1, p2.f1 AS point2 | (-5,-12) | (5.1,34.5) | (-5,-12) | (1e-300,-1e-300) | (-5,-12) | (1e+300,Infinity) + | (-5,-12) | (Infinity,1e+300) | (-5,-12) | (NaN,NaN) | (-5,-12) | (10,10) | (1e-300,-1e-300) | (-10,0) @@ -292,6 +321,7 @@ SELECT '' AS thirty, p1.f1 AS point1, p2.f1 AS point2 | (1e-300,-1e-300) | (5.1,34.5) | (1e-300,-1e-300) | (-5,-12) | (1e-300,-1e-300) | (1e+300,Infinity) + | (1e-300,-1e-300) | (Infinity,1e+300) | (1e-300,-1e-300) | (NaN,NaN) | (1e-300,-1e-300) | (10,10) | (1e+300,Infinity) | (0,0) @@ -301,8 +331,19 @@ SELECT '' AS thirty, p1.f1 AS point1, p2.f1 AS point2 | (1e+300,Infinity) | (-5,-12) | (1e+300,Infinity) | (1e-300,-1e-300) | (1e+300,Infinity) | (1e+300,Infinity) + | (1e+300,Infinity) | (Infinity,1e+300) | (1e+300,Infinity) | (NaN,NaN) | (1e+300,Infinity) | (10,10) + | (Infinity,1e+300) | (0,0) + | (Infinity,1e+300) | (-10,0) + | (Infinity,1e+300) | (-3,4) + | (Infinity,1e+300) | (5.1,34.5) + | (Infinity,1e+300) | (-5,-12) + | (Infinity,1e+300) | (1e-300,-1e-300) + | (Infinity,1e+300) | (1e+300,Infinity) + | (Infinity,1e+300) | (Infinity,1e+300) + | (Infinity,1e+300) | (NaN,NaN) + | (Infinity,1e+300) | (10,10) | (NaN,NaN) | (0,0) | (NaN,NaN) | (-10,0) | (NaN,NaN) | (-3,4) @@ -310,6 +351,7 @@ SELECT '' AS thirty, p1.f1 AS point1, p2.f1 AS point2 | (NaN,NaN) | (-5,-12) | (NaN,NaN) | (1e-300,-1e-300) | (NaN,NaN) | (1e+300,Infinity) + | (NaN,NaN) | (Infinity,1e+300) | (NaN,NaN) | (NaN,NaN) | (NaN,NaN) | (10,10) | (10,10) | (0,0) @@ -319,57 +361,67 @@ SELECT '' AS thirty, p1.f1 AS point1, p2.f1 AS point2 | (10,10) | (-5,-12) | (10,10) | (1e-300,-1e-300) | (10,10) | (1e+300,Infinity) + | (10,10) | (Infinity,1e+300) | (10,10) | (NaN,NaN) -(72 rows) +(91 rows) -- put distance result into output to allow sorting with GEQ optimizer - tgl 97/05/10 SELECT '' AS fifteen, p1.f1 AS point1, p2.f1 AS point2, (p1.f1 <-> p2.f1) AS distance FROM POINT_TBL p1, POINT_TBL p2 WHERE (p1.f1 <-> p2.f1) > 3 and p1.f1 << p2.f1 ORDER BY distance, p1.f1[0], p2.f1[0]; - fifteen | point1 | point2 | distance ----------+------------------+-------------------+------------------ - | (-3,4) | (0,0) | 5 - | (-3,4) | (1e-300,-1e-300) | 5 - | (-10,0) | (-3,4) | 8.06225774829855 - | (-10,0) | (0,0) | 10 - | (-10,0) | (1e-300,-1e-300) | 10 - | (-10,0) | (-5,-12) | 13 - | (-5,-12) | (0,0) | 13 - | (-5,-12) | (1e-300,-1e-300) | 13 - | (0,0) | (10,10) | 14.142135623731 - | (1e-300,-1e-300) | (10,10) | 14.142135623731 - | (-3,4) | (10,10) | 14.3178210632764 - | (-5,-12) | (-3,4) | 16.1245154965971 - | (-10,0) | (10,10) | 22.3606797749979 - | (5.1,34.5) | (10,10) | 24.9851956166046 - | (-5,-12) | (10,10) | 26.6270539113887 - | (-3,4) | (5.1,34.5) | 31.5572495632937 - | (0,0) | (5.1,34.5) | 34.8749193547455 - | (1e-300,-1e-300) | (5.1,34.5) | 34.8749193547455 - | (-10,0) | (5.1,34.5) | 37.6597928831267 - | (-5,-12) | (5.1,34.5) | 47.5842410888311 - | (-10,0) | (1e+300,Infinity) | Infinity - | (-5,-12) | (1e+300,Infinity) | Infinity - | (-3,4) | (1e+300,Infinity) | Infinity - | (0,0) | (1e+300,Infinity) | Infinity - | (1e-300,-1e-300) | (1e+300,Infinity) | Infinity - | (5.1,34.5) | (1e+300,Infinity) | Infinity - | (10,10) | (1e+300,Infinity) | Infinity -(27 rows) + fifteen | point1 | point2 | distance +---------+-------------------+-------------------+------------------ + | (-3,4) | (0,0) | 5 + | (-3,4) | (1e-300,-1e-300) | 5 + | (-10,0) | (-3,4) | 8.06225774829855 + | (-10,0) | (0,0) | 10 + | (-10,0) | (1e-300,-1e-300) | 10 + | (-10,0) | (-5,-12) | 13 + | (-5,-12) | (0,0) | 13 + | (-5,-12) | (1e-300,-1e-300) | 13 + | (0,0) | (10,10) | 14.142135623731 + | (1e-300,-1e-300) | (10,10) | 14.142135623731 + | (-3,4) | (10,10) | 14.3178210632764 + | (-5,-12) | (-3,4) | 16.1245154965971 + | (-10,0) | (10,10) | 22.3606797749979 + | (5.1,34.5) | (10,10) | 24.9851956166046 + | (-5,-12) | (10,10) | 26.6270539113887 + | (-3,4) | (5.1,34.5) | 31.5572495632937 + | (0,0) | (5.1,34.5) | 34.8749193547455 + | (1e-300,-1e-300) | (5.1,34.5) | 34.8749193547455 + | (-10,0) | (5.1,34.5) | 37.6597928831267 + | (-5,-12) | (5.1,34.5) | 47.5842410888311 + | (-10,0) | (1e+300,Infinity) | Infinity + | (-10,0) | (Infinity,1e+300) | Infinity + | (-5,-12) | (1e+300,Infinity) | Infinity + | (-5,-12) | (Infinity,1e+300) | Infinity + | (-3,4) | (1e+300,Infinity) | Infinity + | (-3,4) | (Infinity,1e+300) | Infinity + | (0,0) | (1e+300,Infinity) | Infinity + | (0,0) | (Infinity,1e+300) | Infinity + | (1e-300,-1e-300) | (1e+300,Infinity) | Infinity + | (1e-300,-1e-300) | (Infinity,1e+300) | Infinity + | (5.1,34.5) | (1e+300,Infinity) | Infinity + | (5.1,34.5) | (Infinity,1e+300) | Infinity + | (10,10) | (1e+300,Infinity) | Infinity + | (10,10) | (Infinity,1e+300) | Infinity + | (1e+300,Infinity) | (Infinity,1e+300) | Infinity +(35 rows) -- put distance result into output to allow sorting with GEQ optimizer - tgl 97/05/10 SELECT '' AS three, p1.f1 AS point1, p2.f1 AS point2, (p1.f1 <-> p2.f1) AS distance FROM POINT_TBL p1, POINT_TBL p2 WHERE (p1.f1 <-> p2.f1) > 3 and p1.f1 << p2.f1 and p1.f1 >^ p2.f1 ORDER BY distance; - three | point1 | point2 | distance --------+------------+------------------+------------------ - | (-3,4) | (0,0) | 5 - | (-3,4) | (1e-300,-1e-300) | 5 - | (-10,0) | (-5,-12) | 13 - | (5.1,34.5) | (10,10) | 24.9851956166046 -(4 rows) + three | point1 | point2 | distance +-------+-------------------+-------------------+------------------ + | (-3,4) | (0,0) | 5 + | (-3,4) | (1e-300,-1e-300) | 5 + | (-10,0) | (-5,-12) | 13 + | (5.1,34.5) | (10,10) | 24.9851956166046 + | (1e+300,Infinity) | (Infinity,1e+300) | Infinity +(5 rows) -- Test that GiST indexes provide same behavior as sequential scan CREATE TEMP TABLE point_gist_tbl(f1 point); diff --git a/src/test/regress/sql/point.sql b/src/test/regress/sql/point.sql index 7f8504dbd1..6a1ca12d5c 100644 --- a/src/test/regress/sql/point.sql +++ b/src/test/regress/sql/point.sql @@ -21,6 +21,8 @@ INSERT INTO POINT_TBL(f1) VALUES ('(1e-300,-1e-300)'); -- To underflow INSERT INTO POINT_TBL(f1) VALUES ('(1e+300,Inf)'); -- To overflow +INSERT INTO POINT_TBL(f1) VALUES ('(Inf,1e+300)'); -- Transposed + INSERT INTO POINT_TBL(f1) VALUES (' ( Nan , NaN ) '); -- bad format points -- 2.18.4 ----Next_Part(Thu_Sep_10_18_37_09_2020_527)---- ^ permalink raw reply [nested|flat] 66+ messages in thread
* Re: Perform streaming logical transactions by background workers and parallel apply @ 2022-04-19 06:57 Amit Kapila <[email protected]> 0 siblings, 1 reply; 66+ messages in thread From: Amit Kapila @ 2022-04-19 06:57 UTC (permalink / raw) To: [email protected] <[email protected]>; +Cc: [email protected] <[email protected]>; PostgreSQL Hackers <[email protected]> On Thu, Apr 14, 2022 at 9:12 AM [email protected] <[email protected]> wrote: > > On Friday, April 8, 2022 5:14 PM [email protected] <[email protected]> wrote: > > Attach a new version patch which improved the error handling and handled the case > when there is no more worker available (will spill the data to the temp file in this case). > > Currently, it still doesn't support skip the streamed transaction in bgworker, because > in this approach, we don't know the last lsn for the streamed transaction being applied, > so cannot get the lsn to SKIP. I will think more about it and keep testing the patch. > I think we can avoid performing the streaming transaction by bgworker if skip_lsn is set. This needs some more thought but anyway I see another problem in this patch. I think we won't be able to make the decision whether to apply the change for a relation that is not in the 'READY' state (see should_apply_changes_for_rel) as we won't know 'remote_final_lsn' by that time for streaming transactions. I think what we can do here is that before assigning the transaction to bgworker, we can check if any of the rels is not in the 'READY' state, we can make the transaction spill the changes as we are doing now. Even if we do such a check, it is still possible that some rel on which this transaction is performing operation can appear to be in 'non-ready' state after starting bgworker and for such a case I think we need to give error and restart the transaction as we have no way to know whether we need to perform an operation on the 'rel'. This is possible if the user performs REFRESH PUBLICATION in parallel to this transaction as that can add a new rel to the pg_subscription_rel. -- With Regards, Amit Kapila. ^ permalink raw reply [nested|flat] 66+ messages in thread
* RE: Perform streaming logical transactions by background workers and parallel apply @ 2022-04-20 08:57 [email protected] <[email protected]> parent: Amit Kapila <[email protected]> 0 siblings, 1 reply; 66+ messages in thread From: [email protected] @ 2022-04-20 08:57 UTC (permalink / raw) To: Amit Kapila <[email protected]>; +Cc: [email protected] <[email protected]>; PostgreSQL Hackers <[email protected]> On Tuesday, April 19, 2022 2:58 PM Amit Kapila <[email protected]> wrote: > > On Thu, Apr 14, 2022 at 9:12 AM [email protected] > <[email protected]> wrote: > > > > On Friday, April 8, 2022 5:14 PM [email protected] > <[email protected]> wrote: > > > > Attach a new version patch which improved the error handling and handled > the case > > when there is no more worker available (will spill the data to the temp file in > this case). > > > > Currently, it still doesn't support skip the streamed transaction in bgworker, > because > > in this approach, we don't know the last lsn for the streamed transaction > being applied, > > so cannot get the lsn to SKIP. I will think more about it and keep testing the > patch. > > > > I think we can avoid performing the streaming transaction by bgworker > if skip_lsn is set. This needs some more thought but anyway I see > another problem in this patch. I think we won't be able to make the > decision whether to apply the change for a relation that is not in the > 'READY' state (see should_apply_changes_for_rel) as we won't know > 'remote_final_lsn' by that time for streaming transactions. I think > what we can do here is that before assigning the transaction to > bgworker, we can check if any of the rels is not in the 'READY' state, > we can make the transaction spill the changes as we are doing now. > Even if we do such a check, it is still possible that some rel on > which this transaction is performing operation can appear to be in > 'non-ready' state after starting bgworker and for such a case I think > we need to give error and restart the transaction as we have no way to > know whether we need to perform an operation on the 'rel'. This is > possible if the user performs REFRESH PUBLICATION in parallel to this > transaction as that can add a new rel to the pg_subscription_rel. Changed as suggested. Attach the new version patch which cleanup some code and fix above problem. For now, it won't apply streaming transaction in bgworker if skiplsn is set or any table is not in 'READY' state. Besides, extent the subscription streaming option to ('on/off/apply(apply in bgworker)/spool(spool to file)') so that user can control whether to apply The transaction in a bgworker. Best regards, Hou zj Attachments: [application/octet-stream] v3-0001-Perform-streaming-logical-transactions-by-background.patch (76.7K, ../../OS0PR01MB571616192F70A18AF88F3B2894F59@OS0PR01MB5716.jpnprd01.prod.outlook.com/2-v3-0001-Perform-streaming-logical-transactions-by-background.patch) download | inline diff: From 7fac0f2d35591155493fec637ec1df3c2b7239e0 Mon Sep 17 00:00:00 2001 From: "houzj.fnst" <[email protected]> Date: Wed, 20 Apr 2022 16:45:07 +0800 Subject: [PATCH] Perform streaming logical transactions by background workers Currently, for large transactions, the publisher sends the data in multiple streams (changes divided into chunks depending upon logical_decoding_work_mem), and then on the subscriber-side, the apply worker writes the changes into temporary files and once it receives the commit, it read from the file and apply the entire transaction. To improve the performance of such transactions, we can instead allow them to be applied via background workers. In this approach, we assign a new bgworker (if available) as soon as the xact's first stream came and the main apply worker will send changes to this new worker via shared memory. The bgworker will directly apply the change instead of writing it to temporary files. We keep this worker assigned till the transaction commit came and also wait for the worker to finish at commit. This preserves commit ordering and avoid writing to and reading from file in most cases. We still need to spill if there is no worker available. We also need to allow stream_stop to complete by the background worker to finish it to avoid deadlocks because T-1's current stream of changes can update rows in conflicting order with T-2's next stream of changes. Also extend the subscription streaming option so that user can control whether apply the streaming transaction in a bgworker or spill the change to disk. User can set the streaming option to 'on/off', 'apply', 'spool'. For now, 'on' and 'apply' means the streaming will be applied via a bgworker if available. 'spool' means the streaming transaction will be spilled to disk. --- doc/src/sgml/ref/create_subscription.sgml | 24 +- src/backend/commands/subscriptioncmds.c | 70 +- src/backend/postmaster/bgworker.c | 3 + src/backend/replication/logical/launcher.c | 72 +- src/backend/replication/logical/origin.c | 10 +- src/backend/replication/logical/proto.c | 7 +- src/backend/replication/logical/tablesync.c | 10 +- src/backend/replication/logical/worker.c | 1532 ++++++++++++++++++++++++--- src/backend/utils/activity/wait_event.c | 3 + src/bin/pg_dump/pg_dump.c | 8 +- src/include/catalog/pg_subscription.h | 16 +- src/include/replication/logicalproto.h | 4 +- src/include/replication/logicalworker.h | 1 + src/include/replication/origin.h | 2 +- src/include/replication/worker_internal.h | 7 +- src/include/utils/wait_event.h | 1 + src/test/subscription/t/029_on_error.pl | 2 +- 17 files changed, 1564 insertions(+), 208 deletions(-) diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml index 203bb41..0b4eb64 100644 --- a/doc/src/sgml/ref/create_subscription.sgml +++ b/doc/src/sgml/ref/create_subscription.sgml @@ -217,13 +217,29 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl </varlistentry> <varlistentry> - <term><literal>streaming</literal> (<type>boolean</type>)</term> + <term><literal>streaming</literal> (<type>enum</type>)</term> <listitem> <para> Specifies whether to enable streaming of in-progress transactions - for this subscription. By default, all transactions - are fully decoded on the publisher and only then sent to the - subscriber as a whole. + for this subscription. The default value is <literal>off</literal>, + all transactions are fully decoded on the publisher and only then + sent to the subscriber as a whole. + </para> + + <para> + If set to <literal>spool</literal> or <literal>on</literal>, the changes + of transaction are written to temporary files and then applied at + once after the transaction is committed on the publisher. + </para> + + <para> + If set to <literal>apply</literal>, incoming changes are directly + applied via one of the background worker, if available. If no + background worker is free to handle streaming transaction then the + changes are written to a file and applied after the transaction is + committed. Note that if error happen when applying changes in + background worker, it might not report the finish LSN of the remote + transaction in server log. </para> </listitem> </varlistentry> diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c index b94236f..ab7a668 100644 --- a/src/backend/commands/subscriptioncmds.c +++ b/src/backend/commands/subscriptioncmds.c @@ -83,7 +83,7 @@ typedef struct SubOpts bool copy_data; bool refresh; bool binary; - bool streaming; + char streaming; bool twophase; bool disableonerr; XLogRecPtr lsn; @@ -96,6 +96,66 @@ static void ReportSlotConnectionError(List *rstates, Oid subid, char *slotname, /* + * Extract the streaming mode value from a DefElem. This is like + * defGetBoolean() but also accepts the special value "spool" and "apply". + */ +static char +defGetStreamingMode(DefElem *def) +{ + /* + * If no parameter given, assume "true" is meant. + */ + if (def->arg == NULL) + return SUBSTREAM_APPLY; + + /* + * Allow 0, 1, "true", "false", "on", "off", "spool" or "apply". + */ + switch (nodeTag(def->arg)) + { + case T_Integer: + switch (intVal(def->arg)) + { + case 0: + return SUBSTREAM_OFF; + case 1: + return SUBSTREAM_APPLY; + default: + /* otherwise, error out below */ + break; + } + break; + default: + { + char *sval = defGetString(def); + + /* + * The set of strings accepted here should match up with the + * grammar's opt_boolean_or_string production. + */ + if (pg_strcasecmp(sval, "true") == 0) + return SUBSTREAM_APPLY; + if (pg_strcasecmp(sval, "false") == 0) + return SUBSTREAM_OFF; + if (pg_strcasecmp(sval, "on") == 0) + return SUBSTREAM_APPLY; + if (pg_strcasecmp(sval, "off") == 0) + return SUBSTREAM_OFF; + if (pg_strcasecmp(sval, "spool") == 0) + return SUBSTREAM_SPOOL; + if (pg_strcasecmp(sval, "apply") == 0) + return SUBSTREAM_APPLY; + } + break; + } + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s requires a Boolean value or \"spool\" or \"apply\"", + def->defname))); + return SUBSTREAM_OFF; /* keep compiler quiet */ +} + +/* * Common option parsing function for CREATE and ALTER SUBSCRIPTION commands. * * Since not all options can be specified in both commands, this function @@ -132,7 +192,7 @@ parse_subscription_options(ParseState *pstate, List *stmt_options, if (IsSet(supported_opts, SUBOPT_BINARY)) opts->binary = false; if (IsSet(supported_opts, SUBOPT_STREAMING)) - opts->streaming = false; + opts->streaming = SUBSTREAM_OFF; if (IsSet(supported_opts, SUBOPT_TWOPHASE_COMMIT)) opts->twophase = false; if (IsSet(supported_opts, SUBOPT_DISABLE_ON_ERR)) @@ -233,7 +293,7 @@ parse_subscription_options(ParseState *pstate, List *stmt_options, errorConflictingDefElem(defel, pstate); opts->specified_opts |= SUBOPT_STREAMING; - opts->streaming = defGetBoolean(defel); + opts->streaming = defGetStreamingMode(defel); } else if (strcmp(defel->defname, "two_phase") == 0) { @@ -601,7 +661,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt, values[Anum_pg_subscription_subowner - 1] = ObjectIdGetDatum(owner); values[Anum_pg_subscription_subenabled - 1] = BoolGetDatum(opts.enabled); values[Anum_pg_subscription_subbinary - 1] = BoolGetDatum(opts.binary); - values[Anum_pg_subscription_substream - 1] = BoolGetDatum(opts.streaming); + values[Anum_pg_subscription_substream - 1] = CharGetDatum(opts.streaming); values[Anum_pg_subscription_subtwophasestate - 1] = CharGetDatum(opts.twophase ? LOGICALREP_TWOPHASE_STATE_PENDING : @@ -1060,7 +1120,7 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt, if (IsSet(opts.specified_opts, SUBOPT_STREAMING)) { values[Anum_pg_subscription_substream - 1] = - BoolGetDatum(opts.streaming); + CharGetDatum(opts.streaming); replaces[Anum_pg_subscription_substream - 1] = true; } diff --git a/src/backend/postmaster/bgworker.c b/src/backend/postmaster/bgworker.c index 30682b6..194a36b 100644 --- a/src/backend/postmaster/bgworker.c +++ b/src/backend/postmaster/bgworker.c @@ -128,6 +128,9 @@ static const struct }, { "ApplyWorkerMain", ApplyWorkerMain + }, + { + "LogicalApplyBgwMain", LogicalApplyBgwMain } }; diff --git a/src/backend/replication/logical/launcher.c b/src/backend/replication/logical/launcher.c index 0adb2d1..bac737c 100644 --- a/src/backend/replication/logical/launcher.c +++ b/src/backend/replication/logical/launcher.c @@ -72,6 +72,7 @@ static void logicalrep_launcher_onexit(int code, Datum arg); static void logicalrep_worker_onexit(int code, Datum arg); static void logicalrep_worker_detach(void); static void logicalrep_worker_cleanup(LogicalRepWorker *worker); +static void stop_worker(LogicalRepWorker *worker); static bool on_commit_launcher_wakeup = false; @@ -225,7 +226,7 @@ logicalrep_worker_find(Oid subid, Oid relid, bool only_running) LogicalRepWorker *w = &LogicalRepCtx->workers[i]; if (w->in_use && w->subid == subid && w->relid == relid && - (!only_running || w->proc)) + (!only_running || w->proc) && !w->subworker) { res = w; break; @@ -262,9 +263,9 @@ logicalrep_workers_find(Oid subid, bool only_running) /* * Start new apply background worker, if possible. */ -void +bool logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, Oid userid, - Oid relid) + Oid relid, dsm_handle subworker_dsm) { BackgroundWorker bgw; BackgroundWorkerHandle *bgw_handle; @@ -351,7 +352,7 @@ retry: if (OidIsValid(relid) && nsyncworkers >= max_sync_workers_per_subscription) { LWLockRelease(LogicalRepWorkerLock); - return; + return false; } /* @@ -365,7 +366,7 @@ retry: (errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED), errmsg("out of logical replication worker slots"), errhint("You might need to increase max_logical_replication_workers."))); - return; + return false; } /* Prepare the worker slot. */ @@ -380,6 +381,7 @@ retry: worker->relstate = SUBREL_STATE_UNKNOWN; worker->relstate_lsn = InvalidXLogRecPtr; worker->stream_fileset = NULL; + worker->subworker = (subworker_dsm != DSM_HANDLE_INVALID); worker->last_lsn = InvalidXLogRecPtr; TIMESTAMP_NOBEGIN(worker->last_send_time); TIMESTAMP_NOBEGIN(worker->last_recv_time); @@ -397,19 +399,31 @@ retry: BGWORKER_BACKEND_DATABASE_CONNECTION; bgw.bgw_start_time = BgWorkerStart_RecoveryFinished; snprintf(bgw.bgw_library_name, BGW_MAXLEN, "postgres"); - snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyWorkerMain"); + + if (subworker_dsm == DSM_HANDLE_INVALID) + snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyWorkerMain"); + else + snprintf(bgw.bgw_function_name, BGW_MAXLEN, "LogicalApplyBgwMain"); + + if (OidIsValid(relid)) snprintf(bgw.bgw_name, BGW_MAXLEN, "logical replication worker for subscription %u sync %u", subid, relid); - else + else if (subworker_dsm == DSM_HANDLE_INVALID) snprintf(bgw.bgw_name, BGW_MAXLEN, "logical replication worker for subscription %u", subid); + else + snprintf(bgw.bgw_name, BGW_MAXLEN, + "logical replication apply worker for subscription %u", subid); snprintf(bgw.bgw_type, BGW_MAXLEN, "logical replication worker"); bgw.bgw_restart_time = BGW_NEVER_RESTART; bgw.bgw_notify_pid = MyProcPid; bgw.bgw_main_arg = Int32GetDatum(slot); + if (subworker_dsm != DSM_HANDLE_INVALID) + memcpy(bgw.bgw_extra, &subworker_dsm, sizeof(dsm_handle)); + if (!RegisterDynamicBackgroundWorker(&bgw, &bgw_handle)) { /* Failed to start worker, so clean up the worker slot. */ @@ -422,11 +436,13 @@ retry: (errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED), errmsg("out of background worker slots"), errhint("You might need to increase max_worker_processes."))); - return; + return false; } /* Now wait until it attaches. */ WaitForReplicationWorkerAttach(worker, generation, bgw_handle); + + return true; } /* @@ -437,7 +453,6 @@ void logicalrep_worker_stop(Oid subid, Oid relid) { LogicalRepWorker *worker; - uint16 generation; LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); @@ -450,6 +465,18 @@ logicalrep_worker_stop(Oid subid, Oid relid) return; } + stop_worker(worker); + + LWLockRelease(LogicalRepWorkerLock); +} + +static void +stop_worker(LogicalRepWorker *worker) +{ + uint16 generation; + + Assert(LWLockHeldByMe(LogicalRepWorkerLock)); + /* * Remember which generation was our worker so we can check if what we see * is still the same one. @@ -523,8 +550,6 @@ logicalrep_worker_stop(Oid subid, Oid relid) LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); } - - LWLockRelease(LogicalRepWorkerLock); } /* @@ -600,6 +625,28 @@ logicalrep_worker_attach(int slot) static void logicalrep_worker_detach(void) { + /* + * If we are the main apply worker, stop all the sub apply workers we + * started before. + */ + if (!MyLogicalRepWorker->subworker) + { + List *workers; + ListCell *lc; + + LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); + + workers = logicalrep_workers_find(MyLogicalRepWorker->subid, true); + foreach(lc, workers) + { + LogicalRepWorker *w = (LogicalRepWorker *) lfirst(lc); + if (w->subworker) + stop_worker(w); + } + + LWLockRelease(LogicalRepWorkerLock); + } + /* Block concurrent access. */ LWLockAcquire(LogicalRepWorkerLock, LW_EXCLUSIVE); @@ -622,6 +669,7 @@ logicalrep_worker_cleanup(LogicalRepWorker *worker) worker->userid = InvalidOid; worker->subid = InvalidOid; worker->relid = InvalidOid; + worker->subworker = false; } /* @@ -869,7 +917,7 @@ ApplyLauncherMain(Datum main_arg) wait_time = wal_retrieve_retry_interval; logicalrep_worker_launch(sub->dbid, sub->oid, sub->name, - sub->owner, InvalidOid); + sub->owner, InvalidOid, DSM_HANDLE_INVALID); } } diff --git a/src/backend/replication/logical/origin.c b/src/backend/replication/logical/origin.c index b0c8b6c..c6ea68a 100644 --- a/src/backend/replication/logical/origin.c +++ b/src/backend/replication/logical/origin.c @@ -1068,7 +1068,7 @@ ReplicationOriginExitCleanup(int code, Datum arg) * with replorigin_session_reset(). */ void -replorigin_session_setup(RepOriginId node) +replorigin_session_setup(RepOriginId node, bool acquire) { static bool registered_cleanup; int i; @@ -1110,7 +1110,11 @@ replorigin_session_setup(RepOriginId node) if (curstate->roident != node) continue; - else if (curstate->acquired_by != 0) + /* + * We allow the apply worker to get the slot which is acquired by its + * leader process. + */ + else if (curstate->acquired_by != 0 && acquire) { ereport(ERROR, (errcode(ERRCODE_OBJECT_IN_USE), @@ -1321,7 +1325,7 @@ pg_replication_origin_session_setup(PG_FUNCTION_ARGS) name = text_to_cstring((text *) DatumGetPointer(PG_GETARG_DATUM(0))); origin = replorigin_by_name(name, false); - replorigin_session_setup(origin); + replorigin_session_setup(origin, true); replorigin_session_origin = origin; diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c index ff8513e..0409fde 100644 --- a/src/backend/replication/logical/proto.c +++ b/src/backend/replication/logical/proto.c @@ -1138,14 +1138,11 @@ logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn, /* * Read STREAM COMMIT from the output stream. */ -TransactionId +void logicalrep_read_stream_commit(StringInfo in, LogicalRepCommitData *commit_data) { - TransactionId xid; uint8 flags; - xid = pq_getmsgint(in, 4); - /* read flags (unused for now) */ flags = pq_getmsgbyte(in); @@ -1156,8 +1153,6 @@ logicalrep_read_stream_commit(StringInfo in, LogicalRepCommitData *commit_data) commit_data->commit_lsn = pq_getmsgint64(in); commit_data->end_lsn = pq_getmsgint64(in); commit_data->committime = pq_getmsgint64(in); - - return xid; } /* diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c index 49ceec3..04c9f96 100644 --- a/src/backend/replication/logical/tablesync.c +++ b/src/backend/replication/logical/tablesync.c @@ -568,7 +568,8 @@ process_syncing_tables_for_apply(XLogRecPtr current_lsn) MySubscription->oid, MySubscription->name, MyLogicalRepWorker->userid, - rstate->relid); + rstate->relid, + DSM_HANDLE_INVALID); hentry->last_start_time = now; } } @@ -589,6 +590,9 @@ process_syncing_tables_for_apply(XLogRecPtr current_lsn) void process_syncing_tables(XLogRecPtr current_lsn) { + if (MyLogicalRepWorker->subworker) + return; + if (am_tablesync_worker()) process_syncing_tables_for_sync(current_lsn); else @@ -1275,7 +1279,7 @@ LogicalRepSyncTableStart(XLogRecPtr *origin_startpos) * time this tablesync was launched. */ originid = replorigin_by_name(originname, false); - replorigin_session_setup(originid); + replorigin_session_setup(originid, true); replorigin_session_origin = originid; *origin_startpos = replorigin_session_get_progress(false); @@ -1386,7 +1390,7 @@ LogicalRepSyncTableStart(XLogRecPtr *origin_startpos) true /* go backward */ , true /* WAL log */ ); UnlockRelationOid(ReplicationOriginRelationId, RowExclusiveLock); - replorigin_session_setup(originid); + replorigin_session_setup(originid, true); replorigin_session_origin = originid; } else diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c index 4171371..dd8e4af 100644 --- a/src/backend/replication/logical/worker.c +++ b/src/backend/replication/logical/worker.c @@ -22,7 +22,23 @@ * STREAMED TRANSACTIONS * --------------------- * Streamed transactions (large transactions exceeding a memory limit on the - * upstream) are not applied immediately, but instead, the data is written + * upstream) are applied via one of two approaches. + * + * 1) Separate background workers + * + * Assign a new bgworker (if available) as soon as the xact's first stream came + * and the main apply worker will send changes to this new worker via shared + * memory. We keep this worker assigned till the transaction commit came and + * also wait for the worker to finish at commit. This preserves commit ordering + * and avoid writing to and reading from file in most cases. We still need to + * spill if there is no worker available. We also need to allow stream_stop to + * complete by the background worker to finish it to avoid deadlocks because + * T-1's current stream of changes can update rows in conflicting order with + * T-2's next stream of changes. + * + * 2) Write to temporary files and apply when the final commit arrives + * + * If no worker is available to handle streamed transaction, we write the data * to temporary files and then applied at once when the final commit arrives. * * Unlike the regular (non-streamed) case, handling streamed transactions has @@ -174,11 +190,15 @@ #include "rewrite/rewriteHandler.h" #include "storage/buffile.h" #include "storage/bufmgr.h" +#include "storage/dsm.h" #include "storage/fd.h" #include "storage/ipc.h" #include "storage/lmgr.h" #include "storage/proc.h" #include "storage/procarray.h" +#include "storage/shm_mq.h" +#include "storage/shm_toc.h" +#include "storage/spin.h" #include "tcop/tcopprot.h" #include "utils/acl.h" #include "utils/builtins.h" @@ -198,6 +218,59 @@ #define NAPTIME_PER_CYCLE 1000 /* max sleep time between cycles (1s) */ +#define PG_LOGICAL_APPLY_SHM_MAGIC 0x79fb2447 // TODO Consider change + +typedef struct ParallelState +{ + slock_t mutex; + bool attached; + bool ready; + bool finished; + bool failed; + Oid subid; + TransactionId stream_xid; + uint32 n; +} ParallelState; + +typedef struct WorkerState +{ + shm_mq_handle *mq_handle; + dsm_segment *dsm_seg; + ParallelState volatile *pstate; +} WorkerState; + +typedef struct WorkerEntry +{ + TransactionId xid; + WorkerState *wstate; +} WorkerEntry; + +/* Apply workers hash table (initialized on first use) */ +static HTAB *ApplyWorkersHash = NULL; +static List *ApplyWorkersIdleList = NIL; +static List *ApplyWorkersList = NIL; +static uint32 nworkers = 0; +static uint32 nfreeworkers = 0; + +/* Fields valid only for apply background workers */ +bool isLogicalApplyWorker = false; +volatile ParallelState *MyParallelState = NULL; +static List *subxactlist = NIL; + +/* Worker setup and interactions */ +static void setup_dsm(WorkerState *wstate); +static WorkerState *setup_background_worker(void); +static void wait_for_worker_ready(WorkerState *wstate, bool notify); +static void wait_for_transaction_finish(WorkerState *wstate); +static void send_data_to_worker(WorkerState *wstate, Size nbytes, + const void *data); +static WorkerState *find_or_start_worker(TransactionId xid, bool start); +static void free_stream_apply_worker(void); +static bool transaction_applied_in_bgworker(TransactionId xid); +static void check_workers_status(void); + +static uint32 nchanges = 0; + typedef struct FlushPosition { dlist_node node; @@ -260,18 +333,22 @@ static XLogRecPtr remote_final_lsn = InvalidXLogRecPtr; static bool in_streamed_transaction = false; static TransactionId stream_xid = InvalidTransactionId; +static WorkerState *stream_apply_worker = NULL; + +#define applying_changes_in_bgworker() (in_streamed_transaction && stream_apply_worker != NULL) /* * We enable skipping all data modification changes (INSERT, UPDATE, etc.) for * the subscription if the remote transaction's finish LSN matches the subskiplsn. * Once we start skipping changes, we don't stop it until we skip all changes of * the transaction even if pg_subscription is updated and MySubscription->skiplsn - * gets changed or reset during that. Also, in streaming transaction cases, we - * don't skip receiving and spooling the changes since we decide whether or not + * gets changed or reset during that. Also, in streaming transaction cases (spool mode), + * we don't skip receiving and spooling the changes since we decide whether or not * to skip applying the changes when starting to apply changes. The subskiplsn is * cleared after successfully skipping the transaction or applying non-empty * transaction. The latter prevents the mistakenly specified subskiplsn from - * being left. + * being left. Note that we cannot skip the streaming transaction in apply mode, + * because we cannot get the finish LSN before applying the changes. */ static XLogRecPtr skip_xact_finish_lsn = InvalidXLogRecPtr; #define is_skipping_changes() (unlikely(!XLogRecPtrIsInvalid(skip_xact_finish_lsn))) @@ -428,41 +505,96 @@ end_replication_step(void) /* * Handle streamed transactions. * - * If in streaming mode (receiving a block of streamed transaction), we - * simply redirect it to a file for the proper toplevel transaction. + * For the main apply worker, if in streaming mode (receiving a block of + * streamed transaction), we send the data to the background worker. + * + * For the background worker, define a savepoint if new subtransaction was + * started. * * Returns true for streamed transactions, false otherwise (regular mode). */ static bool handle_streamed_transaction(LogicalRepMsgType action, StringInfo s) { - TransactionId xid; + TransactionId current_xid = InvalidTransactionId; /* not in streaming mode */ - if (!in_streamed_transaction) + if (!in_streamed_transaction && !isLogicalApplyWorker) return false; - Assert(stream_fd != NULL); Assert(TransactionIdIsValid(stream_xid)); /* * We should have received XID of the subxact as the first part of the * message, so extract it. */ - xid = pq_getmsgint(s, 4); + current_xid = pq_getmsgint(s, 4); - if (!TransactionIdIsValid(xid)) + if (!TransactionIdIsValid(current_xid)) ereport(ERROR, (errcode(ERRCODE_PROTOCOL_VIOLATION), errmsg_internal("invalid transaction ID in streamed replication transaction"))); - /* Add the new subxact to the array (unless already there). */ - subxact_info_add(xid); + if (isLogicalApplyWorker) + { + /* + * Inside logical apply worker we can figure out that new + * subtransaction was started if new change arrived with different xid. + * In that case we can define named savepoint, so that we were able to + * commit/rollback it separately later. + * + * Special case is if the first change comes from subtransaction, then + * we check that current_xid differs from stream_xid. + */ + if (current_xid != stream_xid && + !list_member_int(subxactlist, (int) current_xid)) + { + MemoryContext oldctx; + char *spname = (char *) palloc(64 * sizeof(char)); + sprintf(spname, "savepoint_for_xid_%u", current_xid); + + elog(LOG, "[Apply BGW #%u] defining savepoint %s", MyParallelState->n, spname); + + DefineSavepoint(spname); + + CommitTransactionCommand(); + + oldctx = MemoryContextSwitchTo(ApplyContext); + subxactlist = lappend_int(subxactlist, (int) current_xid); + MemoryContextSwitchTo(oldctx); + } + } + else if (applying_changes_in_bgworker()) + { + /* + * If we decided to apply the changes of this transaction in a + * bgworker, pass the data to the worker. + */ + send_data_to_worker(stream_apply_worker, s->len, s->data); + nchanges += 1; + + /* + * XXX The publisher side don't always send relation update message + * after the streaming transaction, so update the relation in main + * worker here. + */ + if (action == LOGICAL_REP_MSG_RELATION) + { + LogicalRepRelation *rel = logicalrep_read_rel(s); + logicalrep_relmap_update(rel); + } + + } + else + { + /* Add the new subxact to the array (unless already there). */ + subxact_info_add(current_xid); - /* write the change to the current file */ - stream_write_change(action, s); + /* write the change to the current file */ + stream_write_change(action, s); + } - return true; + return !isLogicalApplyWorker; } /* @@ -844,6 +976,9 @@ apply_handle_commit(StringInfo s) apply_handle_commit_internal(&commit_data); + /* Check the status of bgworkers if any. */ + check_workers_status(); + /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(commit_data.end_lsn); @@ -898,7 +1033,9 @@ apply_handle_prepare_internal(LogicalRepPreparedTxnData *prepare_data) * BeginTransactionBlock is necessary to balance the EndTransactionBlock * called within the PrepareTransactionBlock below. */ - BeginTransactionBlock(); + if (!IsTransactionBlock()) + BeginTransactionBlock(); + CommitTransactionCommand(); /* Completes the preceding Begin command. */ /* @@ -950,6 +1087,9 @@ apply_handle_prepare(StringInfo s) in_remote_transaction = false; + /* Check the status of bgworkers if any. */ + check_workers_status(); + /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(prepare_data.end_lsn); @@ -976,30 +1116,51 @@ apply_handle_commit_prepared(StringInfo s) char gid[GIDSIZE]; logicalrep_read_commit_prepared(s, &prepare_data); + set_apply_error_context_xact(prepare_data.xid, prepare_data.commit_lsn); - /* Compute GID for two_phase transactions. */ - TwoPhaseTransactionGid(MySubscription->oid, prepare_data.xid, - gid, sizeof(gid)); + /* Check if we have prepared transaction in another bgworker */ + if (transaction_applied_in_bgworker(prepare_data.xid)) + { + elog(DEBUG1, "received commit for streamed transaction %u", prepare_data.xid); - /* There is no transaction when COMMIT PREPARED is called */ - begin_replication_step(); + /* Send commit message */ + send_data_to_worker(stream_apply_worker, s->len, s->data); - /* - * Update origin state so we can restart streaming from correct position - * in case of crash. - */ - replorigin_session_origin_lsn = prepare_data.end_lsn; - replorigin_session_origin_timestamp = prepare_data.commit_time; + /* Notify worker, that we are done with this xact */ + wait_for_transaction_finish(stream_apply_worker); + + free_stream_apply_worker(); + } + else + { + /* Compute GID for two_phase transactions. */ + TwoPhaseTransactionGid(MySubscription->oid, prepare_data.xid, + gid, sizeof(gid)); + + /* There is no transaction when COMMIT PREPARED is called */ + begin_replication_step(); + + /* + * Update origin state so we can restart streaming from correct position + * in case of crash. + */ + replorigin_session_origin_lsn = prepare_data.end_lsn; + replorigin_session_origin_timestamp = prepare_data.commit_time; + + FinishPreparedTransaction(gid, true); + end_replication_step(); + CommitTransactionCommand(); + } - FinishPreparedTransaction(gid, true); - end_replication_step(); - CommitTransactionCommand(); pgstat_report_stat(false); store_flush_position(prepare_data.end_lsn); in_remote_transaction = false; + /* Check the status of bgworkers if any. */ + check_workers_status(); + /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(prepare_data.end_lsn); @@ -1019,35 +1180,51 @@ apply_handle_rollback_prepared(StringInfo s) char gid[GIDSIZE]; logicalrep_read_rollback_prepared(s, &rollback_data); + set_apply_error_context_xact(rollback_data.xid, rollback_data.rollback_end_lsn); - /* Compute GID for two_phase transactions. */ - TwoPhaseTransactionGid(MySubscription->oid, rollback_data.xid, - gid, sizeof(gid)); + /* Check if we are processing the prepared transaction in a bgworker */ + if (transaction_applied_in_bgworker(rollback_data.xid)) + { + send_data_to_worker(stream_apply_worker, s->len, s->data); - /* - * It is possible that we haven't received prepare because it occurred - * before walsender reached a consistent point or the two_phase was still - * not enabled by that time, so in such cases, we need to skip rollback - * prepared. - */ - if (LookupGXact(gid, rollback_data.prepare_end_lsn, - rollback_data.prepare_time)) + /* Notify worker, that we are done with this xact */ + wait_for_transaction_finish(stream_apply_worker); + + elog(LOG, "rollback prepared streaming of xid %u", rollback_data.xid); + + free_stream_apply_worker(); + } + else { + /* Compute GID for two_phase transactions. */ + TwoPhaseTransactionGid(MySubscription->oid, rollback_data.xid, + gid, sizeof(gid)); + /* - * Update origin state so we can restart streaming from correct - * position in case of crash. + * It is possible that we haven't received prepare because it occurred + * before walsender reached a consistent point or the two_phase was still + * not enabled by that time, so in such cases, we need to skip rollback + * prepared. */ - replorigin_session_origin_lsn = rollback_data.rollback_end_lsn; - replorigin_session_origin_timestamp = rollback_data.rollback_time; + if (LookupGXact(gid, rollback_data.prepare_end_lsn, + rollback_data.prepare_time)) + { + /* + * Update origin state so we can restart streaming from correct + * position in case of crash. + */ + replorigin_session_origin_lsn = rollback_data.rollback_end_lsn; + replorigin_session_origin_timestamp = rollback_data.rollback_time; - /* There is no transaction when ABORT/ROLLBACK PREPARED is called */ - begin_replication_step(); - FinishPreparedTransaction(gid, false); - end_replication_step(); - CommitTransactionCommand(); + /* There is no transaction when ABORT/ROLLBACK PREPARED is called */ + begin_replication_step(); + FinishPreparedTransaction(gid, false); + end_replication_step(); + CommitTransactionCommand(); - clear_subscription_skip_lsn(rollback_data.rollback_end_lsn); + clear_subscription_skip_lsn(rollback_data.rollback_end_lsn); + } } pgstat_report_stat(false); @@ -1055,6 +1232,9 @@ apply_handle_rollback_prepared(StringInfo s) store_flush_position(rollback_data.rollback_end_lsn); in_remote_transaction = false; + /* Check the status of bgworkers if any. */ + check_workers_status(); + /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(rollback_data.rollback_end_lsn); @@ -1063,11 +1243,148 @@ apply_handle_rollback_prepared(StringInfo s) } /* - * Handle STREAM PREPARE. + * Look up worker inside ApplyWorkersHash for requested xid. * - * Logic is in two parts: - * 1. Replay all the spooled operations - * 2. Mark the transaction as prepared + * If start flag is true, try to start a new worker if not found in hash table. + */ +static WorkerState * +find_or_start_worker(TransactionId xid, bool start) +{ + bool found; + WorkerState *wstate; + WorkerEntry *entry = NULL; + + /* + * We don't start new background worker when stream option is off or spool. + */ + if (MySubscription->stream != SUBSTREAM_APPLY) + return NULL; + + /* + * We don't start new background worker if user has set skiplsn as it's + * possible that user want to skip the streaming transaction. For streaming + * transaction, we need to spill the transaction to disk so that we can get + * the last LSN of the transaction to judge whether to skip before starting + * to apply the change. + */ + else if (start && !XLogRecPtrIsInvalid(MySubscription->skiplsn)) + return NULL; + + /* + * For streaming transactions that is being applied in bgworker, we cannot + * decide whether to apply the change for a relation that is not in the + * READY state (see should_apply_changes_for_rel) as we won't know + * remote_final_lsn by that time. So, we don't start new bgworker in this + * case. + */ + else if (start && !AllTablesyncsReady()) + return NULL; + else if (!start && ApplyWorkersHash == NULL) + return NULL; + + /* First time through, initialize apply workers hashtable */ + if (ApplyWorkersHash == NULL) + { + HASHCTL ctl; + + MemSet(&ctl, 0, sizeof(ctl)); + ctl.keysize = sizeof(TransactionId); + ctl.entrysize = sizeof(WorkerEntry); + ctl.hcxt = ApplyContext; + + ApplyWorkersHash = hash_create("logical apply workers hash", 8, &ctl, + HASH_ELEM | HASH_BLOBS | HASH_CONTEXT); + } + + /* + * Find entry for requested transaction. + */ + entry = hash_search(ApplyWorkersHash, &xid, start ? HASH_ENTER : HASH_FIND, + &found); + if (found) + return entry->wstate; + else if (!start) + return NULL; + + /* If there is at least one worker in the idle list, then take one. */ + if (list_length(ApplyWorkersIdleList) > 0) + { + wstate = (WorkerState *) llast(ApplyWorkersIdleList); + ApplyWorkersIdleList = list_delete_last(ApplyWorkersIdleList); + } + else + { + wstate = setup_background_worker(); + + if (wstate == NULL) + { + /* + * If there is no more worker can be launched here, remove the + * entry in hash table. + */ + hash_search(ApplyWorkersHash, &xid, HASH_REMOVE, &found); + return NULL; + } + } + + /* Fill up the hash entry */ + wstate->pstate->finished = false; + wstate->pstate->stream_xid = xid; + entry->wstate = wstate; + entry->xid = xid; + + return wstate; +} + +/* + * Add the worker to the freelist and remove the entry from hash table. + */ +static void +free_stream_apply_worker(void) +{ + bool found; + MemoryContext oldctx; + TransactionId xid = stream_apply_worker->pstate->stream_xid; + + Assert(stream_apply_worker); + + oldctx = MemoryContextSwitchTo(ApplyContext); + + hash_search(ApplyWorkersHash, &xid, + HASH_REMOVE, &found); + + elog(LOG, "adding finished apply worker #%u for xid %u to the idle list", + stream_apply_worker->pstate->n, stream_apply_worker->pstate->stream_xid); + + ApplyWorkersIdleList = lappend(ApplyWorkersIdleList, stream_apply_worker); + stream_apply_worker = NULL; + + MemoryContextSwitchTo(oldctx); +} + +static void +serialize_stream_prepare(LogicalRepPreparedTxnData *prepare_data) +{ + /* Replay all the spooled operations. */ + apply_spooled_messages(prepare_data->xid, prepare_data->prepare_lsn); + + /* Mark the transaction as prepared. */ + apply_handle_prepare_internal(prepare_data); + + CommitTransactionCommand(); + + pgstat_report_stat(false); + + store_flush_position(prepare_data->end_lsn); + + in_remote_transaction = false; + + /* unlink the files with serialized changes and subxact info. */ + stream_cleanup_files(MyLogicalRepWorker->subid, prepare_data->xid); +} + +/* + * Handle STREAM PREPARE. */ static void apply_handle_stream_prepare(StringInfo s) @@ -1088,24 +1405,49 @@ apply_handle_stream_prepare(StringInfo s) logicalrep_read_stream_prepare(s, &prepare_data); set_apply_error_context_xact(prepare_data.xid, prepare_data.prepare_lsn); - elog(DEBUG1, "received prepare for streamed transaction %u", prepare_data.xid); + /* + * If we are in a bgworker, just prepare the transaction. + */ + if (isLogicalApplyWorker) + { + elog(LOG, "received prepare for streamed transaction %u", prepare_data.xid); - /* Replay all the spooled operations. */ - apply_spooled_messages(prepare_data.xid, prepare_data.prepare_lsn); + /* Mark the transaction as prepared. */ + apply_handle_prepare_internal(&prepare_data); - /* Mark the transaction as prepared. */ - apply_handle_prepare_internal(&prepare_data); + CommitTransactionCommand(); - CommitTransactionCommand(); + pgstat_report_stat(false); - pgstat_report_stat(false); + list_free(subxactlist); + subxactlist = NIL; + } - store_flush_position(prepare_data.end_lsn); + /* + * If we are in main apply worker, check if we are processing this + * transaction in a bgworker. + */ + else if (transaction_applied_in_bgworker(prepare_data.xid)) + { + send_data_to_worker(stream_apply_worker, s->len, s->data); + wait_for_worker_ready(stream_apply_worker, true); + + pgstat_report_stat(false); + store_flush_position(prepare_data.end_lsn); + } + + /* + * If we are in main apply worker and the transaction has been serialized + * to file, replay all the spooled operations. + */ + else + serialize_stream_prepare(&prepare_data); in_remote_transaction = false; + stream_apply_worker = NULL; - /* unlink the files with serialized changes and subxact info. */ - stream_cleanup_files(MyLogicalRepWorker->subid, prepare_data.xid); + /* Check the status of bgworkers if any. */ + check_workers_status(); /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(prepare_data.end_lsn); @@ -1142,19 +1484,9 @@ apply_handle_origin(StringInfo s) errmsg_internal("ORIGIN message sent out of order"))); } -/* - * Handle STREAM START message. - */ static void -apply_handle_stream_start(StringInfo s) +serialize_stream_start(bool first_segment) { - bool first_segment; - - if (in_streamed_transaction) - ereport(ERROR, - (errcode(ERRCODE_PROTOCOL_VIOLATION), - errmsg_internal("duplicate STREAM START message"))); - /* * Start a transaction on stream start, this transaction will be committed * on the stream stop unless it is a tablesync worker in which case it @@ -1164,19 +1496,6 @@ apply_handle_stream_start(StringInfo s) */ begin_replication_step(); - /* notify handle methods we're processing a remote transaction */ - in_streamed_transaction = true; - - /* extract XID of the top-level transaction */ - stream_xid = logicalrep_read_stream_start(s, &first_segment); - - if (!TransactionIdIsValid(stream_xid)) - ereport(ERROR, - (errcode(ERRCODE_PROTOCOL_VIOLATION), - errmsg_internal("invalid transaction ID in streamed replication transaction"))); - - set_apply_error_context_xact(stream_xid, InvalidXLogRecPtr); - /* * Initialize the worker's stream_fileset if we haven't yet. This will be * used for the entire duration of the worker so create it in a permanent @@ -1204,60 +1523,132 @@ apply_handle_stream_start(StringInfo s) if (!first_segment) subxact_info_read(MyLogicalRepWorker->subid, stream_xid); - pgstat_report_activity(STATE_RUNNING, NULL); - end_replication_step(); } /* - * Handle STREAM STOP message. + * Handle STREAM START message. */ static void -apply_handle_stream_stop(StringInfo s) +apply_handle_stream_start(StringInfo s) { - if (!in_streamed_transaction) + bool first_segment; + + if (in_streamed_transaction) ereport(ERROR, (errcode(ERRCODE_PROTOCOL_VIOLATION), - errmsg_internal("STREAM STOP message without STREAM START"))); - - /* - * Close the file with serialized changes, and serialize information about - * subxacts for the toplevel transaction. - */ - subxact_info_write(MyLogicalRepWorker->subid, stream_xid); - stream_close_file(); - - /* We must be in a valid transaction state */ - Assert(IsTransactionState()); - - /* Commit the per-stream transaction */ - CommitTransactionCommand(); - - in_streamed_transaction = false; - - /* Reset per-stream context */ - MemoryContextReset(LogicalStreamingContext); + errmsg_internal("duplicate STREAM START message"))); - pgstat_report_activity(STATE_IDLE, NULL); - reset_apply_error_context_info(); -} + /* notify handle methods we're processing a remote transaction */ + in_streamed_transaction = true; -/* - * Handle STREAM abort message. - */ -static void -apply_handle_stream_abort(StringInfo s) -{ - TransactionId xid; - TransactionId subxid; + /* extract XID of the top-level transaction */ + stream_xid = logicalrep_read_stream_start(s, &first_segment); - if (in_streamed_transaction) + if (!TransactionIdIsValid(stream_xid)) ereport(ERROR, (errcode(ERRCODE_PROTOCOL_VIOLATION), - errmsg_internal("STREAM ABORT message without STREAM STOP"))); + errmsg_internal("invalid transaction ID in streamed replication transaction"))); - logicalrep_read_stream_abort(s, &xid, &subxid); + set_apply_error_context_xact(stream_xid, InvalidXLogRecPtr); + if (isLogicalApplyWorker) + { + /* If we are in a bgworker, begin the transaction */ + AcceptInvalidationMessages(); + maybe_reread_subscription(); + + StartTransactionCommand(); + BeginTransactionBlock(); + CommitTransactionCommand(); + + return; + } + + /* + * If we are in main apply worker, check if there is any free bgworker + * we can use to process this transaction. + */ + stream_apply_worker = find_or_start_worker(stream_xid, first_segment); + + if (applying_changes_in_bgworker()) + { + /* + * If we have free worker or we already started to apply this + * transaction in bgworker, we pass the data to worker. + */ + if (first_segment) + send_data_to_worker(stream_apply_worker, s->len, s->data); + + nchanges = 0; + + SpinLockAcquire(&stream_apply_worker->pstate->mutex); + stream_apply_worker->pstate->ready = false; + SpinLockRelease(&stream_apply_worker->pstate->mutex); + + elog(LOG, "starting streaming of xid %u", stream_xid); + } + + /* + * If no worker is available for the first stream start, we start to + * serialize all the changes of the transaction. + */ + else + serialize_stream_start(first_segment); + + pgstat_report_activity(STATE_RUNNING, NULL); +} + +static void +serialize_stream_stop() +{ + /* + * Close the file with serialized changes, and serialize information about + * subxacts for the toplevel transaction. + */ + subxact_info_write(MyLogicalRepWorker->subid, stream_xid); + stream_close_file(); + + /* We must be in a valid transaction state */ + Assert(IsTransactionState()); + + /* Commit the per-stream transaction */ + CommitTransactionCommand(); + + /* Reset per-stream context */ + MemoryContextReset(LogicalStreamingContext); +} + +/* + * Handle STREAM STOP message. + */ +static void +apply_handle_stream_stop(StringInfo s) +{ + if (!in_streamed_transaction) + ereport(ERROR, + (errcode(ERRCODE_PROTOCOL_VIOLATION), + errmsg_internal("STREAM STOP message without STREAM START"))); + + if (applying_changes_in_bgworker()) + { + wait_for_worker_ready(stream_apply_worker, true); + + elog(LOG, "stopped streaming of xid %u, %u changes streamed", stream_xid, nchanges); + } + else + serialize_stream_stop(); + + in_streamed_transaction = false; + stream_apply_worker = NULL; + + pgstat_report_activity(STATE_IDLE, NULL); + reset_apply_error_context_info(); +} + +static void +serialize_stream_abort(TransactionId xid, TransactionId subxid) +{ /* * If the two XIDs are the same, it's in fact abort of toplevel xact, so * just delete the files with serialized info. @@ -1339,8 +1730,114 @@ apply_handle_stream_abort(StringInfo s) end_replication_step(); CommitTransactionCommand(); } +} - reset_apply_error_context_info(); +/* + * Handle STREAM ABORT message. + */ +static void +apply_handle_stream_abort(StringInfo s) +{ + TransactionId xid; + TransactionId subxid; + + if (in_streamed_transaction) + ereport(ERROR, + (errcode(ERRCODE_PROTOCOL_VIOLATION), + errmsg_internal("STREAM COMMIT message without STREAM STOP"))); + + logicalrep_read_stream_abort(s, &xid, &subxid); + + if (isLogicalApplyWorker) + { + ereport(LOG, + (errcode_for_file_access(), + errmsg("[Apply BGW #%u] aborting current transaction xid=%u, subxid=%u", + MyParallelState->n, GetCurrentTransactionIdIfAny(), GetCurrentSubTransactionId()))); + + /* + * If the two XIDs are the same, it's in fact abort of toplevel xact, + * so just free the subxactlist. + */ + if (subxid == xid) + { + set_apply_error_context_xact(subxid, InvalidXLogRecPtr); + + AbortCurrentTransaction(); + + EndTransactionBlock(false); + CommitTransactionCommand(); + + list_free(subxactlist); + subxactlist = NIL; + } + else + { + /* + * OK, so it's a subxact. Rollback to the savepoint. + * + * We also need to read the subxactlist, determine the offset + * tracked for the subxact, and truncate the list. + */ + int i; + bool found = false; + char *spname = (char *) palloc(64 * sizeof(char)); + + set_apply_error_context_xact(subxid, InvalidXLogRecPtr); + + sprintf(spname, "savepoint_for_xid_%u", subxid); + + ereport(LOG, + (errcode_for_file_access(), + errmsg("[Apply BGW #%u] rolling back to savepoint %s", MyParallelState->n, spname))); + + for(i = list_length(subxactlist) - 1; i >= 0; i--) + { + xid = (TransactionId) list_nth_int(subxactlist, i); + if (xid == subxid) + { + found = true; + break; + } + } + + if (found) + { + elog(LOG, "rolled back to savepoint %s", spname); + RollbackToSavepoint(spname); + CommitTransactionCommand(); + subxactlist = list_truncate(subxactlist, i + 1); + } + + pfree(spname); + } + + reset_apply_error_context_info(); + } + + /* + * If we are in main apply worker, check if we are processing this + * transaction in a bgworker. + */ + else if (transaction_applied_in_bgworker(xid)) + { + send_data_to_worker(stream_apply_worker, s->len, s->data); + + if (subxid == xid) + { + wait_for_transaction_finish(stream_apply_worker); + free_stream_apply_worker(); + } + else + wait_for_worker_ready(stream_apply_worker, true); + } + + /* + * We are in main apply worker and the transaction has been serialized + * to file. + */ + else + serialize_stream_abort(xid, subxid); } /* @@ -1463,40 +1960,6 @@ apply_spooled_messages(TransactionId xid, XLogRecPtr lsn) } /* - * Handle STREAM COMMIT message. - */ -static void -apply_handle_stream_commit(StringInfo s) -{ - TransactionId xid; - LogicalRepCommitData commit_data; - - if (in_streamed_transaction) - ereport(ERROR, - (errcode(ERRCODE_PROTOCOL_VIOLATION), - errmsg_internal("STREAM COMMIT message without STREAM STOP"))); - - xid = logicalrep_read_stream_commit(s, &commit_data); - set_apply_error_context_xact(xid, commit_data.commit_lsn); - - elog(DEBUG1, "received commit for streamed transaction %u", xid); - - apply_spooled_messages(xid, commit_data.commit_lsn); - - apply_handle_commit_internal(&commit_data); - - /* unlink the files with serialized changes and subxact info */ - stream_cleanup_files(MyLogicalRepWorker->subid, xid); - - /* Process any tables that are being synchronized in parallel. */ - process_syncing_tables(commit_data.end_lsn); - - pgstat_report_activity(STATE_IDLE, NULL); - - reset_apply_error_context_info(); -} - -/* * Helper function for apply_handle_commit and apply_handle_stream_commit. */ static void @@ -2445,6 +2908,93 @@ apply_handle_truncate(StringInfo s) end_replication_step(); } +/* + * Handle STREAM COMMIT message. + */ +static void +apply_handle_stream_commit(StringInfo s) +{ + LogicalRepCommitData commit_data; + TransactionId xid; + + if (in_streamed_transaction) + ereport(ERROR, + (errcode(ERRCODE_PROTOCOL_VIOLATION), + errmsg_internal("STREAM COMMIT message without STREAM STOP"))); + + xid = pq_getmsgint(s, 4); + logicalrep_read_stream_commit(s, &commit_data); + set_apply_error_context_xact(xid, commit_data.commit_lsn); + + elog(DEBUG1, "received commit for streamed transaction %u", xid); + + if (isLogicalApplyWorker) + { + /* + * Update origin state so we can restart streaming from correct + * position in case of crash. + */ + replorigin_session_origin_lsn = commit_data.end_lsn; + replorigin_session_origin_timestamp = commit_data.committime; + + CommitTransactionCommand(); + EndTransactionBlock(false); + CommitTransactionCommand(); + + pgstat_report_stat(false); + + list_free(subxactlist); + subxactlist = NIL; + } + + /* + * If we are in main apply worker, check if we are processing this + * transaction in a bgworker. + */ + else if (transaction_applied_in_bgworker(xid)) + { + /* Send commit message */ + send_data_to_worker(stream_apply_worker, s->len, s->data); + + /* Notify worker, that we are done with this xact */ + wait_for_transaction_finish(stream_apply_worker); + + pgstat_report_stat(false); + store_flush_position(commit_data.end_lsn); + stop_skipping_changes(); + + free_stream_apply_worker(); + + /* + * The transaction is either non-empty or skipped, so we clear the + * subskiplsn. + */ + clear_subscription_skip_lsn(commit_data.commit_lsn); + } + else + { + /* + * If we are in main apply worker and the transaction has been + * serialized to file, replay all the spooled operations. + */ + apply_spooled_messages(xid, commit_data.commit_lsn); + + apply_handle_commit_internal(&commit_data); + + /* unlink the files with serialized changes and subxact info */ + stream_cleanup_files(MyLogicalRepWorker->subid, xid); + } + + /* Check the status of bgworkers if any. */ + check_workers_status(); + + /* Process any tables that are being synchronized in parallel. */ + process_syncing_tables(commit_data.end_lsn); + + pgstat_report_activity(STATE_IDLE, NULL); + + reset_apply_error_context_info(); +} /* * Logical replication protocol message dispatcher. @@ -2511,6 +3061,7 @@ apply_dispatch(StringInfo s) break; case LOGICAL_REP_MSG_STREAM_START: + elog(LOG, "LOGICAL_REP_MSG_STREAM_START"); apply_handle_stream_start(s); break; @@ -2618,6 +3169,10 @@ store_flush_position(XLogRecPtr remote_lsn) { FlushPosition *flushpos; + /* We only need to collect the LSN in main apply worker */ + if (isLogicalApplyWorker) + return; + /* Need to do this in permanent context */ MemoryContextSwitchTo(ApplyContext); @@ -2794,6 +3349,9 @@ LogicalRepApplyLoop(XLogRecPtr last_received) AcceptInvalidationMessages(); maybe_reread_subscription(); + /* Check the status of bgworkers if any. */ + check_workers_status(); + /* Process any table synchronization changes. */ process_syncing_tables(last_received); } @@ -3467,6 +4025,7 @@ TwoPhaseTransactionGid(Oid subid, TransactionId xid, char *gid, int szgid) snprintf(gid, szgid, "pg_gid_%u_%u", subid, xid); } + /* * Execute the initial sync with error handling. Disable the subscription, * if it's required. @@ -3686,7 +4245,7 @@ ApplyWorkerMain(Datum main_arg) originid = replorigin_by_name(originname, true); if (!OidIsValid(originid)) originid = replorigin_create(originname); - replorigin_session_setup(originid); + replorigin_session_setup(originid, true); replorigin_session_origin = originid; origin_startpos = replorigin_session_get_progress(false); CommitTransactionCommand(); @@ -3733,7 +4292,7 @@ ApplyWorkerMain(Datum main_arg) options.proto.logical.publication_names = MySubscription->publications; options.proto.logical.binary = MySubscription->binary; - options.proto.logical.streaming = MySubscription->stream; + options.proto.logical.streaming = (MySubscription->stream != SUBSTREAM_OFF); options.proto.logical.twophase = false; if (!am_tablesync_worker()) @@ -3891,7 +4450,8 @@ clear_subscription_skip_lsn(XLogRecPtr finish_lsn) XLogRecPtr myskiplsn = MySubscription->skiplsn; bool started_tx = false; - if (likely(XLogRecPtrIsInvalid(myskiplsn))) + if (likely(XLogRecPtrIsInvalid(myskiplsn)) || + isLogicalApplyWorker) return; if (!IsTransactionState()) @@ -4027,3 +4587,643 @@ reset_apply_error_context_info(void) apply_error_callback_arg.remote_attnum = -1; set_apply_error_context_xact(InvalidTransactionId, InvalidXLogRecPtr); } + +/* Apply Background Worker main loop */ +static void +LogicalApplyBgwLoop(shm_mq_handle *mqh, volatile ParallelState *pst) +{ + shm_mq_result shmq_res; + PGPROC *registrant; + ErrorContextCallback errcallback; + XLogRecPtr last_received = InvalidXLogRecPtr; + + registrant = BackendPidGetProc(MyBgworkerEntry->bgw_notify_pid); + SetLatch(®istrant->procLatch); + + /* + * Push apply error context callback. Fields will be filled during + * applying a change. + */ + errcallback.callback = apply_error_callback; + errcallback.previous = error_context_stack; + error_context_stack = &errcallback; + + for (;;) + { + void *data; + Size len; + StringInfoData s; + MemoryContext oldctx; + + CHECK_FOR_INTERRUPTS(); + + /* Ensure we are reading the data into our memory context. */ + oldctx = MemoryContextSwitchTo(ApplyMessageContext); + + shmq_res = shm_mq_receive(mqh, &len, &data, false); + + if (shmq_res != SHM_MQ_SUCCESS) + break; + + if (len == 0) + { + elog(LOG, "[Apply BGW #%u] got zero-length message, stopping", pst->n); + break; + } + else + { + XLogRecPtr start_lsn; + XLogRecPtr end_lsn; + TimestampTz send_time; + + s.cursor = 0; + s.maxlen = -1; + s.data = (char *) data; + s.len = len; + + /* + * We use first byte of message for additional communication between + * main Logical replication worker and Apply BGWorkers, so if it + * differs from 'w', then process it first. + */ + switch (pq_getmsgbyte(&s)) + { + /* Stream stop */ + case 'E': + { + elog(LOG, "[Apply BGW #%u] ended processing streaming chunk," + "waiting on shm_mq_receive", pst->n); + + SpinLockAcquire(&pst->mutex); + pst->ready = true; + SpinLockRelease(&pst->mutex); + + SetLatch(®istrant->procLatch); + + in_streamed_transaction = false; + + pgstat_report_activity(STATE_IDLE, NULL); + + continue; + } + /* Finished processing xact */ + case 'F': + { + elog(LOG, "[Apply BGW #%u] finished processing xact %u", pst->n, stream_xid); + + SpinLockAcquire(&pst->mutex); + pst->finished = true; + SpinLockRelease(&pst->mutex); + + in_remote_transaction = false; + + continue; + } + default: + break; + } + + start_lsn = pq_getmsgint64(&s); + end_lsn = pq_getmsgint64(&s); + send_time = pq_getmsgint64(&s); + + if (last_received < start_lsn) + last_received = start_lsn; + + if (last_received < end_lsn) + last_received = end_lsn; + + /* + * TO IMPROVE: Do we need to display the bgworker's information in + * pg_stat_replication ? + */ + UpdateWorkerStats(last_received, send_time, false); + + /* + * Make sure the handle apply_dispatch methods are aware we're in a remote + * transaction. + */ + in_remote_transaction = true; + + apply_dispatch(&s); + + if (ConfigReloadPending) + { + ConfigReloadPending = false; + ProcessConfigFile(PGC_SIGHUP); + } + } + + MemoryContextSwitchTo(oldctx); + MemoryContextReset(ApplyMessageContext); + } + + MemoryContextSwitchTo(TopMemoryContext); + MemoryContextReset(ApplyContext); + + /* Pop the error context stack */ + error_context_stack = errcallback.previous; + + elog(LOG, "[Apply BGW #%u] exiting", pst->n); + + /* Signal main process that we are done. */ + SetLatch(®istrant->procLatch); +} + +/* + * Set the failed flag so that the main apply worker can realize we have + * shutdown. + */ +static void +ApplyBgwShutdown(int code, Datum arg) +{ + SpinLockAcquire(&MyParallelState->mutex); + MyParallelState->failed = true; + SpinLockRelease(&MyParallelState->mutex); + + dsm_detach((dsm_segment *) DatumGetPointer(arg)); +} + +/* + * Apply Background Worker entry point + */ +void +LogicalApplyBgwMain(Datum main_arg) +{ + volatile ParallelState *pst; + + dsm_handle handle; + dsm_segment *seg; + shm_toc *toc; + shm_mq *mq; + shm_mq_handle *mqh; + MemoryContext oldcontext; + RepOriginId originid; + int worker_slot = DatumGetInt32(main_arg); + char originname[NAMEDATALEN]; + + /* Attach to slot */ + logicalrep_worker_attach(worker_slot); + + MemoryContextSwitchTo(TopMemoryContext); + + /* Load the subscription into persistent memory context. */ + ApplyContext = AllocSetContextCreate(TopMemoryContext, + "ApplyContext", + ALLOCSET_DEFAULT_SIZES); + + /* + * Init the ApplyMessageContext which we clean up after each replication + * protocol message. + */ + ApplyMessageContext = AllocSetContextCreate(ApplyContext, + "ApplyMessageContext", + ALLOCSET_DEFAULT_SIZES); + + isLogicalApplyWorker = true; + + /* Setup signal handling */ + pqsignal(SIGHUP, SignalHandlerForConfigReload); + pqsignal(SIGTERM, die); + BackgroundWorkerUnblockSignals(); + + /* + * Connect to the dynamic shared memory segment. + * + * The backend that registered this worker passed us the ID of a shared + * memory segment to which we must attach for further instructions. In + * order to attach to dynamic shared memory, we need a resource owner. + * Once we've mapped the segment in our address space, attach to the table + * of contents so we can locate the various data structures we'll need to + * find within the segment. + */ + CurrentResourceOwner = ResourceOwnerCreate(NULL, "Logical apply worker"); + memcpy(&handle, MyBgworkerEntry->bgw_extra, sizeof(dsm_handle)); + seg = dsm_attach(handle); + if (seg == NULL) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("unable to map dynamic shared memory segment"))); + toc = shm_toc_attach(PG_LOGICAL_APPLY_SHM_MAGIC, dsm_segment_address(seg)); + if (toc == NULL) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("bad magic number in dynamic shared memory segment"))); + + before_shmem_exit(ApplyBgwShutdown, PointerGetDatum(seg)); + + /* + * Acquire a worker number. + * + * By convention, the process registering this background worker should + * have stored the control structure at key 0. We look up that key to + * find it. Our worker number gives our identity: there may be just one + * worker involved in this parallel operation, or there may be many. + */ + pst = shm_toc_lookup(toc, 0, false); + MyParallelState = pst; + + SpinLockAcquire(&pst->mutex); + pst->attached = true; + SpinLockRelease(&pst->mutex); + + /* + * Attach to the message queue. + */ + mq = shm_toc_lookup(toc, 1, false); + shm_mq_set_receiver(mq, MyProc); + mqh = shm_mq_attach(mq, seg, NULL); + + /* Connect to our database. */ + BackgroundWorkerInitializeConnectionByOid(MyLogicalRepWorker->dbid, + MyLogicalRepWorker->userid, + 0); + + /* + * Set the client encoding to the database encoding, since that is what + * the leader will expect. + */ + SetClientEncoding(GetDatabaseEncoding()); + + stream_xid = pst->stream_xid; + + StartTransactionCommand(); + oldcontext = MemoryContextSwitchTo(ApplyContext); + + MySubscription = GetSubscription(MyLogicalRepWorker->subid, true); + if (!MySubscription) + { + ereport(LOG, + (errmsg("logical replication apply worker for subscription %u will not " + "start because the subscription was removed during startup", + MyLogicalRepWorker->subid))); + proc_exit(0); + } + + MySubscriptionValid = true; + MemoryContextSwitchTo(oldcontext); + + /* Setup synchronous commit according to the user's wishes */ + SetConfigOption("synchronous_commit", MySubscription->synccommit, + PGC_BACKEND, PGC_S_OVERRIDE); + + /* Keep us informed about subscription changes. */ + CacheRegisterSyscacheCallback(SUBSCRIPTIONOID, + subscription_change_cb, + (Datum) 0); + + CommitTransactionCommand(); + + /* Setup replication origin tracking. */ + StartTransactionCommand(); + snprintf(originname, sizeof(originname), "pg_%u", MySubscription->oid); + originid = replorigin_by_name(originname, true); + if (!OidIsValid(originid)) + originid = replorigin_create(originname); + replorigin_session_setup(originid, false); + replorigin_session_origin = originid; + CommitTransactionCommand(); + + /* + * Allocate the origin name in long-lived context for error context + * message. + */ + apply_error_callback_arg.origin_name = MemoryContextStrdup(ApplyContext, + originname); + + /* + * Indicate that we're fully initialized and ready to begin the main part + * of the parallel operation. + * + * Once we signal that we're ready, the user backend is entitled to assume + * that our on_dsm_detach callbacks will fire before we disconnect from + * the shared memory segment and exit. Generally, that means we must have + * attached to all relevant dynamic shared memory data structures by now. + */ + SpinLockAcquire(&pst->mutex); + pst->ready = true; + SpinLockRelease(&pst->mutex); + + elog(LOG, "[Apply BGW #%u] started", pst->n); + + PG_TRY(); + { + LogicalApplyBgwLoop(mqh, pst); + } + PG_CATCH(); + { + /* + * Report the worker failed while applying streaming transaction + * changes. Abort the current transaction so that the stats message is + * sent in an idle state. + */ + AbortOutOfAnyTransaction(); + pgstat_report_subscription_error(MySubscription->oid, false); + + PG_RE_THROW(); + } + PG_END_TRY(); + + /* + * We're done. Explicitly detach the shared memory segment so that we + * don't get a resource leak warning at commit time. This will fire any + * on_dsm_detach callbacks we've registered, as well. Once that's done, + * we can go ahead and exit. + */ + dsm_detach(seg); + proc_exit(0); +} + +/* + * Set up a dynamic shared memory segment. + * + * We set up a control region that contains a ParallelState, + * plus one region per message queue. There are as many message queues as + * the number of workers. + */ +static void +setup_dsm(WorkerState *wstate) +{ + shm_toc_estimator e; + int toc_key = 0; + Size segsize; + dsm_segment *seg; + shm_toc *toc; + ParallelState *pst; + shm_mq *mq; + int64 queue_size = 160000000; /* 16 MB for now */ + + /* + * Estimate how much shared memory we need. + * + * Because the TOC machinery may choose to insert padding of oddly-sized + * requests, we must estimate each chunk separately. + * + * We need one key to register the location of the header, and we need + * nworkers keys to track the locations of the message queues. + */ + shm_toc_initialize_estimator(&e); + shm_toc_estimate_chunk(&e, sizeof(ParallelState)); + shm_toc_estimate_chunk(&e, (Size) queue_size); + + shm_toc_estimate_keys(&e, 1 + 1); + segsize = shm_toc_estimate(&e); + + /* Create the shared memory segment and establish a table of contents. */ + seg = dsm_create(shm_toc_estimate(&e), 0); + toc = shm_toc_create(PG_LOGICAL_APPLY_SHM_MAGIC, dsm_segment_address(seg), + segsize); + + /* Set up the header region. */ + pst = shm_toc_allocate(toc, sizeof(ParallelState)); + SpinLockInit(&pst->mutex); + pst->attached = false; + pst->ready = false; + pst->finished = false; + pst->failed = false; + pst->subid = MyLogicalRepWorker->subid; + pst->stream_xid = stream_xid; + pst->n = nworkers + 1; + + shm_toc_insert(toc, toc_key++, pst); + + /* Set up one message queue per worker, plus one. */ + mq = shm_mq_create(shm_toc_allocate(toc, (Size) queue_size), + (Size) queue_size); + shm_toc_insert(toc, toc_key++, mq); + shm_mq_set_sender(mq, MyProc); + + /* Attach the queues. */ + wstate->mq_handle = shm_mq_attach(mq, seg, NULL); + + /* Return results to caller. */ + wstate->dsm_seg = seg; + wstate->pstate = pst; +} + +/* + * Register background workers. + */ +static WorkerState * +setup_background_worker(void) +{ + MemoryContext oldcontext; + bool launched; + WorkerState *wstate; + + elog(LOG, "setting up apply worker #%u", nworkers + 1); + + oldcontext = MemoryContextSwitchTo(ApplyContext); + + wstate = (WorkerState *) palloc0(sizeof(WorkerState)); + + /* Setup shared memory */ + setup_dsm(wstate); + + launched = logicalrep_worker_launch(MyLogicalRepWorker->dbid, + MySubscription->oid, + MySubscription->name, + MyLogicalRepWorker->userid, + InvalidOid, + dsm_segment_handle(wstate->dsm_seg)); + + if (launched) + { + /* Wait for worker to become ready. */ + wait_for_worker_ready(wstate, false); + + ApplyWorkersList = lappend(ApplyWorkersList, wstate); + nworkers += 1; + } + else + { + pfree(wstate); + wstate = NULL; + } + + MemoryContextSwitchTo(oldcontext); + + return wstate; +} + +/* + * Send the data to worker via shared-memory queue. + */ +static void +send_data_to_worker(WorkerState *wstate, Size nbytes, const void *data) +{ + shm_mq_result result; + + result = shm_mq_send(wstate->mq_handle, nbytes, data, false, true); + + if (result != SHM_MQ_SUCCESS) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("could not send tuple to shared-memory queue"))); +} + +static void +wait_for_worker_ready(WorkerState *wstate, bool notify) +{ + bool result = false; + + if (notify) + { + char action = 'E'; + + /* Notify worker that we received STREAM STOP */ + send_data_to_worker(wstate, 1, &action); + } + + for (;;) + { + bool ready; + bool failed; + + /* If the worker is ready, we have succeeded. */ + SpinLockAcquire(&wstate->pstate->mutex); + ready = wstate->pstate->ready; + failed = wstate->pstate->failed; + SpinLockRelease(&wstate->pstate->mutex); + + if (ready) + { + result = true; + break; + } + + /* If any workers (or the postmaster) have died, we have failed. */ + if (failed) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("Background worker %u failed to apply transaction %u", + wstate->pstate->n, wstate->pstate->stream_xid))); + + /* Wait to be signalled. */ + WaitLatch(MyLatch, WL_LATCH_SET | WL_EXIT_ON_PM_DEATH, 0, + WAIT_EVENT_LOGICAL_APPLY_WORKER_READY); + + /* Reset the latch so we don't spin. */ + ResetLatch(MyLatch); + + /* An interrupt may have occurred while we were waiting. */ + CHECK_FOR_INTERRUPTS(); + } + + if (!result) + ereport(ERROR, + (errcode(ERRCODE_INSUFFICIENT_RESOURCES), + errmsg("one or more background workers failed to start"))); +} + +static void +wait_for_transaction_finish(WorkerState *wstate) +{ + char action = 'F'; + + /* Notify worker that the streaming transaction has finished */ + send_data_to_worker(wstate, 1, &action); + + elog(LOG, "waiting for apply worker #%u to finish processing xid %u", + wstate->pstate->n, wstate->pstate->stream_xid); + + for (;;) + { + bool finished; + bool failed; + + /* If the worker is finished, we have succeeded. */ + SpinLockAcquire(&wstate->pstate->mutex); + finished = wstate->pstate->finished; + failed = wstate->pstate->failed; + SpinLockRelease(&wstate->pstate->mutex); + + if (failed) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("Background worker %u failed to apply transaction %u", + wstate->pstate->n, wstate->pstate->stream_xid))); + + if (finished) + { + break; + } + + /* + * TO IMPROVE. Would it be better to use ConditionVariable instead of + * Wait/Reset Latch here? + */ + + /* Wait to be signalled. */ + WaitLatch(MyLatch, WL_LATCH_SET | WL_EXIT_ON_PM_DEATH, 0, + WAIT_EVENT_LOGICAL_APPLY_WORKER_READY); + + /* Reset the latch so we don't spin. */ + ResetLatch(MyLatch); + + /* An interrupt may have occurred while we were waiting. */ + CHECK_FOR_INTERRUPTS(); + } +} + +/* + * Check if the streamed transaction was being processed in a bgworker. + */ +static bool +transaction_applied_in_bgworker(TransactionId xid) +{ + if (!TransactionIdIsValid(xid) || isLogicalApplyWorker || + ApplyWorkersHash == NULL) + return false; + + stream_apply_worker = find_or_start_worker(xid, false); + + return stream_apply_worker != NULL; +} + +/* + * Check the status of workers and report an error if any bgworker exit + * unexpectedly. + * + * Exit if any relation is not in the READY state and if any worker is handling + * the streaming transaction at the same time. Because for streaming + * transactions that is being applied in bgworker, we cannot decide whether to + * apply the change for a relation that is not in the READY state (see + * should_apply_changes_for_rel) as we won't know remote_final_lsn by that + * time. + */ +static void +check_workers_status(void) +{ + ListCell *lc; + + if (isLogicalApplyWorker || MySubscription->stream != SUBSTREAM_APPLY) + return; + + foreach(lc, ApplyWorkersList) + { + WorkerState *wstate = (WorkerState *) lfirst(lc); + + /* + * We don't lock here as in the worst case we will just detect the + * failure of worker a bit later. + */ + if (wstate->pstate->failed) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("Background worker %u exited unexpectedly", + wstate->pstate->n))); + } + + if (!AllTablesyncsReady() && nfreeworkers != list_length(ApplyWorkersList)) + { + ereport(LOG, + (errmsg("logical replication apply workers for subscription \"%s\" will restart", + MySubscription->name), + errdetail("Cannot start table synchronization while bgworkers are " + "handling streamed replication transaction"))); + + proc_exit(0); + } + +} diff --git a/src/backend/utils/activity/wait_event.c b/src/backend/utils/activity/wait_event.c index 87c15b9..24a6fac 100644 --- a/src/backend/utils/activity/wait_event.c +++ b/src/backend/utils/activity/wait_event.c @@ -388,6 +388,9 @@ pgstat_get_wait_ipc(WaitEventIPC w) case WAIT_EVENT_HASH_GROW_BUCKETS_REINSERT: event_name = "HashGrowBucketsReinsert"; break; + case WAIT_EVENT_LOGICAL_APPLY_WORKER_READY: + event_name = "LogicalApplyWorkerReady"; + break; case WAIT_EVENT_LOGICAL_SYNC_DATA: event_name = "LogicalSyncData"; break; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 969e2a7..8c39050 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -4521,8 +4521,12 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo) if (strcmp(subinfo->subbinary, "t") == 0) appendPQExpBufferStr(query, ", binary = true"); - if (strcmp(subinfo->substream, "f") != 0) - appendPQExpBufferStr(query, ", streaming = on"); + if (strcmp(subinfo->substream, "f") == 0) + appendPQExpBufferStr(query, ", streaming = off"); + else if (strcmp(subinfo->substream, "s") == 0) + appendPQExpBufferStr(query, ", streaming = spool"); + else if (strcmp(subinfo->substream, "a") == 0) + appendPQExpBufferStr(query, ", streaming = apply"); if (strcmp(subinfo->subtwophasestate, two_phase_disabled) != 0) appendPQExpBufferStr(query, ", two_phase = on"); diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h index f006a92..1cd176a 100644 --- a/src/include/catalog/pg_subscription.h +++ b/src/include/catalog/pg_subscription.h @@ -68,7 +68,7 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW bool subbinary; /* True if the subscription wants the * publisher to send data in binary */ - bool substream; /* Stream in-progress transactions. */ + char substream; /* Stream in-progress transactions. */ char subtwophasestate; /* Stream two-phase transactions */ @@ -111,7 +111,7 @@ typedef struct Subscription bool enabled; /* Indicates if the subscription is enabled */ bool binary; /* Indicates if the subscription wants data in * binary format */ - bool stream; /* Allow streaming in-progress transactions. */ + char stream; /* Allow streaming in-progress transactions. */ char twophasestate; /* Allow streaming two-phase transactions */ bool disableonerr; /* Indicates if the subscription should be * automatically disabled if a worker error @@ -122,6 +122,18 @@ typedef struct Subscription List *publications; /* List of publication names to subscribe to */ } Subscription; +/* Disallow streaming in-progress transactions */ +#define SUBSTREAM_OFF 'f' + +/* + * Streaming transactions are written to a temporary file and applied only + * after the transaction is committed on upstream. + */ +#define SUBSTREAM_SPOOL 's' + +/* Streaming transactions are appied immediately via a background worker */ +#define SUBSTREAM_APPLY 'a' + extern Subscription *GetSubscription(Oid subid, bool missing_ok); extern void FreeSubscription(Subscription *sub); extern void DisableSubscription(Oid subid); diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h index a771ab8..0116867 100644 --- a/src/include/replication/logicalproto.h +++ b/src/include/replication/logicalproto.h @@ -243,8 +243,10 @@ extern TransactionId logicalrep_read_stream_start(StringInfo in, extern void logicalrep_write_stream_stop(StringInfo out); extern void logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn, XLogRecPtr commit_lsn); -extern TransactionId logicalrep_read_stream_commit(StringInfo out, +extern TransactionId logicalrep_read_stream_commit_old(StringInfo out, LogicalRepCommitData *commit_data); +extern void logicalrep_read_stream_commit(StringInfo out, + LogicalRepCommitData *commit_data); extern void logicalrep_write_stream_abort(StringInfo out, TransactionId xid, TransactionId subxid); extern void logicalrep_read_stream_abort(StringInfo in, TransactionId *xid, diff --git a/src/include/replication/logicalworker.h b/src/include/replication/logicalworker.h index cd1b6e8..3bb3511 100644 --- a/src/include/replication/logicalworker.h +++ b/src/include/replication/logicalworker.h @@ -13,6 +13,7 @@ #define LOGICALWORKER_H extern void ApplyWorkerMain(Datum main_arg); +extern void LogicalApplyBgwMain(Datum main_arg); extern bool IsLogicalWorker(void); diff --git a/src/include/replication/origin.h b/src/include/replication/origin.h index 14d5c49..1ed51bc 100644 --- a/src/include/replication/origin.h +++ b/src/include/replication/origin.h @@ -53,7 +53,7 @@ extern XLogRecPtr replorigin_get_progress(RepOriginId node, bool flush); extern void replorigin_session_advance(XLogRecPtr remote_commit, XLogRecPtr local_commit); -extern void replorigin_session_setup(RepOriginId node); +extern void replorigin_session_setup(RepOriginId node, bool acquire); extern void replorigin_session_reset(void); extern XLogRecPtr replorigin_session_get_progress(bool flush); diff --git a/src/include/replication/worker_internal.h b/src/include/replication/worker_internal.h index 4485d4e..4d11a13 100644 --- a/src/include/replication/worker_internal.h +++ b/src/include/replication/worker_internal.h @@ -60,6 +60,8 @@ typedef struct LogicalRepWorker */ FileSet *stream_fileset; + bool subworker; + /* Stats. */ XLogRecPtr last_lsn; TimestampTz last_send_time; @@ -84,8 +86,9 @@ extern void logicalrep_worker_attach(int slot); extern LogicalRepWorker *logicalrep_worker_find(Oid subid, Oid relid, bool only_running); extern List *logicalrep_workers_find(Oid subid, bool only_running); -extern void logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, - Oid userid, Oid relid); +extern bool logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, + Oid userid, Oid relid, + dsm_handle subworker_dsm); extern void logicalrep_worker_stop(Oid subid, Oid relid); extern void logicalrep_worker_wakeup(Oid subid, Oid relid); extern void logicalrep_worker_wakeup_ptr(LogicalRepWorker *worker); diff --git a/src/include/utils/wait_event.h b/src/include/utils/wait_event.h index b578e2e..d5081d9 100644 --- a/src/include/utils/wait_event.h +++ b/src/include/utils/wait_event.h @@ -105,6 +105,7 @@ typedef enum WAIT_EVENT_HASH_GROW_BUCKETS_ALLOCATE, WAIT_EVENT_HASH_GROW_BUCKETS_ELECT, WAIT_EVENT_HASH_GROW_BUCKETS_REINSERT, + WAIT_EVENT_LOGICAL_APPLY_WORKER_READY, WAIT_EVENT_LOGICAL_SYNC_DATA, WAIT_EVENT_LOGICAL_SYNC_STATE_CHANGE, WAIT_EVENT_MQ_INTERNAL, diff --git a/src/test/subscription/t/029_on_error.pl b/src/test/subscription/t/029_on_error.pl index e8b904b..9659fb9 100644 --- a/src/test/subscription/t/029_on_error.pl +++ b/src/test/subscription/t/029_on_error.pl @@ -108,7 +108,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres'; $node_publisher->safe_psql('postgres', "CREATE PUBLICATION pub FOR TABLE tbl"); $node_subscriber->safe_psql('postgres', - "CREATE SUBSCRIPTION sub CONNECTION '$publisher_connstr' PUBLICATION pub WITH (disable_on_error = true, streaming = on, two_phase = on)" + "CREATE SUBSCRIPTION sub CONNECTION '$publisher_connstr' PUBLICATION pub WITH (disable_on_error = true, streaming = spool, two_phase = on)" ); # Initial synchronization failure causes the subscription to be disabled. -- 2.7.2.windows.1 ^ permalink raw reply [nested|flat] 66+ messages in thread
* RE: Perform streaming logical transactions by background workers and parallel apply @ 2022-04-20 12:22 [email protected] <[email protected]> parent: [email protected] <[email protected]> 0 siblings, 1 reply; 66+ messages in thread From: [email protected] @ 2022-04-20 12:22 UTC (permalink / raw) To: Amit Kapila <[email protected]>; +Cc: [email protected] <[email protected]>; PostgreSQL Hackers <[email protected]> On Wednesday, April 20, 2022 4:57 PM [email protected] wrote: > > On Tuesday, April 19, 2022 2:58 PM Amit Kapila <[email protected]> > wrote: > > > > On Thu, Apr 14, 2022 at 9:12 AM [email protected] > > <[email protected]> wrote: > > > > > > On Friday, April 8, 2022 5:14 PM [email protected] > > <[email protected]> wrote: > > > > > > Attach a new version patch which improved the error handling and > > > handled > > the case > > > when there is no more worker available (will spill the data to the > > > temp file in > > this case). > > > > > > Currently, it still doesn't support skip the streamed transaction in > > > bgworker, > > because > > > in this approach, we don't know the last lsn for the streamed > > > transaction > > being applied, > > > so cannot get the lsn to SKIP. I will think more about it and keep > > > testing the > > patch. > > > > > > > I think we can avoid performing the streaming transaction by bgworker > > if skip_lsn is set. This needs some more thought but anyway I see > > another problem in this patch. I think we won't be able to make the > > decision whether to apply the change for a relation that is not in the > > 'READY' state (see should_apply_changes_for_rel) as we won't know > > 'remote_final_lsn' by that time for streaming transactions. I think > > what we can do here is that before assigning the transaction to > > bgworker, we can check if any of the rels is not in the 'READY' state, > > we can make the transaction spill the changes as we are doing now. > > Even if we do such a check, it is still possible that some rel on > > which this transaction is performing operation can appear to be in > > 'non-ready' state after starting bgworker and for such a case I think > > we need to give error and restart the transaction as we have no way to > > know whether we need to perform an operation on the 'rel'. This is > > possible if the user performs REFRESH PUBLICATION in parallel to this > > transaction as that can add a new rel to the pg_subscription_rel. > > Changed as suggested. > > Attach the new version patch which cleanup some code and fix above problem. > For now, it won't apply streaming transaction in bgworker if skiplsn is set or any > table is not in 'READY' state. > > Besides, extent the subscription streaming option to ('on/off/apply(apply in > bgworker)/spool(spool to file)') so that user can control whether to apply The > transaction in a bgworker. Sorry, there was a miss in the pg_dump testcase which cause failure in CFbot. Attach a new version patch which fix that. Best regards, Hou zj Attachments: [application/octet-stream] v4-0001-Perform-streaming-logical-transactions-by-background.patch (81.9K, ../../OS0PR01MB5716A6C18E69FB1EA9A27EED94F59@OS0PR01MB5716.jpnprd01.prod.outlook.com/2-v4-0001-Perform-streaming-logical-transactions-by-background.patch) download | inline diff: From 6909fe8dd7b5f1c751ee74b817752e0515abbbed Mon Sep 17 00:00:00 2001 From: "houzj.fnst" <[email protected]> Date: Wed, 20 Apr 2022 16:45:07 +0800 Subject: [PATCH] Perform streaming logical transactions by background workers Currently, for large transactions, the publisher sends the data in multiple streams (changes divided into chunks depending upon logical_decoding_work_mem), and then on the subscriber-side, the apply worker writes the changes into temporary files and once it receives the commit, it read from the file and apply the entire transaction. To improve the performance of such transactions, we can instead allow them to be applied via background workers. In this approach, we assign a new bgworker (if available) as soon as the xact's first stream came and the main apply worker will send changes to this new worker via shared memory. The bgworker will directly apply the change instead of writing it to temporary files. We keep this worker assigned till the transaction commit came and also wait for the worker to finish at commit. This preserves commit ordering and avoid writing to and reading from file in most cases. We still need to spill if there is no worker available. We also need to allow stream_stop to complete by the background worker to finish it to avoid deadlocks because T-1's current stream of changes can update rows in conflicting order with T-2's next stream of changes. Also extend the subscription streaming option so that user can control whether apply the streaming transaction in a bgworker or spill the change to disk. User can set the streaming option to 'on/off', 'apply', 'spool'. For now, 'on' and 'apply' means the streaming will be applied via a bgworker if available. 'spool' means the streaming transaction will be spilled to disk. --- doc/src/sgml/catalogs.sgml | 10 +- doc/src/sgml/ref/create_subscription.sgml | 24 +- src/backend/commands/subscriptioncmds.c | 70 +- src/backend/postmaster/bgworker.c | 3 + src/backend/replication/logical/launcher.c | 72 +- src/backend/replication/logical/origin.c | 10 +- src/backend/replication/logical/proto.c | 7 +- src/backend/replication/logical/tablesync.c | 10 +- src/backend/replication/logical/worker.c | 1532 ++++++++++++++++++++++++--- src/backend/utils/activity/wait_event.c | 3 + src/bin/pg_dump/pg_dump.c | 8 +- src/include/catalog/pg_subscription.h | 16 +- src/include/replication/logicalproto.h | 4 +- src/include/replication/logicalworker.h | 1 + src/include/replication/origin.h | 2 +- src/include/replication/worker_internal.h | 7 +- src/include/utils/wait_event.h | 1 + src/test/regress/expected/subscription.out | 8 +- src/test/subscription/t/029_on_error.pl | 2 +- 19 files changed, 1574 insertions(+), 216 deletions(-) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index a533a21..d3da9d1 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -7863,11 +7863,15 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l <row> <entry role="catalog_table_entry"><para role="column_definition"> - <structfield>substream</structfield> <type>bool</type> + <structfield>substream</structfield> <type>char</type> </para> <para> - If true, the subscription will allow streaming of in-progress - transactions + Controls in which modes we handle the streaming of in-progress transactions. + <literal>f</literal> = disallow streaming of in-progress transactions + <literal>s</literal> = spill the changes of in-progress transactions to + disk and apply at once after the transaction is committed on the + publisher. + <literal>a</literal> = apply changes directly in background worker </para></entry> </row> diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml index 203bb41..2206180 100644 --- a/doc/src/sgml/ref/create_subscription.sgml +++ b/doc/src/sgml/ref/create_subscription.sgml @@ -217,13 +217,29 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl </varlistentry> <varlistentry> - <term><literal>streaming</literal> (<type>boolean</type>)</term> + <term><literal>streaming</literal> (<type>enum</type>)</term> <listitem> <para> Specifies whether to enable streaming of in-progress transactions - for this subscription. By default, all transactions - are fully decoded on the publisher and only then sent to the - subscriber as a whole. + for this subscription. The default value is <literal>off</literal>, + all transactions are fully decoded on the publisher and only then + sent to the subscriber as a whole. + </para> + + <para> + If set to <literal>spool</literal>, the changes of transaction are + written to temporary files and then applied at once after the + transaction is committed on the publisher. + </para> + + <para> + If set to <literal>apply</literal> or <literal>on</literal>, incoming + changes are directly applied via one of the background worker, if + available. If no background worker is free to handle streaming + transaction then the changes are written to a file and applied after + the transaction is committed. Note that if error happen when applying + changes in background worker, it might not report the finish LSN of + the remote transaction in server log. </para> </listitem> </varlistentry> diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c index b94236f..ab7a668 100644 --- a/src/backend/commands/subscriptioncmds.c +++ b/src/backend/commands/subscriptioncmds.c @@ -83,7 +83,7 @@ typedef struct SubOpts bool copy_data; bool refresh; bool binary; - bool streaming; + char streaming; bool twophase; bool disableonerr; XLogRecPtr lsn; @@ -96,6 +96,66 @@ static void ReportSlotConnectionError(List *rstates, Oid subid, char *slotname, /* + * Extract the streaming mode value from a DefElem. This is like + * defGetBoolean() but also accepts the special value "spool" and "apply". + */ +static char +defGetStreamingMode(DefElem *def) +{ + /* + * If no parameter given, assume "true" is meant. + */ + if (def->arg == NULL) + return SUBSTREAM_APPLY; + + /* + * Allow 0, 1, "true", "false", "on", "off", "spool" or "apply". + */ + switch (nodeTag(def->arg)) + { + case T_Integer: + switch (intVal(def->arg)) + { + case 0: + return SUBSTREAM_OFF; + case 1: + return SUBSTREAM_APPLY; + default: + /* otherwise, error out below */ + break; + } + break; + default: + { + char *sval = defGetString(def); + + /* + * The set of strings accepted here should match up with the + * grammar's opt_boolean_or_string production. + */ + if (pg_strcasecmp(sval, "true") == 0) + return SUBSTREAM_APPLY; + if (pg_strcasecmp(sval, "false") == 0) + return SUBSTREAM_OFF; + if (pg_strcasecmp(sval, "on") == 0) + return SUBSTREAM_APPLY; + if (pg_strcasecmp(sval, "off") == 0) + return SUBSTREAM_OFF; + if (pg_strcasecmp(sval, "spool") == 0) + return SUBSTREAM_SPOOL; + if (pg_strcasecmp(sval, "apply") == 0) + return SUBSTREAM_APPLY; + } + break; + } + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s requires a Boolean value or \"spool\" or \"apply\"", + def->defname))); + return SUBSTREAM_OFF; /* keep compiler quiet */ +} + +/* * Common option parsing function for CREATE and ALTER SUBSCRIPTION commands. * * Since not all options can be specified in both commands, this function @@ -132,7 +192,7 @@ parse_subscription_options(ParseState *pstate, List *stmt_options, if (IsSet(supported_opts, SUBOPT_BINARY)) opts->binary = false; if (IsSet(supported_opts, SUBOPT_STREAMING)) - opts->streaming = false; + opts->streaming = SUBSTREAM_OFF; if (IsSet(supported_opts, SUBOPT_TWOPHASE_COMMIT)) opts->twophase = false; if (IsSet(supported_opts, SUBOPT_DISABLE_ON_ERR)) @@ -233,7 +293,7 @@ parse_subscription_options(ParseState *pstate, List *stmt_options, errorConflictingDefElem(defel, pstate); opts->specified_opts |= SUBOPT_STREAMING; - opts->streaming = defGetBoolean(defel); + opts->streaming = defGetStreamingMode(defel); } else if (strcmp(defel->defname, "two_phase") == 0) { @@ -601,7 +661,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt, values[Anum_pg_subscription_subowner - 1] = ObjectIdGetDatum(owner); values[Anum_pg_subscription_subenabled - 1] = BoolGetDatum(opts.enabled); values[Anum_pg_subscription_subbinary - 1] = BoolGetDatum(opts.binary); - values[Anum_pg_subscription_substream - 1] = BoolGetDatum(opts.streaming); + values[Anum_pg_subscription_substream - 1] = CharGetDatum(opts.streaming); values[Anum_pg_subscription_subtwophasestate - 1] = CharGetDatum(opts.twophase ? LOGICALREP_TWOPHASE_STATE_PENDING : @@ -1060,7 +1120,7 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt, if (IsSet(opts.specified_opts, SUBOPT_STREAMING)) { values[Anum_pg_subscription_substream - 1] = - BoolGetDatum(opts.streaming); + CharGetDatum(opts.streaming); replaces[Anum_pg_subscription_substream - 1] = true; } diff --git a/src/backend/postmaster/bgworker.c b/src/backend/postmaster/bgworker.c index 30682b6..194a36b 100644 --- a/src/backend/postmaster/bgworker.c +++ b/src/backend/postmaster/bgworker.c @@ -128,6 +128,9 @@ static const struct }, { "ApplyWorkerMain", ApplyWorkerMain + }, + { + "LogicalApplyBgwMain", LogicalApplyBgwMain } }; diff --git a/src/backend/replication/logical/launcher.c b/src/backend/replication/logical/launcher.c index 0adb2d1..bac737c 100644 --- a/src/backend/replication/logical/launcher.c +++ b/src/backend/replication/logical/launcher.c @@ -72,6 +72,7 @@ static void logicalrep_launcher_onexit(int code, Datum arg); static void logicalrep_worker_onexit(int code, Datum arg); static void logicalrep_worker_detach(void); static void logicalrep_worker_cleanup(LogicalRepWorker *worker); +static void stop_worker(LogicalRepWorker *worker); static bool on_commit_launcher_wakeup = false; @@ -225,7 +226,7 @@ logicalrep_worker_find(Oid subid, Oid relid, bool only_running) LogicalRepWorker *w = &LogicalRepCtx->workers[i]; if (w->in_use && w->subid == subid && w->relid == relid && - (!only_running || w->proc)) + (!only_running || w->proc) && !w->subworker) { res = w; break; @@ -262,9 +263,9 @@ logicalrep_workers_find(Oid subid, bool only_running) /* * Start new apply background worker, if possible. */ -void +bool logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, Oid userid, - Oid relid) + Oid relid, dsm_handle subworker_dsm) { BackgroundWorker bgw; BackgroundWorkerHandle *bgw_handle; @@ -351,7 +352,7 @@ retry: if (OidIsValid(relid) && nsyncworkers >= max_sync_workers_per_subscription) { LWLockRelease(LogicalRepWorkerLock); - return; + return false; } /* @@ -365,7 +366,7 @@ retry: (errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED), errmsg("out of logical replication worker slots"), errhint("You might need to increase max_logical_replication_workers."))); - return; + return false; } /* Prepare the worker slot. */ @@ -380,6 +381,7 @@ retry: worker->relstate = SUBREL_STATE_UNKNOWN; worker->relstate_lsn = InvalidXLogRecPtr; worker->stream_fileset = NULL; + worker->subworker = (subworker_dsm != DSM_HANDLE_INVALID); worker->last_lsn = InvalidXLogRecPtr; TIMESTAMP_NOBEGIN(worker->last_send_time); TIMESTAMP_NOBEGIN(worker->last_recv_time); @@ -397,19 +399,31 @@ retry: BGWORKER_BACKEND_DATABASE_CONNECTION; bgw.bgw_start_time = BgWorkerStart_RecoveryFinished; snprintf(bgw.bgw_library_name, BGW_MAXLEN, "postgres"); - snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyWorkerMain"); + + if (subworker_dsm == DSM_HANDLE_INVALID) + snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyWorkerMain"); + else + snprintf(bgw.bgw_function_name, BGW_MAXLEN, "LogicalApplyBgwMain"); + + if (OidIsValid(relid)) snprintf(bgw.bgw_name, BGW_MAXLEN, "logical replication worker for subscription %u sync %u", subid, relid); - else + else if (subworker_dsm == DSM_HANDLE_INVALID) snprintf(bgw.bgw_name, BGW_MAXLEN, "logical replication worker for subscription %u", subid); + else + snprintf(bgw.bgw_name, BGW_MAXLEN, + "logical replication apply worker for subscription %u", subid); snprintf(bgw.bgw_type, BGW_MAXLEN, "logical replication worker"); bgw.bgw_restart_time = BGW_NEVER_RESTART; bgw.bgw_notify_pid = MyProcPid; bgw.bgw_main_arg = Int32GetDatum(slot); + if (subworker_dsm != DSM_HANDLE_INVALID) + memcpy(bgw.bgw_extra, &subworker_dsm, sizeof(dsm_handle)); + if (!RegisterDynamicBackgroundWorker(&bgw, &bgw_handle)) { /* Failed to start worker, so clean up the worker slot. */ @@ -422,11 +436,13 @@ retry: (errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED), errmsg("out of background worker slots"), errhint("You might need to increase max_worker_processes."))); - return; + return false; } /* Now wait until it attaches. */ WaitForReplicationWorkerAttach(worker, generation, bgw_handle); + + return true; } /* @@ -437,7 +453,6 @@ void logicalrep_worker_stop(Oid subid, Oid relid) { LogicalRepWorker *worker; - uint16 generation; LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); @@ -450,6 +465,18 @@ logicalrep_worker_stop(Oid subid, Oid relid) return; } + stop_worker(worker); + + LWLockRelease(LogicalRepWorkerLock); +} + +static void +stop_worker(LogicalRepWorker *worker) +{ + uint16 generation; + + Assert(LWLockHeldByMe(LogicalRepWorkerLock)); + /* * Remember which generation was our worker so we can check if what we see * is still the same one. @@ -523,8 +550,6 @@ logicalrep_worker_stop(Oid subid, Oid relid) LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); } - - LWLockRelease(LogicalRepWorkerLock); } /* @@ -600,6 +625,28 @@ logicalrep_worker_attach(int slot) static void logicalrep_worker_detach(void) { + /* + * If we are the main apply worker, stop all the sub apply workers we + * started before. + */ + if (!MyLogicalRepWorker->subworker) + { + List *workers; + ListCell *lc; + + LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); + + workers = logicalrep_workers_find(MyLogicalRepWorker->subid, true); + foreach(lc, workers) + { + LogicalRepWorker *w = (LogicalRepWorker *) lfirst(lc); + if (w->subworker) + stop_worker(w); + } + + LWLockRelease(LogicalRepWorkerLock); + } + /* Block concurrent access. */ LWLockAcquire(LogicalRepWorkerLock, LW_EXCLUSIVE); @@ -622,6 +669,7 @@ logicalrep_worker_cleanup(LogicalRepWorker *worker) worker->userid = InvalidOid; worker->subid = InvalidOid; worker->relid = InvalidOid; + worker->subworker = false; } /* @@ -869,7 +917,7 @@ ApplyLauncherMain(Datum main_arg) wait_time = wal_retrieve_retry_interval; logicalrep_worker_launch(sub->dbid, sub->oid, sub->name, - sub->owner, InvalidOid); + sub->owner, InvalidOid, DSM_HANDLE_INVALID); } } diff --git a/src/backend/replication/logical/origin.c b/src/backend/replication/logical/origin.c index b0c8b6c..c6ea68a 100644 --- a/src/backend/replication/logical/origin.c +++ b/src/backend/replication/logical/origin.c @@ -1068,7 +1068,7 @@ ReplicationOriginExitCleanup(int code, Datum arg) * with replorigin_session_reset(). */ void -replorigin_session_setup(RepOriginId node) +replorigin_session_setup(RepOriginId node, bool acquire) { static bool registered_cleanup; int i; @@ -1110,7 +1110,11 @@ replorigin_session_setup(RepOriginId node) if (curstate->roident != node) continue; - else if (curstate->acquired_by != 0) + /* + * We allow the apply worker to get the slot which is acquired by its + * leader process. + */ + else if (curstate->acquired_by != 0 && acquire) { ereport(ERROR, (errcode(ERRCODE_OBJECT_IN_USE), @@ -1321,7 +1325,7 @@ pg_replication_origin_session_setup(PG_FUNCTION_ARGS) name = text_to_cstring((text *) DatumGetPointer(PG_GETARG_DATUM(0))); origin = replorigin_by_name(name, false); - replorigin_session_setup(origin); + replorigin_session_setup(origin, true); replorigin_session_origin = origin; diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c index ff8513e..0409fde 100644 --- a/src/backend/replication/logical/proto.c +++ b/src/backend/replication/logical/proto.c @@ -1138,14 +1138,11 @@ logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn, /* * Read STREAM COMMIT from the output stream. */ -TransactionId +void logicalrep_read_stream_commit(StringInfo in, LogicalRepCommitData *commit_data) { - TransactionId xid; uint8 flags; - xid = pq_getmsgint(in, 4); - /* read flags (unused for now) */ flags = pq_getmsgbyte(in); @@ -1156,8 +1153,6 @@ logicalrep_read_stream_commit(StringInfo in, LogicalRepCommitData *commit_data) commit_data->commit_lsn = pq_getmsgint64(in); commit_data->end_lsn = pq_getmsgint64(in); commit_data->committime = pq_getmsgint64(in); - - return xid; } /* diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c index 49ceec3..04c9f96 100644 --- a/src/backend/replication/logical/tablesync.c +++ b/src/backend/replication/logical/tablesync.c @@ -568,7 +568,8 @@ process_syncing_tables_for_apply(XLogRecPtr current_lsn) MySubscription->oid, MySubscription->name, MyLogicalRepWorker->userid, - rstate->relid); + rstate->relid, + DSM_HANDLE_INVALID); hentry->last_start_time = now; } } @@ -589,6 +590,9 @@ process_syncing_tables_for_apply(XLogRecPtr current_lsn) void process_syncing_tables(XLogRecPtr current_lsn) { + if (MyLogicalRepWorker->subworker) + return; + if (am_tablesync_worker()) process_syncing_tables_for_sync(current_lsn); else @@ -1275,7 +1279,7 @@ LogicalRepSyncTableStart(XLogRecPtr *origin_startpos) * time this tablesync was launched. */ originid = replorigin_by_name(originname, false); - replorigin_session_setup(originid); + replorigin_session_setup(originid, true); replorigin_session_origin = originid; *origin_startpos = replorigin_session_get_progress(false); @@ -1386,7 +1390,7 @@ LogicalRepSyncTableStart(XLogRecPtr *origin_startpos) true /* go backward */ , true /* WAL log */ ); UnlockRelationOid(ReplicationOriginRelationId, RowExclusiveLock); - replorigin_session_setup(originid); + replorigin_session_setup(originid, true); replorigin_session_origin = originid; } else diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c index 4171371..dd8e4af 100644 --- a/src/backend/replication/logical/worker.c +++ b/src/backend/replication/logical/worker.c @@ -22,7 +22,23 @@ * STREAMED TRANSACTIONS * --------------------- * Streamed transactions (large transactions exceeding a memory limit on the - * upstream) are not applied immediately, but instead, the data is written + * upstream) are applied via one of two approaches. + * + * 1) Separate background workers + * + * Assign a new bgworker (if available) as soon as the xact's first stream came + * and the main apply worker will send changes to this new worker via shared + * memory. We keep this worker assigned till the transaction commit came and + * also wait for the worker to finish at commit. This preserves commit ordering + * and avoid writing to and reading from file in most cases. We still need to + * spill if there is no worker available. We also need to allow stream_stop to + * complete by the background worker to finish it to avoid deadlocks because + * T-1's current stream of changes can update rows in conflicting order with + * T-2's next stream of changes. + * + * 2) Write to temporary files and apply when the final commit arrives + * + * If no worker is available to handle streamed transaction, we write the data * to temporary files and then applied at once when the final commit arrives. * * Unlike the regular (non-streamed) case, handling streamed transactions has @@ -174,11 +190,15 @@ #include "rewrite/rewriteHandler.h" #include "storage/buffile.h" #include "storage/bufmgr.h" +#include "storage/dsm.h" #include "storage/fd.h" #include "storage/ipc.h" #include "storage/lmgr.h" #include "storage/proc.h" #include "storage/procarray.h" +#include "storage/shm_mq.h" +#include "storage/shm_toc.h" +#include "storage/spin.h" #include "tcop/tcopprot.h" #include "utils/acl.h" #include "utils/builtins.h" @@ -198,6 +218,59 @@ #define NAPTIME_PER_CYCLE 1000 /* max sleep time between cycles (1s) */ +#define PG_LOGICAL_APPLY_SHM_MAGIC 0x79fb2447 // TODO Consider change + +typedef struct ParallelState +{ + slock_t mutex; + bool attached; + bool ready; + bool finished; + bool failed; + Oid subid; + TransactionId stream_xid; + uint32 n; +} ParallelState; + +typedef struct WorkerState +{ + shm_mq_handle *mq_handle; + dsm_segment *dsm_seg; + ParallelState volatile *pstate; +} WorkerState; + +typedef struct WorkerEntry +{ + TransactionId xid; + WorkerState *wstate; +} WorkerEntry; + +/* Apply workers hash table (initialized on first use) */ +static HTAB *ApplyWorkersHash = NULL; +static List *ApplyWorkersIdleList = NIL; +static List *ApplyWorkersList = NIL; +static uint32 nworkers = 0; +static uint32 nfreeworkers = 0; + +/* Fields valid only for apply background workers */ +bool isLogicalApplyWorker = false; +volatile ParallelState *MyParallelState = NULL; +static List *subxactlist = NIL; + +/* Worker setup and interactions */ +static void setup_dsm(WorkerState *wstate); +static WorkerState *setup_background_worker(void); +static void wait_for_worker_ready(WorkerState *wstate, bool notify); +static void wait_for_transaction_finish(WorkerState *wstate); +static void send_data_to_worker(WorkerState *wstate, Size nbytes, + const void *data); +static WorkerState *find_or_start_worker(TransactionId xid, bool start); +static void free_stream_apply_worker(void); +static bool transaction_applied_in_bgworker(TransactionId xid); +static void check_workers_status(void); + +static uint32 nchanges = 0; + typedef struct FlushPosition { dlist_node node; @@ -260,18 +333,22 @@ static XLogRecPtr remote_final_lsn = InvalidXLogRecPtr; static bool in_streamed_transaction = false; static TransactionId stream_xid = InvalidTransactionId; +static WorkerState *stream_apply_worker = NULL; + +#define applying_changes_in_bgworker() (in_streamed_transaction && stream_apply_worker != NULL) /* * We enable skipping all data modification changes (INSERT, UPDATE, etc.) for * the subscription if the remote transaction's finish LSN matches the subskiplsn. * Once we start skipping changes, we don't stop it until we skip all changes of * the transaction even if pg_subscription is updated and MySubscription->skiplsn - * gets changed or reset during that. Also, in streaming transaction cases, we - * don't skip receiving and spooling the changes since we decide whether or not + * gets changed or reset during that. Also, in streaming transaction cases (spool mode), + * we don't skip receiving and spooling the changes since we decide whether or not * to skip applying the changes when starting to apply changes. The subskiplsn is * cleared after successfully skipping the transaction or applying non-empty * transaction. The latter prevents the mistakenly specified subskiplsn from - * being left. + * being left. Note that we cannot skip the streaming transaction in apply mode, + * because we cannot get the finish LSN before applying the changes. */ static XLogRecPtr skip_xact_finish_lsn = InvalidXLogRecPtr; #define is_skipping_changes() (unlikely(!XLogRecPtrIsInvalid(skip_xact_finish_lsn))) @@ -428,41 +505,96 @@ end_replication_step(void) /* * Handle streamed transactions. * - * If in streaming mode (receiving a block of streamed transaction), we - * simply redirect it to a file for the proper toplevel transaction. + * For the main apply worker, if in streaming mode (receiving a block of + * streamed transaction), we send the data to the background worker. + * + * For the background worker, define a savepoint if new subtransaction was + * started. * * Returns true for streamed transactions, false otherwise (regular mode). */ static bool handle_streamed_transaction(LogicalRepMsgType action, StringInfo s) { - TransactionId xid; + TransactionId current_xid = InvalidTransactionId; /* not in streaming mode */ - if (!in_streamed_transaction) + if (!in_streamed_transaction && !isLogicalApplyWorker) return false; - Assert(stream_fd != NULL); Assert(TransactionIdIsValid(stream_xid)); /* * We should have received XID of the subxact as the first part of the * message, so extract it. */ - xid = pq_getmsgint(s, 4); + current_xid = pq_getmsgint(s, 4); - if (!TransactionIdIsValid(xid)) + if (!TransactionIdIsValid(current_xid)) ereport(ERROR, (errcode(ERRCODE_PROTOCOL_VIOLATION), errmsg_internal("invalid transaction ID in streamed replication transaction"))); - /* Add the new subxact to the array (unless already there). */ - subxact_info_add(xid); + if (isLogicalApplyWorker) + { + /* + * Inside logical apply worker we can figure out that new + * subtransaction was started if new change arrived with different xid. + * In that case we can define named savepoint, so that we were able to + * commit/rollback it separately later. + * + * Special case is if the first change comes from subtransaction, then + * we check that current_xid differs from stream_xid. + */ + if (current_xid != stream_xid && + !list_member_int(subxactlist, (int) current_xid)) + { + MemoryContext oldctx; + char *spname = (char *) palloc(64 * sizeof(char)); + sprintf(spname, "savepoint_for_xid_%u", current_xid); + + elog(LOG, "[Apply BGW #%u] defining savepoint %s", MyParallelState->n, spname); + + DefineSavepoint(spname); + + CommitTransactionCommand(); + + oldctx = MemoryContextSwitchTo(ApplyContext); + subxactlist = lappend_int(subxactlist, (int) current_xid); + MemoryContextSwitchTo(oldctx); + } + } + else if (applying_changes_in_bgworker()) + { + /* + * If we decided to apply the changes of this transaction in a + * bgworker, pass the data to the worker. + */ + send_data_to_worker(stream_apply_worker, s->len, s->data); + nchanges += 1; + + /* + * XXX The publisher side don't always send relation update message + * after the streaming transaction, so update the relation in main + * worker here. + */ + if (action == LOGICAL_REP_MSG_RELATION) + { + LogicalRepRelation *rel = logicalrep_read_rel(s); + logicalrep_relmap_update(rel); + } + + } + else + { + /* Add the new subxact to the array (unless already there). */ + subxact_info_add(current_xid); - /* write the change to the current file */ - stream_write_change(action, s); + /* write the change to the current file */ + stream_write_change(action, s); + } - return true; + return !isLogicalApplyWorker; } /* @@ -844,6 +976,9 @@ apply_handle_commit(StringInfo s) apply_handle_commit_internal(&commit_data); + /* Check the status of bgworkers if any. */ + check_workers_status(); + /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(commit_data.end_lsn); @@ -898,7 +1033,9 @@ apply_handle_prepare_internal(LogicalRepPreparedTxnData *prepare_data) * BeginTransactionBlock is necessary to balance the EndTransactionBlock * called within the PrepareTransactionBlock below. */ - BeginTransactionBlock(); + if (!IsTransactionBlock()) + BeginTransactionBlock(); + CommitTransactionCommand(); /* Completes the preceding Begin command. */ /* @@ -950,6 +1087,9 @@ apply_handle_prepare(StringInfo s) in_remote_transaction = false; + /* Check the status of bgworkers if any. */ + check_workers_status(); + /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(prepare_data.end_lsn); @@ -976,30 +1116,51 @@ apply_handle_commit_prepared(StringInfo s) char gid[GIDSIZE]; logicalrep_read_commit_prepared(s, &prepare_data); + set_apply_error_context_xact(prepare_data.xid, prepare_data.commit_lsn); - /* Compute GID for two_phase transactions. */ - TwoPhaseTransactionGid(MySubscription->oid, prepare_data.xid, - gid, sizeof(gid)); + /* Check if we have prepared transaction in another bgworker */ + if (transaction_applied_in_bgworker(prepare_data.xid)) + { + elog(DEBUG1, "received commit for streamed transaction %u", prepare_data.xid); - /* There is no transaction when COMMIT PREPARED is called */ - begin_replication_step(); + /* Send commit message */ + send_data_to_worker(stream_apply_worker, s->len, s->data); - /* - * Update origin state so we can restart streaming from correct position - * in case of crash. - */ - replorigin_session_origin_lsn = prepare_data.end_lsn; - replorigin_session_origin_timestamp = prepare_data.commit_time; + /* Notify worker, that we are done with this xact */ + wait_for_transaction_finish(stream_apply_worker); + + free_stream_apply_worker(); + } + else + { + /* Compute GID for two_phase transactions. */ + TwoPhaseTransactionGid(MySubscription->oid, prepare_data.xid, + gid, sizeof(gid)); + + /* There is no transaction when COMMIT PREPARED is called */ + begin_replication_step(); + + /* + * Update origin state so we can restart streaming from correct position + * in case of crash. + */ + replorigin_session_origin_lsn = prepare_data.end_lsn; + replorigin_session_origin_timestamp = prepare_data.commit_time; + + FinishPreparedTransaction(gid, true); + end_replication_step(); + CommitTransactionCommand(); + } - FinishPreparedTransaction(gid, true); - end_replication_step(); - CommitTransactionCommand(); pgstat_report_stat(false); store_flush_position(prepare_data.end_lsn); in_remote_transaction = false; + /* Check the status of bgworkers if any. */ + check_workers_status(); + /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(prepare_data.end_lsn); @@ -1019,35 +1180,51 @@ apply_handle_rollback_prepared(StringInfo s) char gid[GIDSIZE]; logicalrep_read_rollback_prepared(s, &rollback_data); + set_apply_error_context_xact(rollback_data.xid, rollback_data.rollback_end_lsn); - /* Compute GID for two_phase transactions. */ - TwoPhaseTransactionGid(MySubscription->oid, rollback_data.xid, - gid, sizeof(gid)); + /* Check if we are processing the prepared transaction in a bgworker */ + if (transaction_applied_in_bgworker(rollback_data.xid)) + { + send_data_to_worker(stream_apply_worker, s->len, s->data); - /* - * It is possible that we haven't received prepare because it occurred - * before walsender reached a consistent point or the two_phase was still - * not enabled by that time, so in such cases, we need to skip rollback - * prepared. - */ - if (LookupGXact(gid, rollback_data.prepare_end_lsn, - rollback_data.prepare_time)) + /* Notify worker, that we are done with this xact */ + wait_for_transaction_finish(stream_apply_worker); + + elog(LOG, "rollback prepared streaming of xid %u", rollback_data.xid); + + free_stream_apply_worker(); + } + else { + /* Compute GID for two_phase transactions. */ + TwoPhaseTransactionGid(MySubscription->oid, rollback_data.xid, + gid, sizeof(gid)); + /* - * Update origin state so we can restart streaming from correct - * position in case of crash. + * It is possible that we haven't received prepare because it occurred + * before walsender reached a consistent point or the two_phase was still + * not enabled by that time, so in such cases, we need to skip rollback + * prepared. */ - replorigin_session_origin_lsn = rollback_data.rollback_end_lsn; - replorigin_session_origin_timestamp = rollback_data.rollback_time; + if (LookupGXact(gid, rollback_data.prepare_end_lsn, + rollback_data.prepare_time)) + { + /* + * Update origin state so we can restart streaming from correct + * position in case of crash. + */ + replorigin_session_origin_lsn = rollback_data.rollback_end_lsn; + replorigin_session_origin_timestamp = rollback_data.rollback_time; - /* There is no transaction when ABORT/ROLLBACK PREPARED is called */ - begin_replication_step(); - FinishPreparedTransaction(gid, false); - end_replication_step(); - CommitTransactionCommand(); + /* There is no transaction when ABORT/ROLLBACK PREPARED is called */ + begin_replication_step(); + FinishPreparedTransaction(gid, false); + end_replication_step(); + CommitTransactionCommand(); - clear_subscription_skip_lsn(rollback_data.rollback_end_lsn); + clear_subscription_skip_lsn(rollback_data.rollback_end_lsn); + } } pgstat_report_stat(false); @@ -1055,6 +1232,9 @@ apply_handle_rollback_prepared(StringInfo s) store_flush_position(rollback_data.rollback_end_lsn); in_remote_transaction = false; + /* Check the status of bgworkers if any. */ + check_workers_status(); + /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(rollback_data.rollback_end_lsn); @@ -1063,11 +1243,148 @@ apply_handle_rollback_prepared(StringInfo s) } /* - * Handle STREAM PREPARE. + * Look up worker inside ApplyWorkersHash for requested xid. * - * Logic is in two parts: - * 1. Replay all the spooled operations - * 2. Mark the transaction as prepared + * If start flag is true, try to start a new worker if not found in hash table. + */ +static WorkerState * +find_or_start_worker(TransactionId xid, bool start) +{ + bool found; + WorkerState *wstate; + WorkerEntry *entry = NULL; + + /* + * We don't start new background worker when stream option is off or spool. + */ + if (MySubscription->stream != SUBSTREAM_APPLY) + return NULL; + + /* + * We don't start new background worker if user has set skiplsn as it's + * possible that user want to skip the streaming transaction. For streaming + * transaction, we need to spill the transaction to disk so that we can get + * the last LSN of the transaction to judge whether to skip before starting + * to apply the change. + */ + else if (start && !XLogRecPtrIsInvalid(MySubscription->skiplsn)) + return NULL; + + /* + * For streaming transactions that is being applied in bgworker, we cannot + * decide whether to apply the change for a relation that is not in the + * READY state (see should_apply_changes_for_rel) as we won't know + * remote_final_lsn by that time. So, we don't start new bgworker in this + * case. + */ + else if (start && !AllTablesyncsReady()) + return NULL; + else if (!start && ApplyWorkersHash == NULL) + return NULL; + + /* First time through, initialize apply workers hashtable */ + if (ApplyWorkersHash == NULL) + { + HASHCTL ctl; + + MemSet(&ctl, 0, sizeof(ctl)); + ctl.keysize = sizeof(TransactionId); + ctl.entrysize = sizeof(WorkerEntry); + ctl.hcxt = ApplyContext; + + ApplyWorkersHash = hash_create("logical apply workers hash", 8, &ctl, + HASH_ELEM | HASH_BLOBS | HASH_CONTEXT); + } + + /* + * Find entry for requested transaction. + */ + entry = hash_search(ApplyWorkersHash, &xid, start ? HASH_ENTER : HASH_FIND, + &found); + if (found) + return entry->wstate; + else if (!start) + return NULL; + + /* If there is at least one worker in the idle list, then take one. */ + if (list_length(ApplyWorkersIdleList) > 0) + { + wstate = (WorkerState *) llast(ApplyWorkersIdleList); + ApplyWorkersIdleList = list_delete_last(ApplyWorkersIdleList); + } + else + { + wstate = setup_background_worker(); + + if (wstate == NULL) + { + /* + * If there is no more worker can be launched here, remove the + * entry in hash table. + */ + hash_search(ApplyWorkersHash, &xid, HASH_REMOVE, &found); + return NULL; + } + } + + /* Fill up the hash entry */ + wstate->pstate->finished = false; + wstate->pstate->stream_xid = xid; + entry->wstate = wstate; + entry->xid = xid; + + return wstate; +} + +/* + * Add the worker to the freelist and remove the entry from hash table. + */ +static void +free_stream_apply_worker(void) +{ + bool found; + MemoryContext oldctx; + TransactionId xid = stream_apply_worker->pstate->stream_xid; + + Assert(stream_apply_worker); + + oldctx = MemoryContextSwitchTo(ApplyContext); + + hash_search(ApplyWorkersHash, &xid, + HASH_REMOVE, &found); + + elog(LOG, "adding finished apply worker #%u for xid %u to the idle list", + stream_apply_worker->pstate->n, stream_apply_worker->pstate->stream_xid); + + ApplyWorkersIdleList = lappend(ApplyWorkersIdleList, stream_apply_worker); + stream_apply_worker = NULL; + + MemoryContextSwitchTo(oldctx); +} + +static void +serialize_stream_prepare(LogicalRepPreparedTxnData *prepare_data) +{ + /* Replay all the spooled operations. */ + apply_spooled_messages(prepare_data->xid, prepare_data->prepare_lsn); + + /* Mark the transaction as prepared. */ + apply_handle_prepare_internal(prepare_data); + + CommitTransactionCommand(); + + pgstat_report_stat(false); + + store_flush_position(prepare_data->end_lsn); + + in_remote_transaction = false; + + /* unlink the files with serialized changes and subxact info. */ + stream_cleanup_files(MyLogicalRepWorker->subid, prepare_data->xid); +} + +/* + * Handle STREAM PREPARE. */ static void apply_handle_stream_prepare(StringInfo s) @@ -1088,24 +1405,49 @@ apply_handle_stream_prepare(StringInfo s) logicalrep_read_stream_prepare(s, &prepare_data); set_apply_error_context_xact(prepare_data.xid, prepare_data.prepare_lsn); - elog(DEBUG1, "received prepare for streamed transaction %u", prepare_data.xid); + /* + * If we are in a bgworker, just prepare the transaction. + */ + if (isLogicalApplyWorker) + { + elog(LOG, "received prepare for streamed transaction %u", prepare_data.xid); - /* Replay all the spooled operations. */ - apply_spooled_messages(prepare_data.xid, prepare_data.prepare_lsn); + /* Mark the transaction as prepared. */ + apply_handle_prepare_internal(&prepare_data); - /* Mark the transaction as prepared. */ - apply_handle_prepare_internal(&prepare_data); + CommitTransactionCommand(); - CommitTransactionCommand(); + pgstat_report_stat(false); - pgstat_report_stat(false); + list_free(subxactlist); + subxactlist = NIL; + } - store_flush_position(prepare_data.end_lsn); + /* + * If we are in main apply worker, check if we are processing this + * transaction in a bgworker. + */ + else if (transaction_applied_in_bgworker(prepare_data.xid)) + { + send_data_to_worker(stream_apply_worker, s->len, s->data); + wait_for_worker_ready(stream_apply_worker, true); + + pgstat_report_stat(false); + store_flush_position(prepare_data.end_lsn); + } + + /* + * If we are in main apply worker and the transaction has been serialized + * to file, replay all the spooled operations. + */ + else + serialize_stream_prepare(&prepare_data); in_remote_transaction = false; + stream_apply_worker = NULL; - /* unlink the files with serialized changes and subxact info. */ - stream_cleanup_files(MyLogicalRepWorker->subid, prepare_data.xid); + /* Check the status of bgworkers if any. */ + check_workers_status(); /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(prepare_data.end_lsn); @@ -1142,19 +1484,9 @@ apply_handle_origin(StringInfo s) errmsg_internal("ORIGIN message sent out of order"))); } -/* - * Handle STREAM START message. - */ static void -apply_handle_stream_start(StringInfo s) +serialize_stream_start(bool first_segment) { - bool first_segment; - - if (in_streamed_transaction) - ereport(ERROR, - (errcode(ERRCODE_PROTOCOL_VIOLATION), - errmsg_internal("duplicate STREAM START message"))); - /* * Start a transaction on stream start, this transaction will be committed * on the stream stop unless it is a tablesync worker in which case it @@ -1164,19 +1496,6 @@ apply_handle_stream_start(StringInfo s) */ begin_replication_step(); - /* notify handle methods we're processing a remote transaction */ - in_streamed_transaction = true; - - /* extract XID of the top-level transaction */ - stream_xid = logicalrep_read_stream_start(s, &first_segment); - - if (!TransactionIdIsValid(stream_xid)) - ereport(ERROR, - (errcode(ERRCODE_PROTOCOL_VIOLATION), - errmsg_internal("invalid transaction ID in streamed replication transaction"))); - - set_apply_error_context_xact(stream_xid, InvalidXLogRecPtr); - /* * Initialize the worker's stream_fileset if we haven't yet. This will be * used for the entire duration of the worker so create it in a permanent @@ -1204,60 +1523,132 @@ apply_handle_stream_start(StringInfo s) if (!first_segment) subxact_info_read(MyLogicalRepWorker->subid, stream_xid); - pgstat_report_activity(STATE_RUNNING, NULL); - end_replication_step(); } /* - * Handle STREAM STOP message. + * Handle STREAM START message. */ static void -apply_handle_stream_stop(StringInfo s) +apply_handle_stream_start(StringInfo s) { - if (!in_streamed_transaction) + bool first_segment; + + if (in_streamed_transaction) ereport(ERROR, (errcode(ERRCODE_PROTOCOL_VIOLATION), - errmsg_internal("STREAM STOP message without STREAM START"))); - - /* - * Close the file with serialized changes, and serialize information about - * subxacts for the toplevel transaction. - */ - subxact_info_write(MyLogicalRepWorker->subid, stream_xid); - stream_close_file(); - - /* We must be in a valid transaction state */ - Assert(IsTransactionState()); - - /* Commit the per-stream transaction */ - CommitTransactionCommand(); - - in_streamed_transaction = false; - - /* Reset per-stream context */ - MemoryContextReset(LogicalStreamingContext); + errmsg_internal("duplicate STREAM START message"))); - pgstat_report_activity(STATE_IDLE, NULL); - reset_apply_error_context_info(); -} + /* notify handle methods we're processing a remote transaction */ + in_streamed_transaction = true; -/* - * Handle STREAM abort message. - */ -static void -apply_handle_stream_abort(StringInfo s) -{ - TransactionId xid; - TransactionId subxid; + /* extract XID of the top-level transaction */ + stream_xid = logicalrep_read_stream_start(s, &first_segment); - if (in_streamed_transaction) + if (!TransactionIdIsValid(stream_xid)) ereport(ERROR, (errcode(ERRCODE_PROTOCOL_VIOLATION), - errmsg_internal("STREAM ABORT message without STREAM STOP"))); + errmsg_internal("invalid transaction ID in streamed replication transaction"))); - logicalrep_read_stream_abort(s, &xid, &subxid); + set_apply_error_context_xact(stream_xid, InvalidXLogRecPtr); + if (isLogicalApplyWorker) + { + /* If we are in a bgworker, begin the transaction */ + AcceptInvalidationMessages(); + maybe_reread_subscription(); + + StartTransactionCommand(); + BeginTransactionBlock(); + CommitTransactionCommand(); + + return; + } + + /* + * If we are in main apply worker, check if there is any free bgworker + * we can use to process this transaction. + */ + stream_apply_worker = find_or_start_worker(stream_xid, first_segment); + + if (applying_changes_in_bgworker()) + { + /* + * If we have free worker or we already started to apply this + * transaction in bgworker, we pass the data to worker. + */ + if (first_segment) + send_data_to_worker(stream_apply_worker, s->len, s->data); + + nchanges = 0; + + SpinLockAcquire(&stream_apply_worker->pstate->mutex); + stream_apply_worker->pstate->ready = false; + SpinLockRelease(&stream_apply_worker->pstate->mutex); + + elog(LOG, "starting streaming of xid %u", stream_xid); + } + + /* + * If no worker is available for the first stream start, we start to + * serialize all the changes of the transaction. + */ + else + serialize_stream_start(first_segment); + + pgstat_report_activity(STATE_RUNNING, NULL); +} + +static void +serialize_stream_stop() +{ + /* + * Close the file with serialized changes, and serialize information about + * subxacts for the toplevel transaction. + */ + subxact_info_write(MyLogicalRepWorker->subid, stream_xid); + stream_close_file(); + + /* We must be in a valid transaction state */ + Assert(IsTransactionState()); + + /* Commit the per-stream transaction */ + CommitTransactionCommand(); + + /* Reset per-stream context */ + MemoryContextReset(LogicalStreamingContext); +} + +/* + * Handle STREAM STOP message. + */ +static void +apply_handle_stream_stop(StringInfo s) +{ + if (!in_streamed_transaction) + ereport(ERROR, + (errcode(ERRCODE_PROTOCOL_VIOLATION), + errmsg_internal("STREAM STOP message without STREAM START"))); + + if (applying_changes_in_bgworker()) + { + wait_for_worker_ready(stream_apply_worker, true); + + elog(LOG, "stopped streaming of xid %u, %u changes streamed", stream_xid, nchanges); + } + else + serialize_stream_stop(); + + in_streamed_transaction = false; + stream_apply_worker = NULL; + + pgstat_report_activity(STATE_IDLE, NULL); + reset_apply_error_context_info(); +} + +static void +serialize_stream_abort(TransactionId xid, TransactionId subxid) +{ /* * If the two XIDs are the same, it's in fact abort of toplevel xact, so * just delete the files with serialized info. @@ -1339,8 +1730,114 @@ apply_handle_stream_abort(StringInfo s) end_replication_step(); CommitTransactionCommand(); } +} - reset_apply_error_context_info(); +/* + * Handle STREAM ABORT message. + */ +static void +apply_handle_stream_abort(StringInfo s) +{ + TransactionId xid; + TransactionId subxid; + + if (in_streamed_transaction) + ereport(ERROR, + (errcode(ERRCODE_PROTOCOL_VIOLATION), + errmsg_internal("STREAM COMMIT message without STREAM STOP"))); + + logicalrep_read_stream_abort(s, &xid, &subxid); + + if (isLogicalApplyWorker) + { + ereport(LOG, + (errcode_for_file_access(), + errmsg("[Apply BGW #%u] aborting current transaction xid=%u, subxid=%u", + MyParallelState->n, GetCurrentTransactionIdIfAny(), GetCurrentSubTransactionId()))); + + /* + * If the two XIDs are the same, it's in fact abort of toplevel xact, + * so just free the subxactlist. + */ + if (subxid == xid) + { + set_apply_error_context_xact(subxid, InvalidXLogRecPtr); + + AbortCurrentTransaction(); + + EndTransactionBlock(false); + CommitTransactionCommand(); + + list_free(subxactlist); + subxactlist = NIL; + } + else + { + /* + * OK, so it's a subxact. Rollback to the savepoint. + * + * We also need to read the subxactlist, determine the offset + * tracked for the subxact, and truncate the list. + */ + int i; + bool found = false; + char *spname = (char *) palloc(64 * sizeof(char)); + + set_apply_error_context_xact(subxid, InvalidXLogRecPtr); + + sprintf(spname, "savepoint_for_xid_%u", subxid); + + ereport(LOG, + (errcode_for_file_access(), + errmsg("[Apply BGW #%u] rolling back to savepoint %s", MyParallelState->n, spname))); + + for(i = list_length(subxactlist) - 1; i >= 0; i--) + { + xid = (TransactionId) list_nth_int(subxactlist, i); + if (xid == subxid) + { + found = true; + break; + } + } + + if (found) + { + elog(LOG, "rolled back to savepoint %s", spname); + RollbackToSavepoint(spname); + CommitTransactionCommand(); + subxactlist = list_truncate(subxactlist, i + 1); + } + + pfree(spname); + } + + reset_apply_error_context_info(); + } + + /* + * If we are in main apply worker, check if we are processing this + * transaction in a bgworker. + */ + else if (transaction_applied_in_bgworker(xid)) + { + send_data_to_worker(stream_apply_worker, s->len, s->data); + + if (subxid == xid) + { + wait_for_transaction_finish(stream_apply_worker); + free_stream_apply_worker(); + } + else + wait_for_worker_ready(stream_apply_worker, true); + } + + /* + * We are in main apply worker and the transaction has been serialized + * to file. + */ + else + serialize_stream_abort(xid, subxid); } /* @@ -1463,40 +1960,6 @@ apply_spooled_messages(TransactionId xid, XLogRecPtr lsn) } /* - * Handle STREAM COMMIT message. - */ -static void -apply_handle_stream_commit(StringInfo s) -{ - TransactionId xid; - LogicalRepCommitData commit_data; - - if (in_streamed_transaction) - ereport(ERROR, - (errcode(ERRCODE_PROTOCOL_VIOLATION), - errmsg_internal("STREAM COMMIT message without STREAM STOP"))); - - xid = logicalrep_read_stream_commit(s, &commit_data); - set_apply_error_context_xact(xid, commit_data.commit_lsn); - - elog(DEBUG1, "received commit for streamed transaction %u", xid); - - apply_spooled_messages(xid, commit_data.commit_lsn); - - apply_handle_commit_internal(&commit_data); - - /* unlink the files with serialized changes and subxact info */ - stream_cleanup_files(MyLogicalRepWorker->subid, xid); - - /* Process any tables that are being synchronized in parallel. */ - process_syncing_tables(commit_data.end_lsn); - - pgstat_report_activity(STATE_IDLE, NULL); - - reset_apply_error_context_info(); -} - -/* * Helper function for apply_handle_commit and apply_handle_stream_commit. */ static void @@ -2445,6 +2908,93 @@ apply_handle_truncate(StringInfo s) end_replication_step(); } +/* + * Handle STREAM COMMIT message. + */ +static void +apply_handle_stream_commit(StringInfo s) +{ + LogicalRepCommitData commit_data; + TransactionId xid; + + if (in_streamed_transaction) + ereport(ERROR, + (errcode(ERRCODE_PROTOCOL_VIOLATION), + errmsg_internal("STREAM COMMIT message without STREAM STOP"))); + + xid = pq_getmsgint(s, 4); + logicalrep_read_stream_commit(s, &commit_data); + set_apply_error_context_xact(xid, commit_data.commit_lsn); + + elog(DEBUG1, "received commit for streamed transaction %u", xid); + + if (isLogicalApplyWorker) + { + /* + * Update origin state so we can restart streaming from correct + * position in case of crash. + */ + replorigin_session_origin_lsn = commit_data.end_lsn; + replorigin_session_origin_timestamp = commit_data.committime; + + CommitTransactionCommand(); + EndTransactionBlock(false); + CommitTransactionCommand(); + + pgstat_report_stat(false); + + list_free(subxactlist); + subxactlist = NIL; + } + + /* + * If we are in main apply worker, check if we are processing this + * transaction in a bgworker. + */ + else if (transaction_applied_in_bgworker(xid)) + { + /* Send commit message */ + send_data_to_worker(stream_apply_worker, s->len, s->data); + + /* Notify worker, that we are done with this xact */ + wait_for_transaction_finish(stream_apply_worker); + + pgstat_report_stat(false); + store_flush_position(commit_data.end_lsn); + stop_skipping_changes(); + + free_stream_apply_worker(); + + /* + * The transaction is either non-empty or skipped, so we clear the + * subskiplsn. + */ + clear_subscription_skip_lsn(commit_data.commit_lsn); + } + else + { + /* + * If we are in main apply worker and the transaction has been + * serialized to file, replay all the spooled operations. + */ + apply_spooled_messages(xid, commit_data.commit_lsn); + + apply_handle_commit_internal(&commit_data); + + /* unlink the files with serialized changes and subxact info */ + stream_cleanup_files(MyLogicalRepWorker->subid, xid); + } + + /* Check the status of bgworkers if any. */ + check_workers_status(); + + /* Process any tables that are being synchronized in parallel. */ + process_syncing_tables(commit_data.end_lsn); + + pgstat_report_activity(STATE_IDLE, NULL); + + reset_apply_error_context_info(); +} /* * Logical replication protocol message dispatcher. @@ -2511,6 +3061,7 @@ apply_dispatch(StringInfo s) break; case LOGICAL_REP_MSG_STREAM_START: + elog(LOG, "LOGICAL_REP_MSG_STREAM_START"); apply_handle_stream_start(s); break; @@ -2618,6 +3169,10 @@ store_flush_position(XLogRecPtr remote_lsn) { FlushPosition *flushpos; + /* We only need to collect the LSN in main apply worker */ + if (isLogicalApplyWorker) + return; + /* Need to do this in permanent context */ MemoryContextSwitchTo(ApplyContext); @@ -2794,6 +3349,9 @@ LogicalRepApplyLoop(XLogRecPtr last_received) AcceptInvalidationMessages(); maybe_reread_subscription(); + /* Check the status of bgworkers if any. */ + check_workers_status(); + /* Process any table synchronization changes. */ process_syncing_tables(last_received); } @@ -3467,6 +4025,7 @@ TwoPhaseTransactionGid(Oid subid, TransactionId xid, char *gid, int szgid) snprintf(gid, szgid, "pg_gid_%u_%u", subid, xid); } + /* * Execute the initial sync with error handling. Disable the subscription, * if it's required. @@ -3686,7 +4245,7 @@ ApplyWorkerMain(Datum main_arg) originid = replorigin_by_name(originname, true); if (!OidIsValid(originid)) originid = replorigin_create(originname); - replorigin_session_setup(originid); + replorigin_session_setup(originid, true); replorigin_session_origin = originid; origin_startpos = replorigin_session_get_progress(false); CommitTransactionCommand(); @@ -3733,7 +4292,7 @@ ApplyWorkerMain(Datum main_arg) options.proto.logical.publication_names = MySubscription->publications; options.proto.logical.binary = MySubscription->binary; - options.proto.logical.streaming = MySubscription->stream; + options.proto.logical.streaming = (MySubscription->stream != SUBSTREAM_OFF); options.proto.logical.twophase = false; if (!am_tablesync_worker()) @@ -3891,7 +4450,8 @@ clear_subscription_skip_lsn(XLogRecPtr finish_lsn) XLogRecPtr myskiplsn = MySubscription->skiplsn; bool started_tx = false; - if (likely(XLogRecPtrIsInvalid(myskiplsn))) + if (likely(XLogRecPtrIsInvalid(myskiplsn)) || + isLogicalApplyWorker) return; if (!IsTransactionState()) @@ -4027,3 +4587,643 @@ reset_apply_error_context_info(void) apply_error_callback_arg.remote_attnum = -1; set_apply_error_context_xact(InvalidTransactionId, InvalidXLogRecPtr); } + +/* Apply Background Worker main loop */ +static void +LogicalApplyBgwLoop(shm_mq_handle *mqh, volatile ParallelState *pst) +{ + shm_mq_result shmq_res; + PGPROC *registrant; + ErrorContextCallback errcallback; + XLogRecPtr last_received = InvalidXLogRecPtr; + + registrant = BackendPidGetProc(MyBgworkerEntry->bgw_notify_pid); + SetLatch(®istrant->procLatch); + + /* + * Push apply error context callback. Fields will be filled during + * applying a change. + */ + errcallback.callback = apply_error_callback; + errcallback.previous = error_context_stack; + error_context_stack = &errcallback; + + for (;;) + { + void *data; + Size len; + StringInfoData s; + MemoryContext oldctx; + + CHECK_FOR_INTERRUPTS(); + + /* Ensure we are reading the data into our memory context. */ + oldctx = MemoryContextSwitchTo(ApplyMessageContext); + + shmq_res = shm_mq_receive(mqh, &len, &data, false); + + if (shmq_res != SHM_MQ_SUCCESS) + break; + + if (len == 0) + { + elog(LOG, "[Apply BGW #%u] got zero-length message, stopping", pst->n); + break; + } + else + { + XLogRecPtr start_lsn; + XLogRecPtr end_lsn; + TimestampTz send_time; + + s.cursor = 0; + s.maxlen = -1; + s.data = (char *) data; + s.len = len; + + /* + * We use first byte of message for additional communication between + * main Logical replication worker and Apply BGWorkers, so if it + * differs from 'w', then process it first. + */ + switch (pq_getmsgbyte(&s)) + { + /* Stream stop */ + case 'E': + { + elog(LOG, "[Apply BGW #%u] ended processing streaming chunk," + "waiting on shm_mq_receive", pst->n); + + SpinLockAcquire(&pst->mutex); + pst->ready = true; + SpinLockRelease(&pst->mutex); + + SetLatch(®istrant->procLatch); + + in_streamed_transaction = false; + + pgstat_report_activity(STATE_IDLE, NULL); + + continue; + } + /* Finished processing xact */ + case 'F': + { + elog(LOG, "[Apply BGW #%u] finished processing xact %u", pst->n, stream_xid); + + SpinLockAcquire(&pst->mutex); + pst->finished = true; + SpinLockRelease(&pst->mutex); + + in_remote_transaction = false; + + continue; + } + default: + break; + } + + start_lsn = pq_getmsgint64(&s); + end_lsn = pq_getmsgint64(&s); + send_time = pq_getmsgint64(&s); + + if (last_received < start_lsn) + last_received = start_lsn; + + if (last_received < end_lsn) + last_received = end_lsn; + + /* + * TO IMPROVE: Do we need to display the bgworker's information in + * pg_stat_replication ? + */ + UpdateWorkerStats(last_received, send_time, false); + + /* + * Make sure the handle apply_dispatch methods are aware we're in a remote + * transaction. + */ + in_remote_transaction = true; + + apply_dispatch(&s); + + if (ConfigReloadPending) + { + ConfigReloadPending = false; + ProcessConfigFile(PGC_SIGHUP); + } + } + + MemoryContextSwitchTo(oldctx); + MemoryContextReset(ApplyMessageContext); + } + + MemoryContextSwitchTo(TopMemoryContext); + MemoryContextReset(ApplyContext); + + /* Pop the error context stack */ + error_context_stack = errcallback.previous; + + elog(LOG, "[Apply BGW #%u] exiting", pst->n); + + /* Signal main process that we are done. */ + SetLatch(®istrant->procLatch); +} + +/* + * Set the failed flag so that the main apply worker can realize we have + * shutdown. + */ +static void +ApplyBgwShutdown(int code, Datum arg) +{ + SpinLockAcquire(&MyParallelState->mutex); + MyParallelState->failed = true; + SpinLockRelease(&MyParallelState->mutex); + + dsm_detach((dsm_segment *) DatumGetPointer(arg)); +} + +/* + * Apply Background Worker entry point + */ +void +LogicalApplyBgwMain(Datum main_arg) +{ + volatile ParallelState *pst; + + dsm_handle handle; + dsm_segment *seg; + shm_toc *toc; + shm_mq *mq; + shm_mq_handle *mqh; + MemoryContext oldcontext; + RepOriginId originid; + int worker_slot = DatumGetInt32(main_arg); + char originname[NAMEDATALEN]; + + /* Attach to slot */ + logicalrep_worker_attach(worker_slot); + + MemoryContextSwitchTo(TopMemoryContext); + + /* Load the subscription into persistent memory context. */ + ApplyContext = AllocSetContextCreate(TopMemoryContext, + "ApplyContext", + ALLOCSET_DEFAULT_SIZES); + + /* + * Init the ApplyMessageContext which we clean up after each replication + * protocol message. + */ + ApplyMessageContext = AllocSetContextCreate(ApplyContext, + "ApplyMessageContext", + ALLOCSET_DEFAULT_SIZES); + + isLogicalApplyWorker = true; + + /* Setup signal handling */ + pqsignal(SIGHUP, SignalHandlerForConfigReload); + pqsignal(SIGTERM, die); + BackgroundWorkerUnblockSignals(); + + /* + * Connect to the dynamic shared memory segment. + * + * The backend that registered this worker passed us the ID of a shared + * memory segment to which we must attach for further instructions. In + * order to attach to dynamic shared memory, we need a resource owner. + * Once we've mapped the segment in our address space, attach to the table + * of contents so we can locate the various data structures we'll need to + * find within the segment. + */ + CurrentResourceOwner = ResourceOwnerCreate(NULL, "Logical apply worker"); + memcpy(&handle, MyBgworkerEntry->bgw_extra, sizeof(dsm_handle)); + seg = dsm_attach(handle); + if (seg == NULL) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("unable to map dynamic shared memory segment"))); + toc = shm_toc_attach(PG_LOGICAL_APPLY_SHM_MAGIC, dsm_segment_address(seg)); + if (toc == NULL) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("bad magic number in dynamic shared memory segment"))); + + before_shmem_exit(ApplyBgwShutdown, PointerGetDatum(seg)); + + /* + * Acquire a worker number. + * + * By convention, the process registering this background worker should + * have stored the control structure at key 0. We look up that key to + * find it. Our worker number gives our identity: there may be just one + * worker involved in this parallel operation, or there may be many. + */ + pst = shm_toc_lookup(toc, 0, false); + MyParallelState = pst; + + SpinLockAcquire(&pst->mutex); + pst->attached = true; + SpinLockRelease(&pst->mutex); + + /* + * Attach to the message queue. + */ + mq = shm_toc_lookup(toc, 1, false); + shm_mq_set_receiver(mq, MyProc); + mqh = shm_mq_attach(mq, seg, NULL); + + /* Connect to our database. */ + BackgroundWorkerInitializeConnectionByOid(MyLogicalRepWorker->dbid, + MyLogicalRepWorker->userid, + 0); + + /* + * Set the client encoding to the database encoding, since that is what + * the leader will expect. + */ + SetClientEncoding(GetDatabaseEncoding()); + + stream_xid = pst->stream_xid; + + StartTransactionCommand(); + oldcontext = MemoryContextSwitchTo(ApplyContext); + + MySubscription = GetSubscription(MyLogicalRepWorker->subid, true); + if (!MySubscription) + { + ereport(LOG, + (errmsg("logical replication apply worker for subscription %u will not " + "start because the subscription was removed during startup", + MyLogicalRepWorker->subid))); + proc_exit(0); + } + + MySubscriptionValid = true; + MemoryContextSwitchTo(oldcontext); + + /* Setup synchronous commit according to the user's wishes */ + SetConfigOption("synchronous_commit", MySubscription->synccommit, + PGC_BACKEND, PGC_S_OVERRIDE); + + /* Keep us informed about subscription changes. */ + CacheRegisterSyscacheCallback(SUBSCRIPTIONOID, + subscription_change_cb, + (Datum) 0); + + CommitTransactionCommand(); + + /* Setup replication origin tracking. */ + StartTransactionCommand(); + snprintf(originname, sizeof(originname), "pg_%u", MySubscription->oid); + originid = replorigin_by_name(originname, true); + if (!OidIsValid(originid)) + originid = replorigin_create(originname); + replorigin_session_setup(originid, false); + replorigin_session_origin = originid; + CommitTransactionCommand(); + + /* + * Allocate the origin name in long-lived context for error context + * message. + */ + apply_error_callback_arg.origin_name = MemoryContextStrdup(ApplyContext, + originname); + + /* + * Indicate that we're fully initialized and ready to begin the main part + * of the parallel operation. + * + * Once we signal that we're ready, the user backend is entitled to assume + * that our on_dsm_detach callbacks will fire before we disconnect from + * the shared memory segment and exit. Generally, that means we must have + * attached to all relevant dynamic shared memory data structures by now. + */ + SpinLockAcquire(&pst->mutex); + pst->ready = true; + SpinLockRelease(&pst->mutex); + + elog(LOG, "[Apply BGW #%u] started", pst->n); + + PG_TRY(); + { + LogicalApplyBgwLoop(mqh, pst); + } + PG_CATCH(); + { + /* + * Report the worker failed while applying streaming transaction + * changes. Abort the current transaction so that the stats message is + * sent in an idle state. + */ + AbortOutOfAnyTransaction(); + pgstat_report_subscription_error(MySubscription->oid, false); + + PG_RE_THROW(); + } + PG_END_TRY(); + + /* + * We're done. Explicitly detach the shared memory segment so that we + * don't get a resource leak warning at commit time. This will fire any + * on_dsm_detach callbacks we've registered, as well. Once that's done, + * we can go ahead and exit. + */ + dsm_detach(seg); + proc_exit(0); +} + +/* + * Set up a dynamic shared memory segment. + * + * We set up a control region that contains a ParallelState, + * plus one region per message queue. There are as many message queues as + * the number of workers. + */ +static void +setup_dsm(WorkerState *wstate) +{ + shm_toc_estimator e; + int toc_key = 0; + Size segsize; + dsm_segment *seg; + shm_toc *toc; + ParallelState *pst; + shm_mq *mq; + int64 queue_size = 160000000; /* 16 MB for now */ + + /* + * Estimate how much shared memory we need. + * + * Because the TOC machinery may choose to insert padding of oddly-sized + * requests, we must estimate each chunk separately. + * + * We need one key to register the location of the header, and we need + * nworkers keys to track the locations of the message queues. + */ + shm_toc_initialize_estimator(&e); + shm_toc_estimate_chunk(&e, sizeof(ParallelState)); + shm_toc_estimate_chunk(&e, (Size) queue_size); + + shm_toc_estimate_keys(&e, 1 + 1); + segsize = shm_toc_estimate(&e); + + /* Create the shared memory segment and establish a table of contents. */ + seg = dsm_create(shm_toc_estimate(&e), 0); + toc = shm_toc_create(PG_LOGICAL_APPLY_SHM_MAGIC, dsm_segment_address(seg), + segsize); + + /* Set up the header region. */ + pst = shm_toc_allocate(toc, sizeof(ParallelState)); + SpinLockInit(&pst->mutex); + pst->attached = false; + pst->ready = false; + pst->finished = false; + pst->failed = false; + pst->subid = MyLogicalRepWorker->subid; + pst->stream_xid = stream_xid; + pst->n = nworkers + 1; + + shm_toc_insert(toc, toc_key++, pst); + + /* Set up one message queue per worker, plus one. */ + mq = shm_mq_create(shm_toc_allocate(toc, (Size) queue_size), + (Size) queue_size); + shm_toc_insert(toc, toc_key++, mq); + shm_mq_set_sender(mq, MyProc); + + /* Attach the queues. */ + wstate->mq_handle = shm_mq_attach(mq, seg, NULL); + + /* Return results to caller. */ + wstate->dsm_seg = seg; + wstate->pstate = pst; +} + +/* + * Register background workers. + */ +static WorkerState * +setup_background_worker(void) +{ + MemoryContext oldcontext; + bool launched; + WorkerState *wstate; + + elog(LOG, "setting up apply worker #%u", nworkers + 1); + + oldcontext = MemoryContextSwitchTo(ApplyContext); + + wstate = (WorkerState *) palloc0(sizeof(WorkerState)); + + /* Setup shared memory */ + setup_dsm(wstate); + + launched = logicalrep_worker_launch(MyLogicalRepWorker->dbid, + MySubscription->oid, + MySubscription->name, + MyLogicalRepWorker->userid, + InvalidOid, + dsm_segment_handle(wstate->dsm_seg)); + + if (launched) + { + /* Wait for worker to become ready. */ + wait_for_worker_ready(wstate, false); + + ApplyWorkersList = lappend(ApplyWorkersList, wstate); + nworkers += 1; + } + else + { + pfree(wstate); + wstate = NULL; + } + + MemoryContextSwitchTo(oldcontext); + + return wstate; +} + +/* + * Send the data to worker via shared-memory queue. + */ +static void +send_data_to_worker(WorkerState *wstate, Size nbytes, const void *data) +{ + shm_mq_result result; + + result = shm_mq_send(wstate->mq_handle, nbytes, data, false, true); + + if (result != SHM_MQ_SUCCESS) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("could not send tuple to shared-memory queue"))); +} + +static void +wait_for_worker_ready(WorkerState *wstate, bool notify) +{ + bool result = false; + + if (notify) + { + char action = 'E'; + + /* Notify worker that we received STREAM STOP */ + send_data_to_worker(wstate, 1, &action); + } + + for (;;) + { + bool ready; + bool failed; + + /* If the worker is ready, we have succeeded. */ + SpinLockAcquire(&wstate->pstate->mutex); + ready = wstate->pstate->ready; + failed = wstate->pstate->failed; + SpinLockRelease(&wstate->pstate->mutex); + + if (ready) + { + result = true; + break; + } + + /* If any workers (or the postmaster) have died, we have failed. */ + if (failed) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("Background worker %u failed to apply transaction %u", + wstate->pstate->n, wstate->pstate->stream_xid))); + + /* Wait to be signalled. */ + WaitLatch(MyLatch, WL_LATCH_SET | WL_EXIT_ON_PM_DEATH, 0, + WAIT_EVENT_LOGICAL_APPLY_WORKER_READY); + + /* Reset the latch so we don't spin. */ + ResetLatch(MyLatch); + + /* An interrupt may have occurred while we were waiting. */ + CHECK_FOR_INTERRUPTS(); + } + + if (!result) + ereport(ERROR, + (errcode(ERRCODE_INSUFFICIENT_RESOURCES), + errmsg("one or more background workers failed to start"))); +} + +static void +wait_for_transaction_finish(WorkerState *wstate) +{ + char action = 'F'; + + /* Notify worker that the streaming transaction has finished */ + send_data_to_worker(wstate, 1, &action); + + elog(LOG, "waiting for apply worker #%u to finish processing xid %u", + wstate->pstate->n, wstate->pstate->stream_xid); + + for (;;) + { + bool finished; + bool failed; + + /* If the worker is finished, we have succeeded. */ + SpinLockAcquire(&wstate->pstate->mutex); + finished = wstate->pstate->finished; + failed = wstate->pstate->failed; + SpinLockRelease(&wstate->pstate->mutex); + + if (failed) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("Background worker %u failed to apply transaction %u", + wstate->pstate->n, wstate->pstate->stream_xid))); + + if (finished) + { + break; + } + + /* + * TO IMPROVE. Would it be better to use ConditionVariable instead of + * Wait/Reset Latch here? + */ + + /* Wait to be signalled. */ + WaitLatch(MyLatch, WL_LATCH_SET | WL_EXIT_ON_PM_DEATH, 0, + WAIT_EVENT_LOGICAL_APPLY_WORKER_READY); + + /* Reset the latch so we don't spin. */ + ResetLatch(MyLatch); + + /* An interrupt may have occurred while we were waiting. */ + CHECK_FOR_INTERRUPTS(); + } +} + +/* + * Check if the streamed transaction was being processed in a bgworker. + */ +static bool +transaction_applied_in_bgworker(TransactionId xid) +{ + if (!TransactionIdIsValid(xid) || isLogicalApplyWorker || + ApplyWorkersHash == NULL) + return false; + + stream_apply_worker = find_or_start_worker(xid, false); + + return stream_apply_worker != NULL; +} + +/* + * Check the status of workers and report an error if any bgworker exit + * unexpectedly. + * + * Exit if any relation is not in the READY state and if any worker is handling + * the streaming transaction at the same time. Because for streaming + * transactions that is being applied in bgworker, we cannot decide whether to + * apply the change for a relation that is not in the READY state (see + * should_apply_changes_for_rel) as we won't know remote_final_lsn by that + * time. + */ +static void +check_workers_status(void) +{ + ListCell *lc; + + if (isLogicalApplyWorker || MySubscription->stream != SUBSTREAM_APPLY) + return; + + foreach(lc, ApplyWorkersList) + { + WorkerState *wstate = (WorkerState *) lfirst(lc); + + /* + * We don't lock here as in the worst case we will just detect the + * failure of worker a bit later. + */ + if (wstate->pstate->failed) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("Background worker %u exited unexpectedly", + wstate->pstate->n))); + } + + if (!AllTablesyncsReady() && nfreeworkers != list_length(ApplyWorkersList)) + { + ereport(LOG, + (errmsg("logical replication apply workers for subscription \"%s\" will restart", + MySubscription->name), + errdetail("Cannot start table synchronization while bgworkers are " + "handling streamed replication transaction"))); + + proc_exit(0); + } + +} diff --git a/src/backend/utils/activity/wait_event.c b/src/backend/utils/activity/wait_event.c index 87c15b9..24a6fac 100644 --- a/src/backend/utils/activity/wait_event.c +++ b/src/backend/utils/activity/wait_event.c @@ -388,6 +388,9 @@ pgstat_get_wait_ipc(WaitEventIPC w) case WAIT_EVENT_HASH_GROW_BUCKETS_REINSERT: event_name = "HashGrowBucketsReinsert"; break; + case WAIT_EVENT_LOGICAL_APPLY_WORKER_READY: + event_name = "LogicalApplyWorkerReady"; + break; case WAIT_EVENT_LOGICAL_SYNC_DATA: event_name = "LogicalSyncData"; break; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 969e2a7..180fc1c 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -4391,7 +4391,7 @@ getSubscriptions(Archive *fout) if (fout->remoteVersion >= 140000) appendPQExpBufferStr(query, " s.substream,\n"); else - appendPQExpBufferStr(query, " false AS substream,\n"); + appendPQExpBufferStr(query, " 'f' AS substream,\n"); if (fout->remoteVersion >= 150000) appendPQExpBufferStr(query, @@ -4521,8 +4521,10 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo) if (strcmp(subinfo->subbinary, "t") == 0) appendPQExpBufferStr(query, ", binary = true"); - if (strcmp(subinfo->substream, "f") != 0) - appendPQExpBufferStr(query, ", streaming = on"); + if (strcmp(subinfo->substream, "s") == 0) + appendPQExpBufferStr(query, ", streaming = spool"); + else if (strcmp(subinfo->substream, "a") == 0) + appendPQExpBufferStr(query, ", streaming = apply"); if (strcmp(subinfo->subtwophasestate, two_phase_disabled) != 0) appendPQExpBufferStr(query, ", two_phase = on"); diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h index f006a92..1cd176a 100644 --- a/src/include/catalog/pg_subscription.h +++ b/src/include/catalog/pg_subscription.h @@ -68,7 +68,7 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW bool subbinary; /* True if the subscription wants the * publisher to send data in binary */ - bool substream; /* Stream in-progress transactions. */ + char substream; /* Stream in-progress transactions. */ char subtwophasestate; /* Stream two-phase transactions */ @@ -111,7 +111,7 @@ typedef struct Subscription bool enabled; /* Indicates if the subscription is enabled */ bool binary; /* Indicates if the subscription wants data in * binary format */ - bool stream; /* Allow streaming in-progress transactions. */ + char stream; /* Allow streaming in-progress transactions. */ char twophasestate; /* Allow streaming two-phase transactions */ bool disableonerr; /* Indicates if the subscription should be * automatically disabled if a worker error @@ -122,6 +122,18 @@ typedef struct Subscription List *publications; /* List of publication names to subscribe to */ } Subscription; +/* Disallow streaming in-progress transactions */ +#define SUBSTREAM_OFF 'f' + +/* + * Streaming transactions are written to a temporary file and applied only + * after the transaction is committed on upstream. + */ +#define SUBSTREAM_SPOOL 's' + +/* Streaming transactions are appied immediately via a background worker */ +#define SUBSTREAM_APPLY 'a' + extern Subscription *GetSubscription(Oid subid, bool missing_ok); extern void FreeSubscription(Subscription *sub); extern void DisableSubscription(Oid subid); diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h index a771ab8..0116867 100644 --- a/src/include/replication/logicalproto.h +++ b/src/include/replication/logicalproto.h @@ -243,8 +243,10 @@ extern TransactionId logicalrep_read_stream_start(StringInfo in, extern void logicalrep_write_stream_stop(StringInfo out); extern void logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn, XLogRecPtr commit_lsn); -extern TransactionId logicalrep_read_stream_commit(StringInfo out, +extern TransactionId logicalrep_read_stream_commit_old(StringInfo out, LogicalRepCommitData *commit_data); +extern void logicalrep_read_stream_commit(StringInfo out, + LogicalRepCommitData *commit_data); extern void logicalrep_write_stream_abort(StringInfo out, TransactionId xid, TransactionId subxid); extern void logicalrep_read_stream_abort(StringInfo in, TransactionId *xid, diff --git a/src/include/replication/logicalworker.h b/src/include/replication/logicalworker.h index cd1b6e8..3bb3511 100644 --- a/src/include/replication/logicalworker.h +++ b/src/include/replication/logicalworker.h @@ -13,6 +13,7 @@ #define LOGICALWORKER_H extern void ApplyWorkerMain(Datum main_arg); +extern void LogicalApplyBgwMain(Datum main_arg); extern bool IsLogicalWorker(void); diff --git a/src/include/replication/origin.h b/src/include/replication/origin.h index 14d5c49..1ed51bc 100644 --- a/src/include/replication/origin.h +++ b/src/include/replication/origin.h @@ -53,7 +53,7 @@ extern XLogRecPtr replorigin_get_progress(RepOriginId node, bool flush); extern void replorigin_session_advance(XLogRecPtr remote_commit, XLogRecPtr local_commit); -extern void replorigin_session_setup(RepOriginId node); +extern void replorigin_session_setup(RepOriginId node, bool acquire); extern void replorigin_session_reset(void); extern XLogRecPtr replorigin_session_get_progress(bool flush); diff --git a/src/include/replication/worker_internal.h b/src/include/replication/worker_internal.h index 4485d4e..4d11a13 100644 --- a/src/include/replication/worker_internal.h +++ b/src/include/replication/worker_internal.h @@ -60,6 +60,8 @@ typedef struct LogicalRepWorker */ FileSet *stream_fileset; + bool subworker; + /* Stats. */ XLogRecPtr last_lsn; TimestampTz last_send_time; @@ -84,8 +86,9 @@ extern void logicalrep_worker_attach(int slot); extern LogicalRepWorker *logicalrep_worker_find(Oid subid, Oid relid, bool only_running); extern List *logicalrep_workers_find(Oid subid, bool only_running); -extern void logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, - Oid userid, Oid relid); +extern bool logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, + Oid userid, Oid relid, + dsm_handle subworker_dsm); extern void logicalrep_worker_stop(Oid subid, Oid relid); extern void logicalrep_worker_wakeup(Oid subid, Oid relid); extern void logicalrep_worker_wakeup_ptr(LogicalRepWorker *worker); diff --git a/src/include/utils/wait_event.h b/src/include/utils/wait_event.h index b578e2e..d5081d9 100644 --- a/src/include/utils/wait_event.h +++ b/src/include/utils/wait_event.h @@ -105,6 +105,7 @@ typedef enum WAIT_EVENT_HASH_GROW_BUCKETS_ALLOCATE, WAIT_EVENT_HASH_GROW_BUCKETS_ELECT, WAIT_EVENT_HASH_GROW_BUCKETS_REINSERT, + WAIT_EVENT_LOGICAL_APPLY_WORKER_READY, WAIT_EVENT_LOGICAL_SYNC_DATA, WAIT_EVENT_LOGICAL_SYNC_STATE_CHANGE, WAIT_EVENT_MQ_INTERNAL, diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out index 7fcfad1..149fce1 100644 --- a/src/test/regress/expected/subscription.out +++ b/src/test/regress/expected/subscription.out @@ -197,7 +197,7 @@ ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); DROP SUBSCRIPTION regress_testsub; -- fail - streaming must be boolean CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = foo); -ERROR: streaming requires a Boolean value +ERROR: streaming requires a Boolean value or "spool" or "apply" -- now it works CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = true); WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables @@ -205,7 +205,7 @@ WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION .. List of subscriptions Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable on error | Synchronous commit | Conninfo | Skip LSN -----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------+---------- - regress_testsub | regress_subscription_user | f | {testpub} | f | t | d | f | off | dbname=regress_doesnotexist | 0/0 + regress_testsub | regress_subscription_user | f | {testpub} | f | a | d | f | off | dbname=regress_doesnotexist | 0/0 (1 row) ALTER SUBSCRIPTION regress_testsub SET (streaming = false); @@ -299,7 +299,7 @@ ALTER SUBSCRIPTION regress_testsub SET (streaming = true); List of subscriptions Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable on error | Synchronous commit | Conninfo | Skip LSN -----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------+---------- - regress_testsub | regress_subscription_user | f | {testpub} | f | t | p | f | off | dbname=regress_doesnotexist | 0/0 + regress_testsub | regress_subscription_user | f | {testpub} | f | a | p | f | off | dbname=regress_doesnotexist | 0/0 (1 row) ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); @@ -311,7 +311,7 @@ WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION .. List of subscriptions Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable on error | Synchronous commit | Conninfo | Skip LSN -----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------+---------- - regress_testsub | regress_subscription_user | f | {testpub} | f | t | p | f | off | dbname=regress_doesnotexist | 0/0 + regress_testsub | regress_subscription_user | f | {testpub} | f | a | p | f | off | dbname=regress_doesnotexist | 0/0 (1 row) ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); diff --git a/src/test/subscription/t/029_on_error.pl b/src/test/subscription/t/029_on_error.pl index e8b904b..9659fb9 100644 --- a/src/test/subscription/t/029_on_error.pl +++ b/src/test/subscription/t/029_on_error.pl @@ -108,7 +108,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres'; $node_publisher->safe_psql('postgres', "CREATE PUBLICATION pub FOR TABLE tbl"); $node_subscriber->safe_psql('postgres', - "CREATE SUBSCRIPTION sub CONNECTION '$publisher_connstr' PUBLICATION pub WITH (disable_on_error = true, streaming = on, two_phase = on)" + "CREATE SUBSCRIPTION sub CONNECTION '$publisher_connstr' PUBLICATION pub WITH (disable_on_error = true, streaming = spool, two_phase = on)" ); # Initial synchronization failure causes the subscription to be disabled. -- 2.7.2.windows.1 ^ permalink raw reply [nested|flat] 66+ messages in thread
* Re: Perform streaming logical transactions by background workers and parallel apply @ 2022-04-22 04:12 Peter Smith <[email protected]> parent: [email protected] <[email protected]> 0 siblings, 1 reply; 66+ messages in thread From: Peter Smith @ 2022-04-22 04:12 UTC (permalink / raw) To: [email protected] <[email protected]>; +Cc: Amit Kapila <[email protected]>; [email protected] <[email protected]>; PostgreSQL Hackers <[email protected]> Hello Hou-san. Here are my review comments for v4-0001. Sorry, there are so many of them (it is a big patch); some are trivial, and others you might easily dismiss due to my misunderstanding of the code. But hopefully, there are at least some comments that can be helpful in improving the patch quality. ====== 1. General comment - terms Needs to be more consistent about what exactly you will call this new worker. Sometimes called "locally apply worker"; sometimes "bgworker"; sometimes "subworker", sometimes "BGW", sometimes other variations etc … Need to pick ONE good name then update all the references/comments in the patch to use that name consistently throughout. ~~~ 2. General comment - option values I felt the "streaming" option values ought to be different from what this patch proposes so it affected some of my following review comments. (Later I give example what I thought the values should be). ~~~ 3. General comment - bool option change to enum This option change for "streaming" is similar to the options change for "copy_data=force" that Vignesh is doing for his "infinite recursion" patch v9-0002 [1]. Yet they seem implemented differently (i.e. char versus enum). I think you should discuss the 2 approaches with Vignesh and then code these option changes in a consistent way. ~~~ 4. General comment - worker.c globals There seems a growing number of global variables in the worker.c code. I was wondering is it really necessary? because the logic becomes more intricate now if you have to know that some global was set up as a side-effect of some other function call. E.g maybe if you could do a few more HTAB lookups to identify the bgworker then might not need to rely on the globals so much? ====== 5. Commit message - typo and then on the subscriber-side, the apply worker writes the changes into temporary files and once it receives the commit, it read from the file and apply the entire transaction. To improve the performance of such transactions, typo: "read" -> "reads" typo: "apply" -> "applies" ~~~ 6. Commit message - wording In this approach, we assign a new bgworker (if available) as soon as the xact's first stream came and the main apply worker will send changes to this new worker via shared memory. The bgworker will directly apply the change instead of writing it to temporary files. We keep this worker assigned till the transaction commit came and also wait for the worker to finish at commit. This wording: "came" -> "is received" (2x) ~~~ 7. Commit message - terms (this is the same point as comment #1) I think there is too much changing of terminology. IMO it will be easier if you always just call the current main apply workers the "apply worker" and always call this new worker the "bgworker" (or some better name). But never just call it the "worker". ~~~ 8. Commit message - typo transaction commit came and also wait for the worker to finish at commit. This preserves commit ordering and avoid writing to and reading from file in most cases. We still need to spill if there is no worker available. We also need to typo: "avoid" -> "avoids" ~~~ 9. Commit message - wording/typo Also extend the subscription streaming option so that user can control whether apply the streaming transaction in a bgworker or spill the change to disk. User wording: "Also extend" -> "This patch also extends" typo: "whether apply" -> "whether to apply" ~~~ 10. Commit message - option values apply the streaming transaction in a bgworker or spill the change to disk. User can set the streaming option to 'on/off', 'apply', 'spool'. For now, 'on' and Those values do not really seem intuitive to me. E.g. if you set "apply" then you already said above that sometimes it might have to spool anyway if there were no bgworkers available. Why not just name them like "on/off/parallel"? (I have written more about this in a later comment #14) ====== 11. doc/src/sgml/catalogs.sgml - wording + Controls in which modes we handle the streaming of in-progress transactions. + <literal>f</literal> = disallow streaming of in-progress transactions wording: "Controls in which modes we handle..." -> "Controls how to handle..." ~~~ 12. doc/src/sgml/catalogs.sgml - wording + <literal>a</literal> = apply changes directly in background worker wording: "in background worker" -> "using a background worker" ~~~ 13. doc/src/sgml/catalogs.sgml - option values Anyway, all this page will be different if I can persuade you to change the option values (see comment #14) ====== 14. doc/src/sgml/ref/create_subscription.sgml - option values Since the default value is "off" I felt these options would be better/simpler if they are just like "off/on/parallel". E.g. Specifically, I think the "on" should behave the same as the current code does, so the user should deliberately choose to use this new bgworker approach. e.g. - "off" = off, same as current PG15 - "on" = on, same as current PG15 - "parallel" = try to use the new bgworker to apply stream ====== 15. src/backend/commands/subscriptioncmds.c - SubOpts Vignesh uses similar code for his "infinite recursion" patch being developed [1] but he used an enum but here you use a char. I think you should discuss together both decide to use either enum or char for the member so there is a consistency. ~~~ 16. src/backend/commands/subscriptioncmds.c - combine conditions + /* + * The set of strings accepted here should match up with the + * grammar's opt_boolean_or_string production. + */ + if (pg_strcasecmp(sval, "true") == 0) + return SUBSTREAM_APPLY; + if (pg_strcasecmp(sval, "false") == 0) + return SUBSTREAM_OFF; + if (pg_strcasecmp(sval, "on") == 0) + return SUBSTREAM_APPLY; + if (pg_strcasecmp(sval, "off") == 0) + return SUBSTREAM_OFF; + if (pg_strcasecmp(sval, "spool") == 0) + return SUBSTREAM_SPOOL; + if (pg_strcasecmp(sval, "apply") == 0) + return SUBSTREAM_APPLY; Because I think the possible option values should be different to these I can’t comment much on this code, except to suggest IMO the if conditions should be combined where the options are considered to be equivalent. ====== 17. src/backend/replication/logical/launcher.c - stop_worker @@ -72,6 +72,7 @@ static void logicalrep_launcher_onexit(int code, Datum arg); static void logicalrep_worker_onexit(int code, Datum arg); static void logicalrep_worker_detach(void); static void logicalrep_worker_cleanup(LogicalRepWorker *worker); +static void stop_worker(LogicalRepWorker *worker); The function name does not seem consistent with the other similar static funcs. ~~~ 18. src/backend/replication/logical/launcher.c - change if @@ -225,7 +226,7 @@ logicalrep_worker_find(Oid subid, Oid relid, bool only_running) LogicalRepWorker *w = &LogicalRepCtx->workers[i]; if (w->in_use && w->subid == subid && w->relid == relid && - (!only_running || w->proc)) + (!only_running || w->proc) && !w->subworker) { Maybe code would be easier (and then you can comment it) if you do like: /* TODO: comment here */ if (w->subworker) continue; ~~~ 19. src/backend/replication/logical/launcher.c - logicalrep_worker_launch comment @@ -262,9 +263,9 @@ logicalrep_workers_find(Oid subid, bool only_running) /* * Start new apply background worker, if possible. */ -void +bool logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, Oid userid, - Oid relid) + Oid relid, dsm_handle subworker_dsm) Saying "start new apply..." comment feels a bit misleading. E.g. this is also called to start the sync worker. And also for the main apply worker (which we are not really calling a "background worker" in other places). So this is the same kind of terminology problem as my review comment #1. ~~~ 20. src/backend/replication/logical/launcher.c - asserts? I thought maybe there should be some assertions in this code upfront. E.g. cannot have OidIsValid(relid) and subworker_dsm valid at the same time. ~~~ 21. src/backend/replication/logical/launcher.c - terms + else + snprintf(bgw.bgw_name, BGW_MAXLEN, + "logical replication apply worker for subscription %u", subid); I think the names of all these workers is a bit vague still in the messages – e.g. "logical replication worker" versus "logical replication apply worker" sounds too similar to me. So this is kind of same as my review comment #1. ~~~ 22. src/backend/replication/logical/launcher.c - logicalrep_worker_stop double unlock? @@ -450,6 +465,18 @@ logicalrep_worker_stop(Oid subid, Oid relid) return; } + stop_worker(worker); + + LWLockRelease(LogicalRepWorkerLock); +} IIUC, sometimes it seems that stop_worker() function might already release the lock before it returns. In that case won’t this other explicit lock release be a problem? ~~~ 23. src/backend/replication/logical/launcher.c - logicalrep_worker_detach @@ -600,6 +625,28 @@ logicalrep_worker_attach(int slot) static void logicalrep_worker_detach(void) { + /* + * If we are the main apply worker, stop all the sub apply workers we + * started before. + */ + if (!MyLogicalRepWorker->subworker) + { + List *workers; + ListCell *lc; + + LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); + + workers = logicalrep_workers_find(MyLogicalRepWorker->subid, true); + foreach(lc, workers) + { + LogicalRepWorker *w = (LogicalRepWorker *) lfirst(lc); + if (w->subworker) + stop_worker(w); + } + + LWLockRelease(LogicalRepWorkerLock); Can this have the same double-unlock problem as I described in the previous review comment #22? ~~~ 24. src/backend/replication/logical/launcher.c - ApplyLauncherMain @@ -869,7 +917,7 @@ ApplyLauncherMain(Datum main_arg) wait_time = wal_retrieve_retry_interval; logicalrep_worker_launch(sub->dbid, sub->oid, sub->name, - sub->owner, InvalidOid); + sub->owner, InvalidOid, DSM_HANDLE_INVALID); } Now that the logicalrep_worker_launch is retuning a bool, should this call be checking the return value and taking appropriate action if it failed? ====== 25. src/backend/replication/logical/origin.c - acquire comment + /* + * We allow the apply worker to get the slot which is acquired by its + * leader process. + */ + else if (curstate->acquired_by != 0 && acquire) The comment was not very clear to me. Does the term "apply worker" in the comment make sense, or should that say "bgworker"? This might be another example of my review comment #1. ~~~ 26. src/backend/replication/logical/origin.c - acquire code + /* + * We allow the apply worker to get the slot which is acquired by its + * leader process. + */ + else if (curstate->acquired_by != 0 && acquire) { ereport(ERROR, I somehow felt that this param would be better called 'skip_acquire', so all the callers would have to use the opposite boolean and then this code would say like below (which seemed easier to me). YMMV. else if (curstate->acquired_by != 0 && !skip_acquire) { ereport(ERROR, ===== 27. src/backend/replication/logical/tablesync.c @@ -568,7 +568,8 @@ process_syncing_tables_for_apply(XLogRecPtr current_lsn) MySubscription->oid, MySubscription->name, MyLogicalRepWorker->userid, - rstate->relid); + rstate->relid, + DSM_HANDLE_INVALID); hentry->last_start_time = now; Now that the logicalrep_worker_launch is returning a bool, should this call be checking that the launch was successful before it changes the last_start_time? ====== 28. src/backend/replication/logical/worker.c - file comment + * 1) Separate background workers + * + * Assign a new bgworker (if available) as soon as the xact's first stream came + * and the main apply worker will send changes to this new worker via shared + * memory. We keep this worker assigned till the transaction commit came and + * also wait for the worker to finish at commit. This preserves commit ordering + * and avoid writing to and reading from file in most cases. We still need to + * spill if there is no worker available. We also need to allow stream_stop to + * complete by the background worker to finish it to avoid deadlocks because + * T-1's current stream of changes can update rows in conflicting order with + * T-2's next stream of changes. This comment fragment looks the same as the commit message so the typos/wording reported already for the commit message are applicable here too. ~~~ 29. src/backend/replication/logical/worker.c - file comment + * If no worker is available to handle streamed transaction, we write the data * to temporary files and then applied at once when the final commit arrives. wording: "we write the data" -> "the data is written" ~~~ 30. src/backend/replication/logical/worker.c - ParallelState +typedef struct ParallelState Add to typedefs.list ~~~ 31. src/backend/replication/logical/worker.c - ParallelState flags +typedef struct ParallelState +{ + slock_t mutex; + bool attached; + bool ready; + bool finished; + bool failed; + Oid subid; + TransactionId stream_xid; + uint32 n; +} ParallelState; Those bool states look independent to me. Should they be one enum member instead of lots of bool members? ~~~ 32. src/backend/replication/logical/worker.c - ParallelState comments +typedef struct ParallelState +{ + slock_t mutex; + bool attached; + bool ready; + bool finished; + bool failed; + Oid subid; + TransactionId stream_xid; + uint32 n; +} ParallelState; Needs some comments. Some might be self-evident but some are not - e.g. what is 'n'? ~~~ 33. src/backend/replication/logical/worker.c - WorkerState +typedef struct WorkerState Add to typedefs.list ~~~ 34. src/backend/replication/logical/worker.c - WorkerEntry +typedef struct WorkerEntry Add to typedefs.list ~~~ 35. src/backend/replication/logical/worker.c - static function names +/* Worker setup and interactions */ +static void setup_dsm(WorkerState *wstate); +static WorkerState *setup_background_worker(void); +static void wait_for_worker_ready(WorkerState *wstate, bool notify); +static void wait_for_transaction_finish(WorkerState *wstate); +static void send_data_to_worker(WorkerState *wstate, Size nbytes, + const void *data); +static WorkerState *find_or_start_worker(TransactionId xid, bool start); +static void free_stream_apply_worker(void); +static bool transaction_applied_in_bgworker(TransactionId xid); +static void check_workers_status(void); All these new functions have random-looking names. Since they all are new to this feature I thought they should all be named similarly... e.g. something like bgworker_setup bgworker_check_status bgworker_wait_for_ready etc. ~~~ 36. src/backend/replication/logical/worker.c - nchanges + +static uint32 nchanges = 0; + What is this? Needs a comment. ~~~ 37. src/backend/replication/logical/worker.c - handle_streamed_transaction static bool handle_streamed_transaction(LogicalRepMsgType action, StringInfo s) { - TransactionId xid; + TransactionId current_xid = InvalidTransactionId; /* not in streaming mode */ - if (!in_streamed_transaction) + if (!in_streamed_transaction && !isLogicalApplyWorker) return false; Is it correct to be testing the isLogicalApplyWorker here? e.g. What if the streaming code is not using bgworkers at all? At least maybe that comment (/* not in streaming mode */) should be updated? ~~~ 38. src/backend/replication/logical/worker.c - handle_streamed_transaction + if (current_xid != stream_xid && + !list_member_int(subxactlist, (int) current_xid)) + { + MemoryContext oldctx; + char *spname = (char *) palloc(64 * sizeof(char)); + sprintf(spname, "savepoint_for_xid_%u", current_xid); Can't the name just be a char[64] on the stack? ~~~ 39. src/backend/replication/logical/worker.c - handle_streamed_transaction + /* + * XXX The publisher side don't always send relation update message + * after the streaming transaction, so update the relation in main + * worker here. + */ typo: "don't" -> "doesn't" ? ~~~ 40. src/backend/replication/logical/worker.c - apply_handle_commit_prepared @@ -976,30 +1116,51 @@ apply_handle_commit_prepared(StringInfo s) char gid[GIDSIZE]; logicalrep_read_commit_prepared(s, &prepare_data); + set_apply_error_context_xact(prepare_data.xid, prepare_data.commit_lsn); Spurious whitespace? ~~~ 41. src/backend/replication/logical/worker.c - apply_handle_commit_prepared + /* Check if we have prepared transaction in another bgworker */ + if (transaction_applied_in_bgworker(prepare_data.xid)) + { + elog(DEBUG1, "received commit for streamed transaction %u", prepare_data.xid); - /* There is no transaction when COMMIT PREPARED is called */ - begin_replication_step(); + /* Send commit message */ + send_data_to_worker(stream_apply_worker, s->len, s->data); It seems a bit complex/tricky that the code is always relying on all the side-effects that the global stream_apply_worker will be set. I am not sure if it is possible to remove the global and untangle everything. E.g. Why not change the transaction_applied_in_bgworker to return the bgworker (instead of return bool) and then can assign it to a local var in this function. Or can’t you do HTAB lookup in a few more places instead of carrying around the knowledge of some global var that was initialized in some other place? It would be easier if you can eliminate having to be aware of side-effects happening behind the scenes. ~~~ 42. src/backend/replication/logical/worker.c - apply_handle_rollback_prepared @@ -1019,35 +1180,51 @@ apply_handle_rollback_prepared(StringInfo s) char gid[GIDSIZE]; logicalrep_read_rollback_prepared(s, &rollback_data); + set_apply_error_context_xact(rollback_data.xid, rollback_data.rollback_end_lsn); Spurious whitespace? ~~~ 43. src/backend/replication/logical/worker.c - apply_handle_rollback_prepared + /* Check if we are processing the prepared transaction in a bgworker */ + if (transaction_applied_in_bgworker(rollback_data.xid)) + { + send_data_to_worker(stream_apply_worker, s->len, s->data); Same as previous comment #41. Relies on the side effect of something setting the global stream_apply_worker. ~~~ 44. src/backend/replication/logical/worker.c - find_or_start_worker + /* + * For streaming transactions that is being applied in bgworker, we cannot + * decide whether to apply the change for a relation that is not in the + * READY state (see should_apply_changes_for_rel) as we won't know + * remote_final_lsn by that time. So, we don't start new bgworker in this + * case. + */ typo: "that is" -> "that are" ~~~ 45. src/backend/replication/logical/worker.c - find_or_start_worker + if (MySubscription->stream != SUBSTREAM_APPLY) + return NULL; ... + else if (start && !XLogRecPtrIsInvalid(MySubscription->skiplsn)) + return NULL; ... + else if (start && !AllTablesyncsReady()) + return NULL; + else if (!start && ApplyWorkersHash == NULL) + return NULL; I am not sure but I think most of that rejection if/else can probably just be "if" (not "else if") because otherwise, the code would have returned anyhow, right? Removing all the "else" might make the code more readable. ~~~ 46. src/backend/replication/logical/worker.c - find_or_start_worker + if (wstate == NULL) + { + /* + * If there is no more worker can be launched here, remove the + * entry in hash table. + */ + hash_search(ApplyWorkersHash, &xid, HASH_REMOVE, &found); + return NULL; + } wording: "If there is no more worker can be launched here, remove" -> "If the bgworker cannot be launched, remove..." ~~~ 47. src/backend/replication/logical/worker.c - free_stream_apply_worker +/* + * Add the worker to the freelist and remove the entry from hash table. + */ +static void +free_stream_apply_worker(void) IMO it might be better to pass the bgworker here instead of silently working with the global stream_apply_worker. ~~~ 48. src/backend/replication/logical/worker.c - free_stream_apply_worker + elog(LOG, "adding finished apply worker #%u for xid %u to the idle list", + stream_apply_worker->pstate->n, stream_apply_worker->pstate->stream_xid); Should the be an Assert here to check the bgworker state really was FINISHED? ~~~ 49. src/backend/replication/logical/worker.c - serialize_stream_prepare +static void +serialize_stream_prepare(LogicalRepPreparedTxnData *prepare_data) Missing function comment. ~~~ 50. src/backend/replication/logical/worker.c - serialize_stream_start -/* - * Handle STREAM START message. - */ static void -apply_handle_stream_start(StringInfo s) +serialize_stream_start(bool first_segment) Missing function comment. ~~~ 51. src/backend/replication/logical/worker.c - serialize_stream_stop +static void +serialize_stream_stop() +{ Missing function comment. ~~~ 52. src/backend/replication/logical/worker.c - general serialize_XXXX I can see now that you have created many serialize_XXX functions which seem to only be called one time. It looks like the only purpose is to encapsulate the code to make the handler function shorter? But it seems a bit uneven that you did this only for the serialize cases. If you really want these separate functions then perhaps there ought to also be the equivalent bgworker functions too. There seem to be always 3 scenarios: i.e 1. Worker is the bgworker 2. Worker is Main Apply but a bgworker exists 3. Worker is Main apply and bgworker does not exist. Perhaps every handler function should have THREE other little functions that it calls appropriately? ~~~ 53. src/backend/replication/logical/worker.c - serialize_stream_abort + +static void +serialize_stream_abort(TransactionId xid, TransactionId subxid) +{ Missing function comment. ~~~ 54. src/backend/replication/logical/worker.c - apply_handle_stream_abort + if (isLogicalApplyWorker) + { + ereport(LOG, + (errcode_for_file_access(), + errmsg("[Apply BGW #%u] aborting current transaction xid=%u, subxid=%u", + MyParallelState->n, GetCurrentTransactionIdIfAny(), GetCurrentSubTransactionId()))); Why is the errcode using errcode_for_file_access? (2x) ~~~ 55. src/backend/replication/logical/worker.c - apply_handle_stream_abort + /* + * OK, so it's a subxact. Rollback to the savepoint. + * + * We also need to read the subxactlist, determine the offset + * tracked for the subxact, and truncate the list. + */ + int i; + bool found = false; + char *spname = (char *) palloc(64 * sizeof(char)); Can that just be char[64] on the stack? ~~~ 56. src/backend/replication/logical/worker.c - apply_dispatch @@ -2511,6 +3061,7 @@ apply_dispatch(StringInfo s) break; case LOGICAL_REP_MSG_STREAM_START: + elog(LOG, "LOGICAL_REP_MSG_STREAM_START"); apply_handle_stream_start(s); break; I guess this is just for debugging purposes so you should put some FIXME comment here as a reminder to get rid of it later? ~~~ 57. src/backend/replication/logical/worker.c - store_flush_position, isLogicalApplyWorker @@ -2618,6 +3169,10 @@ store_flush_position(XLogRecPtr remote_lsn) { FlushPosition *flushpos; + /* We only need to collect the LSN in main apply worker */ + if (isLogicalApplyWorker) + return; + This comment is not specific to this function, but for global isLogicalApplyWorker IMO this should be implemented to look more like the inline function am_tablesync_worker(). e.g. I think you should replace this global with something like am_apply_bgworker() Maybe it should do something like check the value of MyLogicalRepWorker->subworker? ~~~ 58. src/backend/replication/logical/worker.c - LogicalRepApplyLoop @@ -3467,6 +4025,7 @@ TwoPhaseTransactionGid(Oid subid, TransactionId xid, char *gid, int szgid) snprintf(gid, szgid, "pg_gid_%u_%u", subid, xid); } + /* * Execute the initial sync with error handling. Disable the subscription, * if it's required. Spurious whitespace ~~~ 59. src/backend/replication/logical/worker.c - ApplyWorkerMain @@ -3733,7 +4292,7 @@ ApplyWorkerMain(Datum main_arg) options.proto.logical.publication_names = MySubscription->publications; options.proto.logical.binary = MySubscription->binary; - options.proto.logical.streaming = MySubscription->stream; + options.proto.logical.streaming = (MySubscription->stream != SUBSTREAM_OFF); options.proto.logical.twophase = false; I was not sure why this is converting from an enum to a boolean? Is it right? ~~~ 60. src/backend/replication/logical/worker.c - LogicalApplyBgwLoop + shmq_res = shm_mq_receive(mqh, &len, &data, false); + + if (shmq_res != SHM_MQ_SUCCESS) + break; Should this log some more error information here? ~~~ 61. src/backend/replication/logical/worker.c - LogicalApplyBgwLoop + if (len == 0) + { + elog(LOG, "[Apply BGW #%u] got zero-length message, stopping", pst->n); + break; + } + else + { + XLogRecPtr start_lsn; + XLogRecPtr end_lsn; + TimestampTz send_time; Maybe the "else" is not needed here, and if you remove it then it will get rid of all the unnecessary indentation. ~~~ 62. src/backend/replication/logical/worker.c - LogicalApplyBgwLoop + /* + * We use first byte of message for additional communication between + * main Logical replication worker and Apply BGWorkers, so if it + * differs from 'w', then process it first. + */ I was thinking maybe this switch should include case 'w': break; because then for the "default" case you should give ERROR because something unexpected arrived. ~~~ 63. src/backend/replication/logical/worker.c - ApplyBgwShutdown +static void +ApplyBgwShutdown(int code, Datum arg) +{ + SpinLockAcquire(&MyParallelState->mutex); + MyParallelState->failed = true; + SpinLockRelease(&MyParallelState->mutex); + + dsm_detach((dsm_segment *) DatumGetPointer(arg)); +} Should this do detach first and set the flag last? ~~~ 64. src/backend/replication/logical/worker.c - LogicalApplyBgwMain + /* + * Acquire a worker number. + * + * By convention, the process registering this background worker should + * have stored the control structure at key 0. We look up that key to + * find it. Our worker number gives our identity: there may be just one + * worker involved in this parallel operation, or there may be many. + */ Maybe there should be another elog closer to this comment? So as soon as you know the BGW number log something? e.g. elog(LOG, "[Apply BGW #%u] starting", pst->n); ~~~ 65. src/backend/replication/logical/worker.c - setup_background_worker +/* + * Register background workers. + */ +static WorkerState * +setup_background_worker(void) I think that comment needs some more info because it is doing more than just registering... it is successfully launching the worker first. ~~~ 66. src/backend/replication/logical/worker.c - setup_background_worker + if (launched) + { + /* Wait for worker to become ready. */ + wait_for_worker_ready(wstate, false); + + ApplyWorkersList = lappend(ApplyWorkersList, wstate); + nworkers += 1; + } Do you really need to carry around this global 'nworkers' variable? Can’t you just check the length of the ApplyWorkerList to get this number? ~~~ 67. src/backend/replication/logical/worker.c - send_data_to_worker +/* + * Send the data to worker via shared-memory queue. + */ +static void +send_data_to_worker(WorkerState *wstate, Size nbytes, const void *data) wording: "to worker" -> "to the specified apply bgworker" This is just another example of my comment #1. ~~~ 68. src/backend/replication/logical/worker.c - send_data_to_worker + if (result != SHM_MQ_SUCCESS) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("could not send tuple to shared-memory queue"))); +} typo: is "tuples" the right word here? ~~~ 69. src/backend/replication/logical/worker.c - wait_for_worker_ready + +static void +wait_for_worker_ready(WorkerState *wstate, bool notify) +{ Missing function comment. ~~~ 70. src/backend/replication/logical/worker.c - wait_for_worker_ready + +static void +wait_for_worker_ready(WorkerState *wstate, bool notify) +{ 'notify' seems a bit of a poor name here. And this param seems a bit of a strange side-effect for something called wait_for_worker_ready. If really need to do this way maybe name it something more verbose like 'notify_received_stream_stop'? ~~~ 71. src/backend/replication/logical/worker.c - wait_for_worker_ready + if (!result) + ereport(ERROR, + (errcode(ERRCODE_INSUFFICIENT_RESOURCES), + errmsg("one or more background workers failed to start"))); Is the ERROR code reachable? IIUC there is no escape from the previous for (;;) loop except when the result is set to true. ~~~ 72. src/backend/replication/logical/worker.c - wait_for_transaction_finish + +static void +wait_for_transaction_finish(WorkerState *wstate) +{ Missing function comment. ~~~ 73. src/backend/replication/logical/worker.c - wait_for_transaction_finish + if (finished) + { + break; + } The brackets are not needed for 1 statement. ~~~ 74. src/backend/replication/logical/worker.c - transaction_applied_in_bgworker +static bool +transaction_applied_in_bgworker(TransactionId xid) Instead of side-effect assigning the global variable, why not return the bgworker (or NULL) and let the caller work with the result? ~~~ 75. src/backend/replication/logical/worker.c - check_workers_status +/* + * Check the status of workers and report an error if any bgworker exit + * unexpectedly. wording: -> "... if any bgworker has exited unexpectedly ..." ~~~ 76. src/backend/replication/logical/worker.c - check_workers_status + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("Background worker %u exited unexpectedly", + wstate->pstate->n))); Should that message also give more identifying info about the *current* worker doing the ERROR - e.g.the one which found this the other bgworker was failed? Or is that just the PIC in the log message good enough? ~~~ 77. src/backend/replication/logical/worker.c - check_workers_status + if (!AllTablesyncsReady() && nfreeworkers != list_length(ApplyWorkersList)) + { I did not really understand this code, but isn't there a possibility that it will cause many restarts if the tablesyncs are taking a long time to complete? ====== 78. src/include/catalog/pg_subscription. @@ -122,6 +122,18 @@ typedef struct Subscription List *publications; /* List of publication names to subscribe to */ } Subscription; +/* Disallow streaming in-progress transactions */ +#define SUBSTREAM_OFF 'f' + +/* + * Streaming transactions are written to a temporary file and applied only + * after the transaction is committed on upstream. + */ +#define SUBSTREAM_SPOOL 's' + +/* Streaming transactions are appied immediately via a background worker */ +#define SUBSTREAM_APPLY 'a' IIRC Vignesh had a similar options requirement for his "infinite recursion" patch [1], except he was using enums instead of #define for char. Maybe discuss with Vignesh (and either he should change or you should change) so there is a consistent code style for the options. ====== 79. src/include/replication/logicalproto.h - old extern @@ -243,8 +243,10 @@ extern TransactionId logicalrep_read_stream_start(StringInfo in, extern void logicalrep_write_stream_stop(StringInfo out); extern void logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn, XLogRecPtr commit_lsn); -extern TransactionId logicalrep_read_stream_commit(StringInfo out, +extern TransactionId logicalrep_read_stream_commit_old(StringInfo out, LogicalRepCommitData *commit_data); Is anybody still using this "old" function? Maybe I missed it. ====== 80. src/include/replication/logicalworker.h @@ -13,6 +13,7 @@ #define LOGICALWORKER_H extern void ApplyWorkerMain(Datum main_arg); +extern void LogicalApplyBgwMain(Datum main_arg); The new name seems inconsistent with the old one. What about calling it ApplyBgworkerMain? ====== 81. src/test/regress/expected/subscription.out Isn't this missing some test cases for the new options added? E.g. I never see streaming value is set to 's'. ====== 82. src/test/subscription/t/029_on_error.pl If options values were changed how I suggested (review comment #14) then I think a change such as this would not be necessary because everything would be backward compatible. ------ [1] https://www.postgresql.org/message-id/CALDaNm2Fe%3Dg4Tx-DhzwD6NU0VRAfaPedXwWO01maNU7_OfS8fw%40mail.g... Kind Regards, Peter Smith. Fujitsu Australia ^ permalink raw reply [nested|flat] 66+ messages in thread
* RE: Perform streaming logical transactions by background workers and parallel apply @ 2022-04-25 08:35 [email protected] <[email protected]> parent: Peter Smith <[email protected]> 0 siblings, 1 reply; 66+ messages in thread From: [email protected] @ 2022-04-25 08:35 UTC (permalink / raw) To: Peter Smith <[email protected]>; +Cc: Amit Kapila <[email protected]>; [email protected] <[email protected]>; PostgreSQL Hackers <[email protected]> On Friday, April 22, 2022 12:12 PM Peter Smith <[email protected]> wrote: > > Hello Hou-san. Here are my review comments for v4-0001. Sorry, there > are so many of them (it is a big patch); some are trivial, and others > you might easily dismiss due to my misunderstanding of the code. But > hopefully, there are at least some comments that can be helpful in > improving the patch quality. Thanks for the comments ! I think most of the comments make sense and here are explanations for some of them. > 24. src/backend/replication/logical/launcher.c - ApplyLauncherMain > > @@ -869,7 +917,7 @@ ApplyLauncherMain(Datum main_arg) > wait_time = wal_retrieve_retry_interval; > > logicalrep_worker_launch(sub->dbid, sub->oid, sub->name, > - sub->owner, InvalidOid); > + sub->owner, InvalidOid, DSM_HANDLE_INVALID); > } > Now that the logicalrep_worker_launch is retuning a bool, should this > call be checking the return value and taking appropriate action if it > failed? Not sure we can change the logic of existing caller. I think only the new caller in the patch is necessary to check this. > 26. src/backend/replication/logical/origin.c - acquire code > > + /* > + * We allow the apply worker to get the slot which is acquired by its > + * leader process. > + */ > + else if (curstate->acquired_by != 0 && acquire) > { > ereport(ERROR, > > I somehow felt that this param would be better called 'skip_acquire', > so all the callers would have to use the opposite boolean and then > this code would say like below (which seemed easier to me). YMMV. > > else if (curstate->acquired_by != 0 && !skip_acquire) > { > ereport(ERROR, Not sure about this. > 59. src/backend/replication/logical/worker.c - ApplyWorkerMain > > @@ -3733,7 +4292,7 @@ ApplyWorkerMain(Datum main_arg) > > options.proto.logical.publication_names = MySubscription->publications; > options.proto.logical.binary = MySubscription->binary; > - options.proto.logical.streaming = MySubscription->stream; > + options.proto.logical.streaming = (MySubscription->stream != SUBSTREAM_OFF); > options.proto.logical.twophase = false; > > I was not sure why this is converting from an enum to a boolean? Is it right? I think it's ok, the "logical.streaming" is used in publisher which don't need to know the exact type of the streaming(it only need to know whether the streaming is enabled for now) > 63. src/backend/replication/logical/worker.c - ApplyBgwShutdown > > +static void > +ApplyBgwShutdown(int code, Datum arg) > +{ > + SpinLockAcquire(&MyParallelState->mutex); > + MyParallelState->failed = true; > + SpinLockRelease(&MyParallelState->mutex); > + > + dsm_detach((dsm_segment *) DatumGetPointer(arg)); > +} > > Should this do detach first and set the flag last? Not sure about this. I think it's fine to detach this at the end. > 76. src/backend/replication/logical/worker.c - check_workers_status > > + ereport(ERROR, > + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), > + errmsg("Background worker %u exited unexpectedly", > + wstate->pstate->n))); > > Should that message also give more identifying info about the > *current* worker doing the ERROR - e.g.the one which found this the > other bgworker was failed? Or is that just the PIC in the log message > good enough? Currently, only the main apply worker should report this error, so not sure do we need to report the current worker. > 77. src/backend/replication/logical/worker.c - check_workers_status > > + if (!AllTablesyncsReady() && nfreeworkers != list_length(ApplyWorkersList)) > + { > > I did not really understand this code, but isn't there a possibility > that it will cause many restarts if the tablesyncs are taking a long > time to complete? I think it's ok, after restarting, we won't start bgworker until all the table is READY. Best regards, Hou zj ^ permalink raw reply [nested|flat] 66+ messages in thread
* RE: Perform streaming logical transactions by background workers and parallel apply @ 2022-04-29 02:06 [email protected] <[email protected]> parent: [email protected] <[email protected]> 0 siblings, 2 replies; 66+ messages in thread From: [email protected] @ 2022-04-29 02:06 UTC (permalink / raw) To: Peter Smith <[email protected]>; +Cc: Amit Kapila <[email protected]>; [email protected] <[email protected]>; PostgreSQL Hackers <[email protected]> On Monday, April 25, 2022 4:35 PM [email protected] <[email protected]> wrote: > On Friday, April 22, 2022 12:12 PM Peter Smith <[email protected]> > wrote: > > > > Hello Hou-san. Here are my review comments for v4-0001. Sorry, there > > are so many of them (it is a big patch); some are trivial, and others > > you might easily dismiss due to my misunderstanding of the code. But > > hopefully, there are at least some comments that can be helpful in > > improving the patch quality. > > Thanks for the comments ! > I think most of the comments make sense and here are explanations for some > of them. Hi, I addressed the rest of Peter's comments and here is a new version patch. The naming of the newly introduced option and worker might need more thought, so I haven't change all of them. I will think over and change it later. One comment I didn't address: > 3. General comment - bool option change to enum > > This option change for "streaming" is similar to the options change > for "copy_data=force" that Vignesh is doing for his "infinite > recursion" patch v9-0002 [1]. Yet they seem implemented differently > (i.e. char versus enum). I think you should discuss the 2 approaches > with Vignesh and then code these option changes in a consistent way. > > [1] https://www.postgresql.org/message-id/CALDaNm2Fe%3Dg4Tx-DhzwD6NU0VRAfaPedXwWO01maNU7_OfS8fw%40mail.g...; com I think the "streaming" option is a bit different from the "copy_data" option. Because the "streaming" is a column of the system table (pg_subscription) which should use "char" type to represent different values in this case(For example: pg_class.relkind/pg_class.relpersistence/pg_class.relreplident ...). And the "copy_data" option is not a system table column and I think it's fine to use Enum for it. Best regards, Hou zj Attachments: [application/octet-stream] v5-0001-Perform-streaming-logical-transactions-by-background.patch (81.8K, ../../OS0PR01MB5716E8D536552467EFB512EF94FC9@OS0PR01MB5716.jpnprd01.prod.outlook.com/2-v5-0001-Perform-streaming-logical-transactions-by-background.patch) download | inline diff: From ec33a270eb80330b0103d6051ae9bea0b2d6f7fd Mon Sep 17 00:00:00 2001 From: "houzj.fnst" <[email protected]> Date: Wed, 20 Apr 2022 16:45:07 +0800 Subject: [PATCH] Perform streaming logical transactions by background workers Currently, for large transactions, the publisher sends the data in multiple streams (changes divided into chunks depending upon logical_decoding_work_mem), and then on the subscriber-side, the apply worker writes the changes into temporary files and once it receives the commit, it reads from the file and applies the entire transaction. To improve the performance of such transactions, we can instead allow them to be applied via background workers. In this approach, we assign a new apply background (if available) as soon as the xact's first stream is received and the main apply worker will send changes to this new worker via shared memory. The apply background will directly apply the change instead of writing it to temporary files. We keep this worker assigned till the transaction commit is received and also wait for the worker to finish at commit. This preserves commit ordering and avoids writing to and reading from file in most cases. We still need to spill if there is no worker available. We also need to allow stream_stop to complete by the apply background to finish it to avoid deadlocks because T-1's current stream of changes can update rows in conflicting order with T-2's next stream of changes. This patch also extends the subscription streaming option so that user can control whether apply the streaming transaction in a apply background or spill the change to disk. User can set the streaming option to 'on/off', 'apply'. For now, 'apply' means the streaming will be applied via a apply background if available. 'on' means the streaming transaction will be spilled to disk. --- doc/src/sgml/catalogs.sgml | 10 +- doc/src/sgml/ref/create_subscription.sgml | 24 +- src/backend/commands/subscriptioncmds.c | 66 +- src/backend/postmaster/bgworker.c | 3 + src/backend/replication/logical/launcher.c | 84 +- src/backend/replication/logical/origin.c | 10 +- src/backend/replication/logical/proto.c | 7 +- src/backend/replication/logical/tablesync.c | 10 +- src/backend/replication/logical/worker.c | 1454 ++++++++++++++++++++++++--- src/backend/utils/activity/wait_event.c | 3 + src/bin/pg_dump/pg_dump.c | 6 +- src/include/catalog/pg_subscription.h | 16 +- src/include/replication/logicalproto.h | 4 +- src/include/replication/logicalworker.h | 1 + src/include/replication/origin.h | 2 +- src/include/replication/worker_internal.h | 7 +- src/include/utils/wait_event.h | 1 + src/test/regress/expected/subscription.out | 8 +- src/tools/pgindent/typedefs.list | 2 + 19 files changed, 1506 insertions(+), 212 deletions(-) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index a533a21..ed61f99 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -7863,11 +7863,15 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l <row> <entry role="catalog_table_entry"><para role="column_definition"> - <structfield>substream</structfield> <type>bool</type> + <structfield>substream</structfield> <type>char</type> </para> <para> - If true, the subscription will allow streaming of in-progress - transactions + Controls how to handle the streaming of in-progress transactions. + <literal>f</literal> = disallow streaming of in-progress transactions + <literal>o</literal> = spill the changes of in-progress transactions to + disk and apply at once after the transaction is committed on the + publisher. + <literal>a</literal> = apply changes directly using a background worker </para></entry> </row> diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml index 203bb41..8441aa0 100644 --- a/doc/src/sgml/ref/create_subscription.sgml +++ b/doc/src/sgml/ref/create_subscription.sgml @@ -217,13 +217,29 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl </varlistentry> <varlistentry> - <term><literal>streaming</literal> (<type>boolean</type>)</term> + <term><literal>streaming</literal> (<type>enum</type>)</term> <listitem> <para> Specifies whether to enable streaming of in-progress transactions - for this subscription. By default, all transactions - are fully decoded on the publisher and only then sent to the - subscriber as a whole. + for this subscription. The default value is <literal>off</literal>, + all transactions are fully decoded on the publisher and only then + sent to the subscriber as a whole. + </para> + + <para> + If set to <literal>on</literal>, the changes of transaction are + written to temporary files and then applied at once after the + transaction is committed on the publisher. + </para> + + <para> + If set to <literal>apply</literal> incoming + changes are directly applied via one of the background worker, if + available. If no background worker is free to handle streaming + transaction then the changes are written to a file and applied after + the transaction is committed. Note that if error happen when applying + changes in background worker, it might not report the finish LSN of + the remote transaction in server log. </para> </listitem> </varlistentry> diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c index b94236f..fe1d373 100644 --- a/src/backend/commands/subscriptioncmds.c +++ b/src/backend/commands/subscriptioncmds.c @@ -83,7 +83,7 @@ typedef struct SubOpts bool copy_data; bool refresh; bool binary; - bool streaming; + char streaming; bool twophase; bool disableonerr; XLogRecPtr lsn; @@ -96,6 +96,62 @@ static void ReportSlotConnectionError(List *rstates, Oid subid, char *slotname, /* + * Extract the streaming mode value from a DefElem. This is like + * defGetBoolean() but also accepts the special value and "apply". + */ +static char +defGetStreamingMode(DefElem *def) +{ + /* + * If no parameter given, assume "true" is meant. + */ + if (def->arg == NULL) + return SUBSTREAM_ON; + + /* + * Allow 0, 1, "true", "false", "on", "off" or "apply". + */ + switch (nodeTag(def->arg)) + { + case T_Integer: + switch (intVal(def->arg)) + { + case 0: + return SUBSTREAM_OFF; + case 1: + return SUBSTREAM_ON; + default: + /* otherwise, error out below */ + break; + } + break; + default: + { + char *sval = defGetString(def); + + /* + * The set of strings accepted here should match up with the + * grammar's opt_boolean_or_string production. + */ + if (pg_strcasecmp(sval, "true") == 0 || + pg_strcasecmp(sval, "on") == 0) + return SUBSTREAM_ON; + if (pg_strcasecmp(sval, "apply") == 0) + return SUBSTREAM_APPLY; + if (pg_strcasecmp(sval, "false") == 0 || + pg_strcasecmp(sval, "off") == 0) + return SUBSTREAM_OFF; + } + break; + } + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s requires a Boolean value or \"apply\"", + def->defname))); + return SUBSTREAM_OFF; /* keep compiler quiet */ +} + +/* * Common option parsing function for CREATE and ALTER SUBSCRIPTION commands. * * Since not all options can be specified in both commands, this function @@ -132,7 +188,7 @@ parse_subscription_options(ParseState *pstate, List *stmt_options, if (IsSet(supported_opts, SUBOPT_BINARY)) opts->binary = false; if (IsSet(supported_opts, SUBOPT_STREAMING)) - opts->streaming = false; + opts->streaming = SUBSTREAM_OFF; if (IsSet(supported_opts, SUBOPT_TWOPHASE_COMMIT)) opts->twophase = false; if (IsSet(supported_opts, SUBOPT_DISABLE_ON_ERR)) @@ -233,7 +289,7 @@ parse_subscription_options(ParseState *pstate, List *stmt_options, errorConflictingDefElem(defel, pstate); opts->specified_opts |= SUBOPT_STREAMING; - opts->streaming = defGetBoolean(defel); + opts->streaming = defGetStreamingMode(defel); } else if (strcmp(defel->defname, "two_phase") == 0) { @@ -601,7 +657,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt, values[Anum_pg_subscription_subowner - 1] = ObjectIdGetDatum(owner); values[Anum_pg_subscription_subenabled - 1] = BoolGetDatum(opts.enabled); values[Anum_pg_subscription_subbinary - 1] = BoolGetDatum(opts.binary); - values[Anum_pg_subscription_substream - 1] = BoolGetDatum(opts.streaming); + values[Anum_pg_subscription_substream - 1] = CharGetDatum(opts.streaming); values[Anum_pg_subscription_subtwophasestate - 1] = CharGetDatum(opts.twophase ? LOGICALREP_TWOPHASE_STATE_PENDING : @@ -1060,7 +1116,7 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt, if (IsSet(opts.specified_opts, SUBOPT_STREAMING)) { values[Anum_pg_subscription_substream - 1] = - BoolGetDatum(opts.streaming); + CharGetDatum(opts.streaming); replaces[Anum_pg_subscription_substream - 1] = true; } diff --git a/src/backend/postmaster/bgworker.c b/src/backend/postmaster/bgworker.c index 30682b6..070d778 100644 --- a/src/backend/postmaster/bgworker.c +++ b/src/backend/postmaster/bgworker.c @@ -128,6 +128,9 @@ static const struct }, { "ApplyWorkerMain", ApplyWorkerMain + }, + { + "ApplyBgworkerMain", ApplyBgworkerMain } }; diff --git a/src/backend/replication/logical/launcher.c b/src/backend/replication/logical/launcher.c index 0adb2d1..299634c 100644 --- a/src/backend/replication/logical/launcher.c +++ b/src/backend/replication/logical/launcher.c @@ -72,6 +72,7 @@ static void logicalrep_launcher_onexit(int code, Datum arg); static void logicalrep_worker_onexit(int code, Datum arg); static void logicalrep_worker_detach(void); static void logicalrep_worker_cleanup(LogicalRepWorker *worker); +static void logicalrep_worker_stop_internal(LogicalRepWorker *worker); static bool on_commit_launcher_wakeup = false; @@ -224,6 +225,10 @@ logicalrep_worker_find(Oid subid, Oid relid, bool only_running) { LogicalRepWorker *w = &LogicalRepCtx->workers[i]; + /* We only need main apply worker or table sync worker here */ + if (w->subworker) + continue; + if (w->in_use && w->subid == subid && w->relid == relid && (!only_running || w->proc)) { @@ -262,9 +267,9 @@ logicalrep_workers_find(Oid subid, bool only_running) /* * Start new apply background worker, if possible. */ -void +bool logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, Oid userid, - Oid relid) + Oid relid, dsm_handle subworker_dsm) { BackgroundWorker bgw; BackgroundWorkerHandle *bgw_handle; @@ -275,6 +280,9 @@ logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, Oid userid, int nsyncworkers; TimestampTz now; + /* We don't support table sync in subworker */ + Assert(!((subworker_dsm != DSM_HANDLE_INVALID) && OidIsValid(relid))); + ereport(DEBUG1, (errmsg_internal("starting logical replication worker for subscription \"%s\"", subname))); @@ -351,7 +359,7 @@ retry: if (OidIsValid(relid) && nsyncworkers >= max_sync_workers_per_subscription) { LWLockRelease(LogicalRepWorkerLock); - return; + return false; } /* @@ -365,7 +373,7 @@ retry: (errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED), errmsg("out of logical replication worker slots"), errhint("You might need to increase max_logical_replication_workers."))); - return; + return false; } /* Prepare the worker slot. */ @@ -380,6 +388,7 @@ retry: worker->relstate = SUBREL_STATE_UNKNOWN; worker->relstate_lsn = InvalidXLogRecPtr; worker->stream_fileset = NULL; + worker->subworker = (subworker_dsm != DSM_HANDLE_INVALID); worker->last_lsn = InvalidXLogRecPtr; TIMESTAMP_NOBEGIN(worker->last_send_time); TIMESTAMP_NOBEGIN(worker->last_recv_time); @@ -397,19 +406,31 @@ retry: BGWORKER_BACKEND_DATABASE_CONNECTION; bgw.bgw_start_time = BgWorkerStart_RecoveryFinished; snprintf(bgw.bgw_library_name, BGW_MAXLEN, "postgres"); - snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyWorkerMain"); + + if (subworker_dsm == DSM_HANDLE_INVALID) + snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyWorkerMain"); + else + snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyBgworkerMain"); + + if (OidIsValid(relid)) snprintf(bgw.bgw_name, BGW_MAXLEN, "logical replication worker for subscription %u sync %u", subid, relid); - else + else if (subworker_dsm == DSM_HANDLE_INVALID) snprintf(bgw.bgw_name, BGW_MAXLEN, "logical replication worker for subscription %u", subid); + else + snprintf(bgw.bgw_name, BGW_MAXLEN, + "logical replication apply worker for subscription %u", subid); snprintf(bgw.bgw_type, BGW_MAXLEN, "logical replication worker"); bgw.bgw_restart_time = BGW_NEVER_RESTART; bgw.bgw_notify_pid = MyProcPid; bgw.bgw_main_arg = Int32GetDatum(slot); + if (subworker_dsm != DSM_HANDLE_INVALID) + memcpy(bgw.bgw_extra, &subworker_dsm, sizeof(dsm_handle)); + if (!RegisterDynamicBackgroundWorker(&bgw, &bgw_handle)) { /* Failed to start worker, so clean up the worker slot. */ @@ -422,11 +443,13 @@ retry: (errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED), errmsg("out of background worker slots"), errhint("You might need to increase max_worker_processes."))); - return; + return false; } /* Now wait until it attaches. */ WaitForReplicationWorkerAttach(worker, generation, bgw_handle); + + return true; } /* @@ -437,7 +460,6 @@ void logicalrep_worker_stop(Oid subid, Oid relid) { LogicalRepWorker *worker; - uint16 generation; LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); @@ -450,6 +472,22 @@ logicalrep_worker_stop(Oid subid, Oid relid) return; } + logicalrep_worker_stop_internal(worker); + + LWLockRelease(LogicalRepWorkerLock); +} + +/* + * Workhorse for logicalrep_worker_stop() and logicalrep_worker_detach(). Stop + * the worker and wait for wait for it to die. + */ +static void +logicalrep_worker_stop_internal(LogicalRepWorker *worker) +{ + uint16 generation; + + Assert(LWLockHeldByMe(LogicalRepWorkerLock)); + /* * Remember which generation was our worker so we can check if what we see * is still the same one. @@ -486,10 +524,7 @@ logicalrep_worker_stop(Oid subid, Oid relid) * different, meaning that a different worker has taken the slot. */ if (!worker->in_use || worker->generation != generation) - { - LWLockRelease(LogicalRepWorkerLock); return; - } /* Worker has assigned proc, so it has started. */ if (worker->proc) @@ -523,8 +558,6 @@ logicalrep_worker_stop(Oid subid, Oid relid) LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); } - - LWLockRelease(LogicalRepWorkerLock); } /* @@ -600,6 +633,28 @@ logicalrep_worker_attach(int slot) static void logicalrep_worker_detach(void) { + /* + * If we are the main apply worker, stop all the apply background workers + * we started before. + */ + if (!MyLogicalRepWorker->subworker) + { + List *workers; + ListCell *lc; + + LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); + + workers = logicalrep_workers_find(MyLogicalRepWorker->subid, true); + foreach(lc, workers) + { + LogicalRepWorker *w = (LogicalRepWorker *) lfirst(lc); + if (w->subworker) + logicalrep_worker_stop_internal(w); + } + + LWLockRelease(LogicalRepWorkerLock); + } + /* Block concurrent access. */ LWLockAcquire(LogicalRepWorkerLock, LW_EXCLUSIVE); @@ -622,6 +677,7 @@ logicalrep_worker_cleanup(LogicalRepWorker *worker) worker->userid = InvalidOid; worker->subid = InvalidOid; worker->relid = InvalidOid; + worker->subworker = false; } /* @@ -869,7 +925,7 @@ ApplyLauncherMain(Datum main_arg) wait_time = wal_retrieve_retry_interval; logicalrep_worker_launch(sub->dbid, sub->oid, sub->name, - sub->owner, InvalidOid); + sub->owner, InvalidOid, DSM_HANDLE_INVALID); } } diff --git a/src/backend/replication/logical/origin.c b/src/backend/replication/logical/origin.c index b0c8b6c..c6ea68a 100644 --- a/src/backend/replication/logical/origin.c +++ b/src/backend/replication/logical/origin.c @@ -1068,7 +1068,7 @@ ReplicationOriginExitCleanup(int code, Datum arg) * with replorigin_session_reset(). */ void -replorigin_session_setup(RepOriginId node) +replorigin_session_setup(RepOriginId node, bool acquire) { static bool registered_cleanup; int i; @@ -1110,7 +1110,11 @@ replorigin_session_setup(RepOriginId node) if (curstate->roident != node) continue; - else if (curstate->acquired_by != 0) + /* + * We allow the apply worker to get the slot which is acquired by its + * leader process. + */ + else if (curstate->acquired_by != 0 && acquire) { ereport(ERROR, (errcode(ERRCODE_OBJECT_IN_USE), @@ -1321,7 +1325,7 @@ pg_replication_origin_session_setup(PG_FUNCTION_ARGS) name = text_to_cstring((text *) DatumGetPointer(PG_GETARG_DATUM(0))); origin = replorigin_by_name(name, false); - replorigin_session_setup(origin); + replorigin_session_setup(origin, true); replorigin_session_origin = origin; diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c index ff8513e..0409fde 100644 --- a/src/backend/replication/logical/proto.c +++ b/src/backend/replication/logical/proto.c @@ -1138,14 +1138,11 @@ logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn, /* * Read STREAM COMMIT from the output stream. */ -TransactionId +void logicalrep_read_stream_commit(StringInfo in, LogicalRepCommitData *commit_data) { - TransactionId xid; uint8 flags; - xid = pq_getmsgint(in, 4); - /* read flags (unused for now) */ flags = pq_getmsgbyte(in); @@ -1156,8 +1153,6 @@ logicalrep_read_stream_commit(StringInfo in, LogicalRepCommitData *commit_data) commit_data->commit_lsn = pq_getmsgint64(in); commit_data->end_lsn = pq_getmsgint64(in); commit_data->committime = pq_getmsgint64(in); - - return xid; } /* diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c index 49ceec3..04c9f96 100644 --- a/src/backend/replication/logical/tablesync.c +++ b/src/backend/replication/logical/tablesync.c @@ -568,7 +568,8 @@ process_syncing_tables_for_apply(XLogRecPtr current_lsn) MySubscription->oid, MySubscription->name, MyLogicalRepWorker->userid, - rstate->relid); + rstate->relid, + DSM_HANDLE_INVALID); hentry->last_start_time = now; } } @@ -589,6 +590,9 @@ process_syncing_tables_for_apply(XLogRecPtr current_lsn) void process_syncing_tables(XLogRecPtr current_lsn) { + if (MyLogicalRepWorker->subworker) + return; + if (am_tablesync_worker()) process_syncing_tables_for_sync(current_lsn); else @@ -1275,7 +1279,7 @@ LogicalRepSyncTableStart(XLogRecPtr *origin_startpos) * time this tablesync was launched. */ originid = replorigin_by_name(originname, false); - replorigin_session_setup(originid); + replorigin_session_setup(originid, true); replorigin_session_origin = originid; *origin_startpos = replorigin_session_get_progress(false); @@ -1386,7 +1390,7 @@ LogicalRepSyncTableStart(XLogRecPtr *origin_startpos) true /* go backward */ , true /* WAL log */ ); UnlockRelationOid(ReplicationOriginRelationId, RowExclusiveLock); - replorigin_session_setup(originid); + replorigin_session_setup(originid, true); replorigin_session_origin = originid; } else diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c index 7da7823..a317f72 100644 --- a/src/backend/replication/logical/worker.c +++ b/src/backend/replication/logical/worker.c @@ -22,8 +22,25 @@ * STREAMED TRANSACTIONS * --------------------- * Streamed transactions (large transactions exceeding a memory limit on the - * upstream) are not applied immediately, but instead, the data is written - * to temporary files and then applied at once when the final commit arrives. + * upstream) are applied via one of two approaches. + * + * 1) Separate background workers + * + * Assign a new bgworker (if available) as soon as the xact's first stream is + * received and the main apply worker will send changes to this new worker via + * shared memory. We keep this worker assigned till the transaction commit is + * received and also wait for the worker to finish at commit. This preserves + * commit ordering and avoids writing to and reading from file in most cases. + * We still need to spill if there is no worker available. We also need to + * allow stream_stop to complete by the background worker to finish it to avoid + * deadlocks because T-1's current stream of changes can update rows in + * conflicting order with T-2's next stream of changes. + * + * 2) Write to temporary files and apply when the final commit arrives + * + * If no worker is available to handle streamed transaction, the data is + * written to temporary files and then applied at once when the final commit + * arrives. * * Unlike the regular (non-streamed) case, handling streamed transactions has * to handle aborts of both the toplevel transaction and subtransactions. This @@ -174,11 +191,15 @@ #include "rewrite/rewriteHandler.h" #include "storage/buffile.h" #include "storage/bufmgr.h" +#include "storage/dsm.h" #include "storage/fd.h" #include "storage/ipc.h" #include "storage/lmgr.h" #include "storage/proc.h" #include "storage/procarray.h" +#include "storage/shm_mq.h" +#include "storage/shm_toc.h" +#include "storage/spin.h" #include "tcop/tcopprot.h" #include "utils/acl.h" #include "utils/builtins.h" @@ -198,6 +219,75 @@ #define NAPTIME_PER_CYCLE 1000 /* max sleep time between cycles (1s) */ +#define PG_LOGICAL_APPLY_SHM_MAGIC 0x79fb2447 // TODO Consider change + +/* + * Shared information among apply workers. + */ +typedef struct ParallelState +{ + slock_t mutex; + + /* State for apply background worker. */ + char state; + + TransactionId stream_xid; + uint32 n; /* id of apply background worker */ +} ParallelState; + +/* + * States for apply background worker. + */ +#define APPLY_BGWORKER_ATTACHED 'a' +#define APPLY_BGWORKER_READY 'r' +#define APPLY_BGWORKER_BUSY 'b' +#define APPLY_BGWORKER_FINISHED 'f' +#define APPLY_BGWORKER_EXIT 'e' + +/* + * Struct for maintaining a apply background worker. + */ +typedef struct WorkerState +{ + shm_mq_handle *mq_handle; + dsm_segment *dsm_seg; + ParallelState volatile *pstate; +} WorkerState; + +typedef struct WorkerEntry +{ + TransactionId xid; + WorkerState *wstate; +} WorkerEntry; + +/* Apply background workers hash table (initialized on first use) */ +static HTAB *ApplyWorkersHash = NULL; +static List *ApplyWorkersIdleList = NIL; +static List *ApplyWorkersList = NIL; + +/* Fields valid only for apply background workers */ +volatile ParallelState *MyParallelState = NULL; +static List *subxactlist = NIL; + +/* The number of changes during one streaming block */ +static uint32 nchanges = 0; + +/* Worker setup and interactions */ +static WorkerState *apply_bgworker_setup(void); +static WorkerState *find_or_start_apply_bgworker(TransactionId xid, + bool start); +static void apply_bgworker_setup_dsm(WorkerState *wstate); +static void apply_bgworker_wait_for(WorkerState *wstate, + char wait_for_state); +static void apply_bgworker_send_data(WorkerState *wstate, Size nbytes, + const void *data); +static void apply_bgworker_free(WorkerState *wstate); +static void apply_bgworker_check_status(void); +static void apply_bgworker_set_state(char state); + +#define am_apply_bgworker() (MyLogicalRepWorker->subworker) +#define applying_changes_in_bgworker() (in_streamed_transaction && stream_apply_worker != NULL) + typedef struct FlushPosition { dlist_node node; @@ -260,18 +350,20 @@ static XLogRecPtr remote_final_lsn = InvalidXLogRecPtr; static bool in_streamed_transaction = false; static TransactionId stream_xid = InvalidTransactionId; +static WorkerState *stream_apply_worker = NULL; /* * We enable skipping all data modification changes (INSERT, UPDATE, etc.) for * the subscription if the remote transaction's finish LSN matches the subskiplsn. * Once we start skipping changes, we don't stop it until we skip all changes of * the transaction even if pg_subscription is updated and MySubscription->skiplsn - * gets changed or reset during that. Also, in streaming transaction cases, we - * don't skip receiving and spooling the changes since we decide whether or not + * gets changed or reset during that. Also, in streaming transaction cases (streaming = on), + * we don't skip receiving and spooling the changes since we decide whether or not * to skip applying the changes when starting to apply changes. The subskiplsn is * cleared after successfully skipping the transaction or applying non-empty * transaction. The latter prevents the mistakenly specified subskiplsn from - * being left. + * being left. Note that we cannot skip the streaming transaction in apply mode, + * because we cannot get the finish LSN before applying the changes. */ static XLogRecPtr skip_xact_finish_lsn = InvalidXLogRecPtr; #define is_skipping_changes() (unlikely(!XLogRecPtrIsInvalid(skip_xact_finish_lsn))) @@ -426,43 +518,103 @@ end_replication_step(void) } /* - * Handle streamed transactions. + * Handle streamed transactions for both main apply worker and apply background + * worker. * - * If in streaming mode (receiving a block of streamed transaction), we - * simply redirect it to a file for the proper toplevel transaction. + * For the main apply worker, if in streaming mode (receiving a block of + * streamed transaction), we send the data to the apply background worker. + * + * For the apply background worker, define a savepoint if new subtransaction + * was started. * * Returns true for streamed transactions, false otherwise (regular mode). */ static bool handle_streamed_transaction(LogicalRepMsgType action, StringInfo s) { - TransactionId xid; + TransactionId current_xid = InvalidTransactionId; - /* not in streaming mode */ - if (!in_streamed_transaction) + /* + * Return if we are not in streaming mode and are not in a apply background + * worker. + */ + if (!in_streamed_transaction && !am_apply_bgworker()) return false; - Assert(stream_fd != NULL); Assert(TransactionIdIsValid(stream_xid)); /* * We should have received XID of the subxact as the first part of the * message, so extract it. */ - xid = pq_getmsgint(s, 4); + current_xid = pq_getmsgint(s, 4); - if (!TransactionIdIsValid(xid)) + if (!TransactionIdIsValid(current_xid)) ereport(ERROR, (errcode(ERRCODE_PROTOCOL_VIOLATION), errmsg_internal("invalid transaction ID in streamed replication transaction"))); - /* Add the new subxact to the array (unless already there). */ - subxact_info_add(xid); + if (am_apply_bgworker()) + { + /* + * Inside apply background worker we can figure out that new + * subtransaction was started if new change arrived with different xid. + * In that case we can define named savepoint, so that we were able to + * commit/rollback it separately later. + * + * Special case is if the first change comes from subtransaction, then + * we check that current_xid differs from stream_xid. + */ + if (current_xid != stream_xid && + !list_member_int(subxactlist, (int) current_xid)) + { + MemoryContext oldctx; + char spname[MAXPGPATH]; + + snprintf(spname, MAXPGPATH, "savepoint_for_xid_%u", current_xid); + + elog(LOG, "[Apply BGW #%u] defining savepoint %s", + MyParallelState->n, spname); + + DefineSavepoint(spname); + CommitTransactionCommand(); + + oldctx = MemoryContextSwitchTo(ApplyContext); + subxactlist = lappend_int(subxactlist, (int) current_xid); + MemoryContextSwitchTo(oldctx); + } + } + else if (applying_changes_in_bgworker()) + { + /* + * If we decided to apply the changes of this transaction in a apply + * background worker, pass the data to the worker. + */ + apply_bgworker_send_data(stream_apply_worker, s->len, s->data); + nchanges += 1; + + /* + * XXX The publisher side doesn't always send relation update message + * after the streaming transaction, so update the relation in main + * apply worker here. + */ + if (action == LOGICAL_REP_MSG_RELATION) + { + LogicalRepRelation *rel = logicalrep_read_rel(s); + logicalrep_relmap_update(rel); + } + + } + else + { + /* Add the new subxact to the array (unless already there). */ + subxact_info_add(current_xid); - /* write the change to the current file */ - stream_write_change(action, s); + /* write the change to the current file */ + stream_write_change(action, s); + } - return true; + return !am_apply_bgworker(); } /* @@ -844,6 +996,9 @@ apply_handle_commit(StringInfo s) apply_handle_commit_internal(&commit_data); + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); + /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(commit_data.end_lsn); @@ -898,7 +1053,9 @@ apply_handle_prepare_internal(LogicalRepPreparedTxnData *prepare_data) * BeginTransactionBlock is necessary to balance the EndTransactionBlock * called within the PrepareTransactionBlock below. */ - BeginTransactionBlock(); + if (!IsTransactionBlock()) + BeginTransactionBlock(); + CommitTransactionCommand(); /* Completes the preceding Begin command. */ /* @@ -950,6 +1107,9 @@ apply_handle_prepare(StringInfo s) in_remote_transaction = false; + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); + /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(prepare_data.end_lsn); @@ -974,32 +1134,57 @@ apply_handle_commit_prepared(StringInfo s) { LogicalRepCommitPreparedTxnData prepare_data; char gid[GIDSIZE]; + WorkerState *wstate; logicalrep_read_commit_prepared(s, &prepare_data); set_apply_error_context_xact(prepare_data.xid, prepare_data.commit_lsn); - /* Compute GID for two_phase transactions. */ - TwoPhaseTransactionGid(MySubscription->oid, prepare_data.xid, - gid, sizeof(gid)); + /* Check if we have prepared transaction in another bgworker */ + wstate = find_or_start_apply_bgworker(prepare_data.xid, false); - /* There is no transaction when COMMIT PREPARED is called */ - begin_replication_step(); + if (wstate) + { + elog(DEBUG1, "received commit for streamed transaction %u", prepare_data.xid); - /* - * Update origin state so we can restart streaming from correct position - * in case of crash. - */ - replorigin_session_origin_lsn = prepare_data.end_lsn; - replorigin_session_origin_timestamp = prepare_data.commit_time; + /* Send commit message */ + apply_bgworker_send_data(wstate, s->len, s->data); + + /* Wait for bgworker to finish */ + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_FINISHED); + + apply_bgworker_free(wstate); + } + else + { + /* Compute GID for two_phase transactions. */ + TwoPhaseTransactionGid(MySubscription->oid, prepare_data.xid, + gid, sizeof(gid)); + + /* There is no transaction when COMMIT PREPARED is called */ + begin_replication_step(); + + /* + * Update origin state so we can restart streaming from correct position + * in case of crash. + */ + replorigin_session_origin_lsn = prepare_data.end_lsn; + replorigin_session_origin_timestamp = prepare_data.commit_time; + + FinishPreparedTransaction(gid, true); + end_replication_step(); + CommitTransactionCommand(); + + apply_bgworker_set_state(APPLY_BGWORKER_FINISHED); + } - FinishPreparedTransaction(gid, true); - end_replication_step(); - CommitTransactionCommand(); pgstat_report_stat(false); store_flush_position(prepare_data.end_lsn); in_remote_transaction = false; + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); + /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(prepare_data.end_lsn); @@ -1017,37 +1202,57 @@ apply_handle_rollback_prepared(StringInfo s) { LogicalRepRollbackPreparedTxnData rollback_data; char gid[GIDSIZE]; + WorkerState *wstate; logicalrep_read_rollback_prepared(s, &rollback_data); set_apply_error_context_xact(rollback_data.xid, rollback_data.rollback_end_lsn); - /* Compute GID for two_phase transactions. */ - TwoPhaseTransactionGid(MySubscription->oid, rollback_data.xid, - gid, sizeof(gid)); + /* Check if we are processing the prepared transaction in a bgworker */ + wstate = find_or_start_apply_bgworker(rollback_data.xid, false); - /* - * It is possible that we haven't received prepare because it occurred - * before walsender reached a consistent point or the two_phase was still - * not enabled by that time, so in such cases, we need to skip rollback - * prepared. - */ - if (LookupGXact(gid, rollback_data.prepare_end_lsn, - rollback_data.prepare_time)) + if (wstate) + { + apply_bgworker_send_data(wstate, s->len, s->data); + + /* Wait for bgworker to finish */ + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_FINISHED); + + elog(LOG, "rollback prepared streaming of xid %u", rollback_data.xid); + + apply_bgworker_free(wstate); + } + else { + /* Compute GID for two_phase transactions. */ + TwoPhaseTransactionGid(MySubscription->oid, rollback_data.xid, + gid, sizeof(gid)); + /* - * Update origin state so we can restart streaming from correct - * position in case of crash. + * It is possible that we haven't received prepare because it occurred + * before walsender reached a consistent point or the two_phase was still + * not enabled by that time, so in such cases, we need to skip rollback + * prepared. */ - replorigin_session_origin_lsn = rollback_data.rollback_end_lsn; - replorigin_session_origin_timestamp = rollback_data.rollback_time; + if (LookupGXact(gid, rollback_data.prepare_end_lsn, + rollback_data.prepare_time)) + { + /* + * Update origin state so we can restart streaming from correct + * position in case of crash. + */ + replorigin_session_origin_lsn = rollback_data.rollback_end_lsn; + replorigin_session_origin_timestamp = rollback_data.rollback_time; - /* There is no transaction when ABORT/ROLLBACK PREPARED is called */ - begin_replication_step(); - FinishPreparedTransaction(gid, false); - end_replication_step(); - CommitTransactionCommand(); + /* There is no transaction when ABORT/ROLLBACK PREPARED is called */ + begin_replication_step(); + FinishPreparedTransaction(gid, false); + end_replication_step(); + CommitTransactionCommand(); + + clear_subscription_skip_lsn(rollback_data.rollback_end_lsn); + } - clear_subscription_skip_lsn(rollback_data.rollback_end_lsn); + apply_bgworker_set_state(APPLY_BGWORKER_FINISHED); } pgstat_report_stat(false); @@ -1055,6 +1260,9 @@ apply_handle_rollback_prepared(StringInfo s) store_flush_position(rollback_data.rollback_end_lsn); in_remote_transaction = false; + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); + /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(rollback_data.rollback_end_lsn); @@ -1063,11 +1271,131 @@ apply_handle_rollback_prepared(StringInfo s) } /* - * Handle STREAM PREPARE. + * Look up worker inside ApplyWorkersHash for requested xid. * - * Logic is in two parts: - * 1. Replay all the spooled operations - * 2. Mark the transaction as prepared + * If start flag is true, try to start a new worker if not found in hash table. + */ +static WorkerState * +find_or_start_apply_bgworker(TransactionId xid, bool start) +{ + bool found; + WorkerState *wstate; + WorkerEntry *entry = NULL; + + if (!TransactionIdIsValid(xid)) + return NULL; + + if (!start && ApplyWorkersHash == NULL) + return NULL; + + /* + * We don't start new background worker if we are not in streaming apply + * mode. + */ + if (MySubscription->stream != SUBSTREAM_APPLY) + return NULL; + + /* + * We don't start new background worker if user has set skiplsn as it's + * possible that user want to skip the streaming transaction. For streaming + * transaction, we need to spill the transaction to disk so that we can get + * the last LSN of the transaction to judge whether to skip before starting + * to apply the change. + */ + if (start && !XLogRecPtrIsInvalid(MySubscription->skiplsn)) + return NULL; + + /* + * For streaming transactions that are being applied in bgworker, we cannot + * decide whether to apply the change for a relation that is not in the + * READY state (see should_apply_changes_for_rel) as we won't know + * remote_final_lsn by that time. So, we don't start new bgworker in this + * case. + */ + if (start && !AllTablesyncsReady()) + return NULL; + + /* First time through, initialize apply workers hashtable */ + if (ApplyWorkersHash == NULL) + { + HASHCTL ctl; + + MemSet(&ctl, 0, sizeof(ctl)); + ctl.keysize = sizeof(TransactionId); + ctl.entrysize = sizeof(WorkerEntry); + ctl.hcxt = ApplyContext; + + ApplyWorkersHash = hash_create("logical apply workers hash", 8, &ctl, + HASH_ELEM | HASH_BLOBS | HASH_CONTEXT); + } + + /* + * Find entry for requested transaction. + */ + entry = hash_search(ApplyWorkersHash, &xid, start ? HASH_ENTER : HASH_FIND, + &found); + if (found) + { + entry->wstate->pstate->state = APPLY_BGWORKER_BUSY; + return entry->wstate; + } + else if (!start) + return NULL; + + /* If there is at least one worker in the idle list, then take one. */ + if (list_length(ApplyWorkersIdleList) > 0) + { + wstate = (WorkerState *) llast(ApplyWorkersIdleList); + ApplyWorkersIdleList = list_delete_last(ApplyWorkersIdleList); + } + else + { + wstate = apply_bgworker_setup(); + + if (wstate == NULL) + { + /* + * If the bgworker cannot be launched, remove entry in hash table. + */ + hash_search(ApplyWorkersHash, &xid, HASH_REMOVE, &found); + return NULL; + } + } + + /* Fill up the hash entry */ + wstate->pstate->state = APPLY_BGWORKER_BUSY; + wstate->pstate->stream_xid = xid; + entry->wstate = wstate; + entry->xid = xid; + + return wstate; +} + +/* + * Add the worker to the freelist and remove the entry from hash table. + */ +static void +apply_bgworker_free(WorkerState *wstate) +{ + bool found; + MemoryContext oldctx; + TransactionId xid = wstate->pstate->stream_xid; + + oldctx = MemoryContextSwitchTo(ApplyContext); + + hash_search(ApplyWorkersHash, &xid, + HASH_REMOVE, &found); + + elog(LOG, "adding finished apply worker #%u for xid %u to the idle list", + wstate->pstate->n, wstate->pstate->stream_xid); + + ApplyWorkersIdleList = lappend(ApplyWorkersIdleList, wstate); + + MemoryContextSwitchTo(oldctx); +} + +/* + * Handle STREAM PREPARE. */ static void apply_handle_stream_prepare(StringInfo s) @@ -1088,24 +1416,71 @@ apply_handle_stream_prepare(StringInfo s) logicalrep_read_stream_prepare(s, &prepare_data); set_apply_error_context_xact(prepare_data.xid, prepare_data.prepare_lsn); - elog(DEBUG1, "received prepare for streamed transaction %u", prepare_data.xid); + /* + * If we are in a bgworker, just prepare the transaction. + */ + if (am_apply_bgworker()) + { + elog(LOG, "received prepare for streamed transaction %u", prepare_data.xid); - /* Replay all the spooled operations. */ - apply_spooled_messages(prepare_data.xid, prepare_data.prepare_lsn); + /* Mark the transaction as prepared. */ + apply_handle_prepare_internal(&prepare_data); - /* Mark the transaction as prepared. */ - apply_handle_prepare_internal(&prepare_data); + CommitTransactionCommand(); - CommitTransactionCommand(); + pgstat_report_stat(false); - pgstat_report_stat(false); + list_free(subxactlist); + subxactlist = NIL; - store_flush_position(prepare_data.end_lsn); + apply_bgworker_set_state(APPLY_BGWORKER_READY); + } + else + { + WorkerState *wstate = find_or_start_apply_bgworker(prepare_data.xid, false); + + /* + * If we are in main apply worker, check if we are processing this + * transaction in a bgworker. + */ + if (wstate) + { + apply_bgworker_send_data(wstate, s->len, s->data); + + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_READY); + + pgstat_report_stat(false); + store_flush_position(prepare_data.end_lsn); + } + else + { + /* + * If we are in main apply worker and the transaction has been + * serialized to file, replay all the spooled operations. + */ + apply_spooled_messages(prepare_data.xid, prepare_data.prepare_lsn); + + /* Mark the transaction as prepared. */ + apply_handle_prepare_internal(&prepare_data); + + CommitTransactionCommand(); + + pgstat_report_stat(false); + + store_flush_position(prepare_data.end_lsn); + + in_remote_transaction = false; + + /* unlink the files with serialized changes and subxact info. */ + stream_cleanup_files(MyLogicalRepWorker->subid, prepare_data.xid); + } + } in_remote_transaction = false; + stream_apply_worker = NULL; - /* unlink the files with serialized changes and subxact info. */ - stream_cleanup_files(MyLogicalRepWorker->subid, prepare_data.xid); + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(prepare_data.end_lsn); @@ -1155,15 +1530,6 @@ apply_handle_stream_start(StringInfo s) (errcode(ERRCODE_PROTOCOL_VIOLATION), errmsg_internal("duplicate STREAM START message"))); - /* - * Start a transaction on stream start, this transaction will be committed - * on the stream stop unless it is a tablesync worker in which case it - * will be committed after processing all the messages. We need the - * transaction for handling the buffile, used for serializing the - * streaming data and subxact info. - */ - begin_replication_step(); - /* notify handle methods we're processing a remote transaction */ in_streamed_transaction = true; @@ -1177,36 +1543,90 @@ apply_handle_stream_start(StringInfo s) set_apply_error_context_xact(stream_xid, InvalidXLogRecPtr); + if (am_apply_bgworker()) + { + /* + * Make sure the handle apply_dispatch methods are aware we're in a + * remote transaction. + */ + in_remote_transaction = true; + + /* If we are in a bgworker, begin the transaction */ + AcceptInvalidationMessages(); + maybe_reread_subscription(); + + StartTransactionCommand(); + BeginTransactionBlock(); + CommitTransactionCommand(); + + return; + } + /* - * Initialize the worker's stream_fileset if we haven't yet. This will be - * used for the entire duration of the worker so create it in a permanent - * context. We create this on the very first streaming message from any - * transaction and then use it for this and other streaming transactions. - * Now, we could create a fileset at the start of the worker as well but - * then we won't be sure that it will ever be used. + * If we are in main apply worker, check if there is any free bgworker + * we can use to process this transaction. */ - if (MyLogicalRepWorker->stream_fileset == NULL) + stream_apply_worker = find_or_start_apply_bgworker(stream_xid, first_segment); + + if (applying_changes_in_bgworker()) { - MemoryContext oldctx; + /* + * If we have free worker or we already started to apply this + * transaction in bgworker, we pass the data to worker. + */ + if (first_segment) + apply_bgworker_send_data(stream_apply_worker, s->len, s->data); - oldctx = MemoryContextSwitchTo(ApplyContext); + nchanges = 0; + elog(LOG, "starting streaming of xid %u", stream_xid); + } - MyLogicalRepWorker->stream_fileset = palloc(sizeof(FileSet)); - FileSetInit(MyLogicalRepWorker->stream_fileset); + /* + * If no worker is available for the first stream start, we start to + * serialize all the changes of the transaction. + */ + else + { + /* + * Start a transaction on stream start, this transaction will be committed + * on the stream stop unless it is a tablesync worker in which case it + * will be committed after processing all the messages. We need the + * transaction for handling the buffile, used for serializing the + * streaming data and subxact info. + */ + begin_replication_step(); - MemoryContextSwitchTo(oldctx); - } + /* + * Initialize the worker's stream_fileset if we haven't yet. This will be + * used for the entire duration of the worker so create it in a permanent + * context. We create this on the very first streaming message from any + * transaction and then use it for this and other streaming transactions. + * Now, we could create a fileset at the start of the worker as well but + * then we won't be sure that it will ever be used. + */ + if (MyLogicalRepWorker->stream_fileset == NULL) + { + MemoryContext oldctx; - /* open the spool file for this transaction */ - stream_open_file(MyLogicalRepWorker->subid, stream_xid, first_segment); + oldctx = MemoryContextSwitchTo(ApplyContext); - /* if this is not the first segment, open existing subxact file */ - if (!first_segment) - subxact_info_read(MyLogicalRepWorker->subid, stream_xid); + MyLogicalRepWorker->stream_fileset = palloc(sizeof(FileSet)); + FileSetInit(MyLogicalRepWorker->stream_fileset); - pgstat_report_activity(STATE_RUNNING, NULL); + MemoryContextSwitchTo(oldctx); + } - end_replication_step(); + /* open the spool file for this transaction */ + stream_open_file(MyLogicalRepWorker->subid, stream_xid, first_segment); + + /* if this is not the first segment, open existing subxact file */ + if (!first_segment) + subxact_info_read(MyLogicalRepWorker->subid, stream_xid); + + end_replication_step(); + } + + pgstat_report_activity(STATE_RUNNING, NULL); } /* @@ -1220,44 +1640,47 @@ apply_handle_stream_stop(StringInfo s) (errcode(ERRCODE_PROTOCOL_VIOLATION), errmsg_internal("STREAM STOP message without STREAM START"))); - /* - * Close the file with serialized changes, and serialize information about - * subxacts for the toplevel transaction. - */ - subxact_info_write(MyLogicalRepWorker->subid, stream_xid); - stream_close_file(); + if (applying_changes_in_bgworker()) + { + char action = LOGICAL_REP_MSG_STREAM_STOP; - /* We must be in a valid transaction state */ - Assert(IsTransactionState()); + apply_bgworker_send_data(stream_apply_worker, 1, &action); + apply_bgworker_wait_for(stream_apply_worker, APPLY_BGWORKER_READY); - /* Commit the per-stream transaction */ - CommitTransactionCommand(); + elog(LOG, "stopped streaming of xid %u, %u changes streamed", stream_xid, nchanges); + } + else + { + /* + * Close the file with serialized changes, and serialize information + * about subxacts for the toplevel transaction. + */ + subxact_info_write(MyLogicalRepWorker->subid, stream_xid); + stream_close_file(); - in_streamed_transaction = false; + /* We must be in a valid transaction state */ + Assert(IsTransactionState()); + + /* Commit the per-stream transaction */ + CommitTransactionCommand(); - /* Reset per-stream context */ - MemoryContextReset(LogicalStreamingContext); + /* Reset per-stream context */ + MemoryContextReset(LogicalStreamingContext); + } + + in_streamed_transaction = false; + stream_apply_worker = NULL; pgstat_report_activity(STATE_IDLE, NULL); reset_apply_error_context_info(); } /* - * Handle STREAM abort message. + * Handle STREAM ABORT message when the transaction was spilled to disk. */ static void -apply_handle_stream_abort(StringInfo s) +serialize_stream_abort(TransactionId xid, TransactionId subxid) { - TransactionId xid; - TransactionId subxid; - - if (in_streamed_transaction) - ereport(ERROR, - (errcode(ERRCODE_PROTOCOL_VIOLATION), - errmsg_internal("STREAM ABORT message without STREAM STOP"))); - - logicalrep_read_stream_abort(s, &xid, &subxid); - /* * If the two XIDs are the same, it's in fact abort of toplevel xact, so * just delete the files with serialized info. @@ -1339,8 +1762,120 @@ apply_handle_stream_abort(StringInfo s) end_replication_step(); CommitTransactionCommand(); } +} - reset_apply_error_context_info(); +/* + * Handle STREAM ABORT message. + */ +static void +apply_handle_stream_abort(StringInfo s) +{ + TransactionId xid; + TransactionId subxid; + + if (in_streamed_transaction) + ereport(ERROR, + (errcode(ERRCODE_PROTOCOL_VIOLATION), + errmsg_internal("STREAM COMMIT message without STREAM STOP"))); + + logicalrep_read_stream_abort(s, &xid, &subxid); + + if (am_apply_bgworker()) + { + ereport(LOG, + (errmsg("[Apply BGW #%u] aborting current transaction xid=%u, subxid=%u", + MyParallelState->n, GetCurrentTransactionIdIfAny(), GetCurrentSubTransactionId()))); + + /* + * If the two XIDs are the same, it's in fact abort of toplevel xact, + * so just free the subxactlist. + */ + if (subxid == xid) + { + set_apply_error_context_xact(subxid, InvalidXLogRecPtr); + + AbortCurrentTransaction(); + + EndTransactionBlock(false); + CommitTransactionCommand(); + + in_remote_transaction = false; + + list_free(subxactlist); + subxactlist = NIL; + + apply_bgworker_set_state(APPLY_BGWORKER_FINISHED); + } + else + { + /* + * OK, so it's a subxact. Rollback to the savepoint. + * + * We also need to read the subxactlist, determine the offset + * tracked for the subxact, and truncate the list. + */ + int i; + bool found = false; + char spname[MAXPGPATH]; + + set_apply_error_context_xact(subxid, InvalidXLogRecPtr); + snprintf(spname, MAXPGPATH, "savepoint_for_xid_%u", subxid); + + ereport(LOG, + (errmsg("[Apply BGW #%u] rolling back to savepoint %s", + MyParallelState->n, spname))); + + for(i = list_length(subxactlist) - 1; i >= 0; i--) + { + xid = (TransactionId) list_nth_int(subxactlist, i); + if (xid == subxid) + { + found = true; + break; + } + } + + if (found) + { + elog(LOG, "rolled back to savepoint %s", spname); + RollbackToSavepoint(spname); + CommitTransactionCommand(); + subxactlist = list_truncate(subxactlist, i + 1); + } + + apply_bgworker_set_state(APPLY_BGWORKER_READY); + } + + reset_apply_error_context_info(); + } + else + { + WorkerState *wstate = find_or_start_apply_bgworker(xid, false); + + /* + * If we are in main apply worker, check if we are processing this + * transaction in a bgworker. + */ + if (wstate) + { + apply_bgworker_send_data(wstate, s->len, s->data); + + if (subxid == xid) + { + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_FINISHED); + apply_bgworker_free(wstate); + } + else + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_READY); + } + + /* + * We are in main apply worker and the transaction has been serialized + * to file. + */ + else + serialize_stream_abort(xid, subxid); + } } /* @@ -1463,40 +1998,6 @@ apply_spooled_messages(TransactionId xid, XLogRecPtr lsn) } /* - * Handle STREAM COMMIT message. - */ -static void -apply_handle_stream_commit(StringInfo s) -{ - TransactionId xid; - LogicalRepCommitData commit_data; - - if (in_streamed_transaction) - ereport(ERROR, - (errcode(ERRCODE_PROTOCOL_VIOLATION), - errmsg_internal("STREAM COMMIT message without STREAM STOP"))); - - xid = logicalrep_read_stream_commit(s, &commit_data); - set_apply_error_context_xact(xid, commit_data.commit_lsn); - - elog(DEBUG1, "received commit for streamed transaction %u", xid); - - apply_spooled_messages(xid, commit_data.commit_lsn); - - apply_handle_commit_internal(&commit_data); - - /* unlink the files with serialized changes and subxact info */ - stream_cleanup_files(MyLogicalRepWorker->subid, xid); - - /* Process any tables that are being synchronized in parallel. */ - process_syncing_tables(commit_data.end_lsn); - - pgstat_report_activity(STATE_IDLE, NULL); - - reset_apply_error_context_info(); -} - -/* * Helper function for apply_handle_commit and apply_handle_stream_commit. */ static void @@ -2445,6 +2946,101 @@ apply_handle_truncate(StringInfo s) end_replication_step(); } +/* + * Handle STREAM COMMIT message. + */ +static void +apply_handle_stream_commit(StringInfo s) +{ + LogicalRepCommitData commit_data; + TransactionId xid; + + if (in_streamed_transaction) + ereport(ERROR, + (errcode(ERRCODE_PROTOCOL_VIOLATION), + errmsg_internal("STREAM COMMIT message without STREAM STOP"))); + + xid = pq_getmsgint(s, 4); + logicalrep_read_stream_commit(s, &commit_data); + set_apply_error_context_xact(xid, commit_data.commit_lsn); + + elog(DEBUG1, "received commit for streamed transaction %u", xid); + + if (am_apply_bgworker()) + { + /* + * Update origin state so we can restart streaming from correct + * position in case of crash. + */ + replorigin_session_origin_lsn = commit_data.end_lsn; + replorigin_session_origin_timestamp = commit_data.committime; + + CommitTransactionCommand(); + EndTransactionBlock(false); + CommitTransactionCommand(); + + in_remote_transaction = false; + + pgstat_report_stat(false); + + list_free(subxactlist); + subxactlist = NIL; + + apply_bgworker_set_state(APPLY_BGWORKER_FINISHED); + } + else + { + /* + * If we are in main apply worker, check if we are processing this + * transaction in a apply background worker. + */ + WorkerState *wstate = find_or_start_apply_bgworker(xid, false); + + if (wstate) + { + /* Send commit message */ + apply_bgworker_send_data(wstate, s->len, s->data); + + /* Wait for apply background worker to finish */ + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_FINISHED); + + pgstat_report_stat(false); + store_flush_position(commit_data.end_lsn); + stop_skipping_changes(); + + apply_bgworker_free(wstate); + + /* + * The transaction is either non-empty or skipped, so we clear the + * subskiplsn. + */ + clear_subscription_skip_lsn(commit_data.commit_lsn); + } + else + { + /* + * If we are in main apply worker and the transaction has been + * serialized to file, replay all the spooled operations. + */ + apply_spooled_messages(xid, commit_data.commit_lsn); + + apply_handle_commit_internal(&commit_data); + + /* unlink the files with serialized changes and subxact info */ + stream_cleanup_files(MyLogicalRepWorker->subid, xid); + } + } + + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); + + /* Process any tables that are being synchronized in parallel. */ + process_syncing_tables(commit_data.end_lsn); + + pgstat_report_activity(STATE_IDLE, NULL); + + reset_apply_error_context_info(); +} /* * Logical replication protocol message dispatcher. @@ -2511,6 +3107,8 @@ apply_dispatch(StringInfo s) break; case LOGICAL_REP_MSG_STREAM_START: + /* FIXME : log for debugging here. */ + elog(LOG, "LOGICAL_REP_MSG_STREAM_START"); apply_handle_stream_start(s); break; @@ -2618,6 +3216,10 @@ store_flush_position(XLogRecPtr remote_lsn) { FlushPosition *flushpos; + /* We only need to collect the LSN in main apply worker */ + if (am_apply_bgworker()) + return; + /* Need to do this in permanent context */ MemoryContextSwitchTo(ApplyContext); @@ -2794,6 +3396,9 @@ LogicalRepApplyLoop(XLogRecPtr last_received) AcceptInvalidationMessages(); maybe_reread_subscription(); + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); + /* Process any table synchronization changes. */ process_syncing_tables(last_received); } @@ -3686,7 +4291,7 @@ ApplyWorkerMain(Datum main_arg) originid = replorigin_by_name(originname, true); if (!OidIsValid(originid)) originid = replorigin_create(originname); - replorigin_session_setup(originid); + replorigin_session_setup(originid, true); replorigin_session_origin = originid; origin_startpos = replorigin_session_get_progress(false); CommitTransactionCommand(); @@ -3733,7 +4338,7 @@ ApplyWorkerMain(Datum main_arg) options.proto.logical.publication_names = MySubscription->publications; options.proto.logical.binary = MySubscription->binary; - options.proto.logical.streaming = MySubscription->stream; + options.proto.logical.streaming = (MySubscription->stream != SUBSTREAM_OFF); options.proto.logical.twophase = false; if (!am_tablesync_worker()) @@ -3891,7 +4496,8 @@ clear_subscription_skip_lsn(XLogRecPtr finish_lsn) XLogRecPtr myskiplsn = MySubscription->skiplsn; bool started_tx = false; - if (likely(XLogRecPtrIsInvalid(myskiplsn))) + if (likely(XLogRecPtrIsInvalid(myskiplsn)) || + am_apply_bgworker()) return; if (!IsTransactionState()) @@ -4027,3 +4633,529 @@ reset_apply_error_context_info(void) apply_error_callback_arg.remote_attnum = -1; set_apply_error_context_xact(InvalidTransactionId, InvalidXLogRecPtr); } + +/* Apply Background Worker main loop */ +static void +LogicalApplyBgwLoop(shm_mq_handle *mqh, volatile ParallelState *pst) +{ + shm_mq_result shmq_res; + PGPROC *registrant; + ErrorContextCallback errcallback; + XLogRecPtr last_received = InvalidXLogRecPtr; + + registrant = BackendPidGetProc(MyBgworkerEntry->bgw_notify_pid); + SetLatch(®istrant->procLatch); + + /* + * Push apply error context callback. Fields will be filled during + * applying a change. + */ + errcallback.callback = apply_error_callback; + errcallback.previous = error_context_stack; + error_context_stack = &errcallback; + + for (;;) + { + void *data; + Size len; + StringInfoData s; + MemoryContext oldctx; + XLogRecPtr start_lsn; + XLogRecPtr end_lsn; + TimestampTz send_time; + + CHECK_FOR_INTERRUPTS(); + + /* Ensure we are reading the data into our memory context. */ + oldctx = MemoryContextSwitchTo(ApplyMessageContext); + + shmq_res = shm_mq_receive(mqh, &len, &data, false); + + if (shmq_res != SHM_MQ_SUCCESS) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("lost connection to the main apply worker"))); + + if (len == 0) + { + elog(LOG, "[Apply BGW #%u] got zero-length message, stopping", pst->n); + break; + } + + s.cursor = 0; + s.maxlen = -1; + s.data = (char *) data; + s.len = len; + + /* + * We use first byte of message for additional communication between + * main Logical replication worker and apply bgworkers, so if it + * differs from 'w', then process it first. + */ + switch (pq_getmsgbyte(&s)) + { + /* End message of streaming chunk */ + case LOGICAL_REP_MSG_STREAM_STOP: + elog(LOG, "[Apply BGW #%u] ended processing streaming chunk," + "waiting on shm_mq_receive", pst->n); + + apply_bgworker_set_state(APPLY_BGWORKER_READY); + + SetLatch(®istrant->procLatch); + + in_streamed_transaction = false; + pgstat_report_activity(STATE_IDLE, NULL); + continue; + case 'w': + break; + default: + elog(ERROR, "unexpected message"); + break; + } + + start_lsn = pq_getmsgint64(&s); + end_lsn = pq_getmsgint64(&s); + send_time = pq_getmsgint64(&s); + + if (last_received < start_lsn) + last_received = start_lsn; + + if (last_received < end_lsn) + last_received = end_lsn; + + /* + * TO IMPROVE: Do we need to display the bgworker's information in + * pg_stat_replication ? + */ + UpdateWorkerStats(last_received, send_time, false); + + apply_dispatch(&s); + + if (ConfigReloadPending) + { + ConfigReloadPending = false; + ProcessConfigFile(PGC_SIGHUP); + } + + MemoryContextSwitchTo(oldctx); + MemoryContextReset(ApplyMessageContext); + } + + MemoryContextSwitchTo(TopMemoryContext); + MemoryContextReset(ApplyContext); + + /* Pop the error context stack */ + error_context_stack = errcallback.previous; + + elog(LOG, "[Apply BGW #%u] exiting", pst->n); + + /* Signal main process that we are done. */ + SetLatch(®istrant->procLatch); +} + +/* + * Set the failed flag so that the main apply worker can realize we have + * shutdown. + */ +static void +ApplyBgwShutdown(int code, Datum arg) +{ + SpinLockAcquire(&MyParallelState->mutex); + MyParallelState->state = APPLY_BGWORKER_EXIT; + SpinLockRelease(&MyParallelState->mutex); + + dsm_detach((dsm_segment *) DatumGetPointer(arg)); +} + +/* + * Apply Background Worker entry point + */ +void +ApplyBgworkerMain(Datum main_arg) +{ + volatile ParallelState *pst; + + dsm_handle handle; + dsm_segment *seg; + shm_toc *toc; + shm_mq *mq; + shm_mq_handle *mqh; + MemoryContext oldcontext; + RepOriginId originid; + int worker_slot = DatumGetInt32(main_arg); + char originname[NAMEDATALEN]; + + /* Attach to slot */ + logicalrep_worker_attach(worker_slot); + + MemoryContextSwitchTo(TopMemoryContext); + + /* Load the subscription into persistent memory context. */ + ApplyContext = AllocSetContextCreate(TopMemoryContext, + "ApplyContext", + ALLOCSET_DEFAULT_SIZES); + + /* + * Init the ApplyMessageContext which we clean up after each replication + * protocol message. + */ + ApplyMessageContext = AllocSetContextCreate(ApplyContext, + "ApplyMessageContext", + ALLOCSET_DEFAULT_SIZES); + + /* Setup signal handling */ + pqsignal(SIGHUP, SignalHandlerForConfigReload); + pqsignal(SIGTERM, die); + BackgroundWorkerUnblockSignals(); + + /* + * Connect to the dynamic shared memory segment. + * + * The backend that registered this worker passed us the ID of a shared + * memory segment to which we must attach for further instructions. In + * order to attach to dynamic shared memory, we need a resource owner. + * Once we've mapped the segment in our address space, attach to the table + * of contents so we can locate the various data structures we'll need to + * find within the segment. + */ + CurrentResourceOwner = ResourceOwnerCreate(NULL, "Logical apply worker"); + memcpy(&handle, MyBgworkerEntry->bgw_extra, sizeof(dsm_handle)); + seg = dsm_attach(handle); + if (seg == NULL) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("unable to map dynamic shared memory segment"))); + toc = shm_toc_attach(PG_LOGICAL_APPLY_SHM_MAGIC, dsm_segment_address(seg)); + if (toc == NULL) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("bad magic number in dynamic shared memory segment"))); + + before_shmem_exit(ApplyBgwShutdown, PointerGetDatum(seg)); + + /* Look up the parallel state. */ + pst = shm_toc_lookup(toc, 0, false); + MyParallelState = pst; + + /* + * Attach to the message queue. + */ + mq = shm_toc_lookup(toc, 1, false); + shm_mq_set_receiver(mq, MyProc); + mqh = shm_mq_attach(mq, seg, NULL); + + /* Connect to our database. */ + BackgroundWorkerInitializeConnectionByOid(MyLogicalRepWorker->dbid, + MyLogicalRepWorker->userid, + 0); + + /* + * Set the client encoding to the database encoding, since that is what + * the leader will expect. + */ + SetClientEncoding(GetDatabaseEncoding()); + + stream_xid = pst->stream_xid; + + StartTransactionCommand(); + oldcontext = MemoryContextSwitchTo(ApplyContext); + + MySubscription = GetSubscription(MyLogicalRepWorker->subid, true); + if (!MySubscription) + { + ereport(LOG, + (errmsg("logical replication apply worker for subscription %u will not " + "start because the subscription was removed during startup", + MyLogicalRepWorker->subid))); + proc_exit(0); + } + + MySubscriptionValid = true; + MemoryContextSwitchTo(oldcontext); + + /* Setup synchronous commit according to the user's wishes */ + SetConfigOption("synchronous_commit", MySubscription->synccommit, + PGC_BACKEND, PGC_S_OVERRIDE); + + /* Keep us informed about subscription changes. */ + CacheRegisterSyscacheCallback(SUBSCRIPTIONOID, + subscription_change_cb, + (Datum) 0); + + CommitTransactionCommand(); + + /* Setup replication origin tracking. */ + StartTransactionCommand(); + snprintf(originname, sizeof(originname), "pg_%u", MySubscription->oid); + originid = replorigin_by_name(originname, true); + if (!OidIsValid(originid)) + originid = replorigin_create(originname); + replorigin_session_setup(originid, false); + replorigin_session_origin = originid; + CommitTransactionCommand(); + + /* + * Allocate the origin name in long-lived context for error context + * message. + */ + apply_error_callback_arg.origin_name = MemoryContextStrdup(ApplyContext, + originname); + + /* + * Indicate that we're fully initialized and ready to begin the main part + * of the apply operation. + */ + apply_bgworker_set_state(APPLY_BGWORKER_ATTACHED); + + elog(LOG, "[Apply BGW #%u] started", pst->n); + + PG_TRY(); + { + LogicalApplyBgwLoop(mqh, pst); + } + PG_CATCH(); + { + /* + * Report the worker failed while applying streaming transaction + * changes. Abort the current transaction so that the stats message is + * sent in an idle state. + */ + AbortOutOfAnyTransaction(); + pgstat_report_subscription_error(MySubscription->oid, false); + + PG_RE_THROW(); + } + PG_END_TRY(); + + /* + * We're done. Explicitly detach the shared memory segment so that we + * don't get a resource leak warning at commit time. This will fire any + * on_dsm_detach callbacks we've registered, as well. Once that's done, + * we can go ahead and exit. + */ + dsm_detach(seg); + proc_exit(0); +} + +/* + * Set up a dynamic shared memory segment. + * + * We set up a control region that contains a ParallelState, + * plus one region per message queue. There are as many message queues as + * the number of workers. + */ +static void +apply_bgworker_setup_dsm(WorkerState *wstate) +{ + shm_toc_estimator e; + int toc_key = 0; + Size segsize; + dsm_segment *seg; + shm_toc *toc; + ParallelState *pst; + shm_mq *mq; + int64 queue_size = 160000000; /* 16 MB for now */ + + /* + * Estimate how much shared memory we need. + * + * Because the TOC machinery may choose to insert padding of oddly-sized + * requests, we must estimate each chunk separately. + * + * We need one key to register the location of the header, and we need + * nworkers keys to track the locations of the message queues. + */ + shm_toc_initialize_estimator(&e); + shm_toc_estimate_chunk(&e, sizeof(ParallelState)); + shm_toc_estimate_chunk(&e, (Size) queue_size); + + shm_toc_estimate_keys(&e, 1 + 1); + segsize = shm_toc_estimate(&e); + + /* Create the shared memory segment and establish a table of contents. */ + seg = dsm_create(shm_toc_estimate(&e), 0); + toc = shm_toc_create(PG_LOGICAL_APPLY_SHM_MAGIC, dsm_segment_address(seg), + segsize); + + /* Set up the header region. */ + pst = shm_toc_allocate(toc, sizeof(ParallelState)); + SpinLockInit(&pst->mutex); + pst->state = APPLY_BGWORKER_BUSY; + pst->stream_xid = stream_xid; + pst->n = list_length(ApplyWorkersList) + 1; + + shm_toc_insert(toc, toc_key++, pst); + + /* Set up one message queue per worker, plus one. */ + mq = shm_mq_create(shm_toc_allocate(toc, (Size) queue_size), + (Size) queue_size); + shm_toc_insert(toc, toc_key++, mq); + shm_mq_set_sender(mq, MyProc); + + /* Attach the queues. */ + wstate->mq_handle = shm_mq_attach(mq, seg, NULL); + + /* Return results to caller. */ + wstate->dsm_seg = seg; + wstate->pstate = pst; +} + +/* + * Start apply worker background worker process and allocat shared memory for + * it. + */ +static WorkerState * +apply_bgworker_setup(void) +{ + MemoryContext oldcontext; + bool launched; + WorkerState *wstate; + + elog(LOG, "setting up apply worker #%u", list_length(ApplyWorkersList) + 1); + + oldcontext = MemoryContextSwitchTo(ApplyContext); + + wstate = (WorkerState *) palloc0(sizeof(WorkerState)); + + /* Setup shared memory */ + apply_bgworker_setup_dsm(wstate); + + launched = logicalrep_worker_launch(MyLogicalRepWorker->dbid, + MySubscription->oid, + MySubscription->name, + MyLogicalRepWorker->userid, + InvalidOid, + dsm_segment_handle(wstate->dsm_seg)); + + if (launched) + { + /* Wait for worker to become ready. */ + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_ATTACHED); + + ApplyWorkersList = lappend(ApplyWorkersList, wstate); + } + else + { + pfree(wstate); + wstate = NULL; + } + + MemoryContextSwitchTo(oldcontext); + + return wstate; +} + +/* + * Send the data to the specified apply bgworker via shared-memory queue. + */ +static void +apply_bgworker_send_data(WorkerState *wstate, Size nbytes, const void *data) +{ + shm_mq_result result; + + result = shm_mq_send(wstate->mq_handle, nbytes, data, false, true); + + if (result != SHM_MQ_SUCCESS) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("could not send tuples to shared-memory queue"))); +} + +/* + * Wait until the state of apply background worker reach the 'wait_for_state' + */ +static void +apply_bgworker_wait_for(WorkerState *wstate, char wait_for_state) +{ + for (;;) + { + char status; + + /* If the worker is ready, we have succeeded. */ + SpinLockAcquire(&wstate->pstate->mutex); + status = wstate->pstate->state; + SpinLockRelease(&wstate->pstate->mutex); + + if (status == wait_for_state) + break; + + /* If any workers (or the postmaster) have died, we have failed. */ + if (status == APPLY_BGWORKER_EXIT) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("Background worker %u failed to apply transaction %u", + wstate->pstate->n, wstate->pstate->stream_xid))); + + /* Wait to be signalled. */ + WaitLatch(MyLatch, WL_LATCH_SET | WL_EXIT_ON_PM_DEATH, 0, + WAIT_EVENT_LOGICAL_APPLY_WORKER_READY); + + /* Reset the latch so we don't spin. */ + ResetLatch(MyLatch); + + /* An interrupt may have occurred while we were waiting. */ + CHECK_FOR_INTERRUPTS(); + } +} + +/* + * Check the status of workers and report an error if any apply background + * worker has exited unexpectedly. + * + * Exit if any relation is not in the READY state and if any worker is handling + * the streaming transaction at the same time. Because for streaming + * transactions that is being applied in apply bgworker, we cannot decide whether to + * apply the change for a relation that is not in the READY state (see + * should_apply_changes_for_rel) as we won't know remote_final_lsn by that + * time. + */ +static void +apply_bgworker_check_status(void) +{ + ListCell *lc; + + if (am_apply_bgworker() || MySubscription->stream != SUBSTREAM_APPLY) + return; + + foreach(lc, ApplyWorkersList) + { + WorkerState *wstate = (WorkerState *) lfirst(lc); + + /* + * We don't lock here as in the worst case we will just detect the + * failure of worker a bit later. + */ + if (wstate->pstate->state == APPLY_BGWORKER_EXIT) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("Background worker %u exited unexpectedly", + wstate->pstate->n))); + } + + if (list_length(ApplyWorkersIdleList) != list_length(ApplyWorkersList) && + !AllTablesyncsReady()) + { + ereport(LOG, + (errmsg("logical replication apply workers for subscription \"%s\" will restart", + MySubscription->name), + errdetail("Cannot start table synchronization while bgworkers are " + "handling streamed replication transaction"))); + + proc_exit(0); + } +} + +/* Set the state of apply background worker */ +static void +apply_bgworker_set_state(char state) +{ + if (!am_apply_bgworker()) + return; + + elog(LOG, "[Apply BGW #%u] set state to %c", + MyParallelState->n, state); + + SpinLockAcquire(&MyParallelState->mutex); + MyParallelState->state = state; + SpinLockRelease(&MyParallelState->mutex); +} diff --git a/src/backend/utils/activity/wait_event.c b/src/backend/utils/activity/wait_event.c index 87c15b9..24a6fac 100644 --- a/src/backend/utils/activity/wait_event.c +++ b/src/backend/utils/activity/wait_event.c @@ -388,6 +388,9 @@ pgstat_get_wait_ipc(WaitEventIPC w) case WAIT_EVENT_HASH_GROW_BUCKETS_REINSERT: event_name = "HashGrowBucketsReinsert"; break; + case WAIT_EVENT_LOGICAL_APPLY_WORKER_READY: + event_name = "LogicalApplyWorkerReady"; + break; case WAIT_EVENT_LOGICAL_SYNC_DATA: event_name = "LogicalSyncData"; break; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 786d592..a476885 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -4449,7 +4449,7 @@ getSubscriptions(Archive *fout) if (fout->remoteVersion >= 140000) appendPQExpBufferStr(query, " s.substream,\n"); else - appendPQExpBufferStr(query, " false AS substream,\n"); + appendPQExpBufferStr(query, " 'f' AS substream,\n"); if (fout->remoteVersion >= 150000) appendPQExpBufferStr(query, @@ -4579,8 +4579,10 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo) if (strcmp(subinfo->subbinary, "t") == 0) appendPQExpBufferStr(query, ", binary = true"); - if (strcmp(subinfo->substream, "f") != 0) + if (strcmp(subinfo->substream, "o") == 0) appendPQExpBufferStr(query, ", streaming = on"); + else if (strcmp(subinfo->substream, "a") == 0) + appendPQExpBufferStr(query, ", streaming = apply"); if (strcmp(subinfo->subtwophasestate, two_phase_disabled) != 0) appendPQExpBufferStr(query, ", two_phase = on"); diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h index d1260f5..b5f270c 100644 --- a/src/include/catalog/pg_subscription.h +++ b/src/include/catalog/pg_subscription.h @@ -68,7 +68,7 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW bool subbinary; /* True if the subscription wants the * publisher to send data in binary */ - bool substream; /* Stream in-progress transactions. */ + char substream; /* Stream in-progress transactions. */ char subtwophasestate; /* Stream two-phase transactions */ @@ -109,7 +109,7 @@ typedef struct Subscription bool enabled; /* Indicates if the subscription is enabled */ bool binary; /* Indicates if the subscription wants data in * binary format */ - bool stream; /* Allow streaming in-progress transactions. */ + char stream; /* Allow streaming in-progress transactions. */ char twophasestate; /* Allow streaming two-phase transactions */ bool disableonerr; /* Indicates if the subscription should be * automatically disabled if a worker error @@ -120,6 +120,18 @@ typedef struct Subscription List *publications; /* List of publication names to subscribe to */ } Subscription; +/* Disallow streaming in-progress transactions */ +#define SUBSTREAM_OFF 'f' + +/* + * Streaming transactions are written to a temporary file and applied only + * after the transaction is committed on upstream. + */ +#define SUBSTREAM_ON 'o' + +/* Streaming transactions are appied immediately via a background worker */ +#define SUBSTREAM_APPLY 'a' + extern Subscription *GetSubscription(Oid subid, bool missing_ok); extern void FreeSubscription(Subscription *sub); extern void DisableSubscription(Oid subid); diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h index a771ab8..1eb8e12 100644 --- a/src/include/replication/logicalproto.h +++ b/src/include/replication/logicalproto.h @@ -243,8 +243,8 @@ extern TransactionId logicalrep_read_stream_start(StringInfo in, extern void logicalrep_write_stream_stop(StringInfo out); extern void logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn, XLogRecPtr commit_lsn); -extern TransactionId logicalrep_read_stream_commit(StringInfo out, - LogicalRepCommitData *commit_data); +extern void logicalrep_read_stream_commit(StringInfo out, + LogicalRepCommitData *commit_data); extern void logicalrep_write_stream_abort(StringInfo out, TransactionId xid, TransactionId subxid); extern void logicalrep_read_stream_abort(StringInfo in, TransactionId *xid, diff --git a/src/include/replication/logicalworker.h b/src/include/replication/logicalworker.h index cd1b6e8..6a1af7f 100644 --- a/src/include/replication/logicalworker.h +++ b/src/include/replication/logicalworker.h @@ -13,6 +13,7 @@ #define LOGICALWORKER_H extern void ApplyWorkerMain(Datum main_arg); +extern void ApplyBgworkerMain(Datum main_arg); extern bool IsLogicalWorker(void); diff --git a/src/include/replication/origin.h b/src/include/replication/origin.h index 14d5c49..1ed51bc 100644 --- a/src/include/replication/origin.h +++ b/src/include/replication/origin.h @@ -53,7 +53,7 @@ extern XLogRecPtr replorigin_get_progress(RepOriginId node, bool flush); extern void replorigin_session_advance(XLogRecPtr remote_commit, XLogRecPtr local_commit); -extern void replorigin_session_setup(RepOriginId node); +extern void replorigin_session_setup(RepOriginId node, bool acquire); extern void replorigin_session_reset(void); extern XLogRecPtr replorigin_session_get_progress(bool flush); diff --git a/src/include/replication/worker_internal.h b/src/include/replication/worker_internal.h index 4485d4e..4d11a13 100644 --- a/src/include/replication/worker_internal.h +++ b/src/include/replication/worker_internal.h @@ -60,6 +60,8 @@ typedef struct LogicalRepWorker */ FileSet *stream_fileset; + bool subworker; + /* Stats. */ XLogRecPtr last_lsn; TimestampTz last_send_time; @@ -84,8 +86,9 @@ extern void logicalrep_worker_attach(int slot); extern LogicalRepWorker *logicalrep_worker_find(Oid subid, Oid relid, bool only_running); extern List *logicalrep_workers_find(Oid subid, bool only_running); -extern void logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, - Oid userid, Oid relid); +extern bool logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, + Oid userid, Oid relid, + dsm_handle subworker_dsm); extern void logicalrep_worker_stop(Oid subid, Oid relid); extern void logicalrep_worker_wakeup(Oid subid, Oid relid); extern void logicalrep_worker_wakeup_ptr(LogicalRepWorker *worker); diff --git a/src/include/utils/wait_event.h b/src/include/utils/wait_event.h index b578e2e..d5081d9 100644 --- a/src/include/utils/wait_event.h +++ b/src/include/utils/wait_event.h @@ -105,6 +105,7 @@ typedef enum WAIT_EVENT_HASH_GROW_BUCKETS_ALLOCATE, WAIT_EVENT_HASH_GROW_BUCKETS_ELECT, WAIT_EVENT_HASH_GROW_BUCKETS_REINSERT, + WAIT_EVENT_LOGICAL_APPLY_WORKER_READY, WAIT_EVENT_LOGICAL_SYNC_DATA, WAIT_EVENT_LOGICAL_SYNC_STATE_CHANGE, WAIT_EVENT_MQ_INTERNAL, diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out index 7fcfad1..e08e4d4 100644 --- a/src/test/regress/expected/subscription.out +++ b/src/test/regress/expected/subscription.out @@ -197,7 +197,7 @@ ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); DROP SUBSCRIPTION regress_testsub; -- fail - streaming must be boolean CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = foo); -ERROR: streaming requires a Boolean value +ERROR: streaming requires a Boolean value or "apply" -- now it works CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = true); WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables @@ -205,7 +205,7 @@ WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION .. List of subscriptions Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable on error | Synchronous commit | Conninfo | Skip LSN -----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------+---------- - regress_testsub | regress_subscription_user | f | {testpub} | f | t | d | f | off | dbname=regress_doesnotexist | 0/0 + regress_testsub | regress_subscription_user | f | {testpub} | f | o | d | f | off | dbname=regress_doesnotexist | 0/0 (1 row) ALTER SUBSCRIPTION regress_testsub SET (streaming = false); @@ -299,7 +299,7 @@ ALTER SUBSCRIPTION regress_testsub SET (streaming = true); List of subscriptions Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable on error | Synchronous commit | Conninfo | Skip LSN -----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------+---------- - regress_testsub | regress_subscription_user | f | {testpub} | f | t | p | f | off | dbname=regress_doesnotexist | 0/0 + regress_testsub | regress_subscription_user | f | {testpub} | f | o | p | f | off | dbname=regress_doesnotexist | 0/0 (1 row) ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); @@ -311,7 +311,7 @@ WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION .. List of subscriptions Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable on error | Synchronous commit | Conninfo | Skip LSN -----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------+---------- - regress_testsub | regress_subscription_user | f | {testpub} | f | t | p | f | off | dbname=regress_doesnotexist | 0/0 + regress_testsub | regress_subscription_user | f | {testpub} | f | o | p | f | off | dbname=regress_doesnotexist | 0/0 (1 row) ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 87ee7bf..9c036d1 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -2918,11 +2918,13 @@ WordEntryPosVector WordEntryPosVector1 WorkTableScan WorkTableScanState +WorkerEntry WorkerInfo WorkerInfoData WorkerInstrumentation WorkerJobDumpPtrType WorkerJobRestorePtrType +WorkerState Working_State WriteBufPtrType WriteBytePtrType -- 2.7.2.windows.1 ^ permalink raw reply [nested|flat] 66+ messages in thread
* RE: Perform streaming logical transactions by background workers and parallel apply @ 2022-04-29 05:22 [email protected] <[email protected]> parent: [email protected] <[email protected]> 1 sibling, 1 reply; 66+ messages in thread From: [email protected] @ 2022-04-29 05:22 UTC (permalink / raw) To: [email protected] <[email protected]>; Peter Smith <[email protected]>; +Cc: Amit Kapila <[email protected]>; PostgreSQL Hackers <[email protected]> On Fri, Apr 29, 2022 10:07 AM Hou, Zhijie/侯 志杰 <[email protected]> wrote: > > I addressed the rest of Peter's comments and here is a new version patch. > Thanks for your patch. The patch modified streaming option in logical replication, it can be set to 'on', 'off' and 'apply'. The new option 'apply' haven't been tested in the tap test. Attach a patch which modified the subscription tap test to cover both 'on' and 'apply' option. (The main patch is also attached to make cfbot happy.) Besides, I noticed that for two-phase commit transactions, if the transaction is prepared by a background worker, the background worker would be asked to handle the message about commit/rollback this transaction. Is it possible that the messages about commit/rollback prepared transaction are handled by apply worker directly? Regards, Shi yu Attachments: [application/octet-stream] v5-0002-Test-streaming-apply-option-in-tap-test.patch (61.9K, ../../OSZPR01MB6310F0FABB05F8E5BB31A5B0FDFC9@OSZPR01MB6310.jpnprd01.prod.outlook.com/2-v5-0002-Test-streaming-apply-option-in-tap-test.patch) download | inline diff: From 680631bd58b9b565a94887f709af81f8e1ba2713 Mon Sep 17 00:00:00 2001 From: "shiy.fnst" <[email protected]> Date: Fri, 29 Apr 2022 11:01:20 +0800 Subject: [PATCH v5 2/2] Test streaming apply option in tap test For the tap tests about streaming option in logical replication, test both 'on' and 'apply' option. --- src/test/subscription/t/015_stream.pl | 168 ++++---- src/test/subscription/t/016_stream_subxact.pl | 84 ++-- src/test/subscription/t/017_stream_ddl.pl | 152 ++++--- .../t/018_stream_subxact_abort.pl | 165 +++++--- .../t/019_stream_subxact_ddl_abort.pl | 71 +++- .../subscription/t/022_twophase_cascade.pl | 278 ++++++------ .../subscription/t/023_twophase_stream.pl | 398 ++++++++++-------- 7 files changed, 740 insertions(+), 576 deletions(-) diff --git a/src/test/subscription/t/015_stream.pl b/src/test/subscription/t/015_stream.pl index 6561b189de..0985a3b2bf 100644 --- a/src/test/subscription/t/015_stream.pl +++ b/src/test/subscription/t/015_stream.pl @@ -8,6 +8,88 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname) = @_; + + # Interleave a pair of transactions, each exceeding the 64kB limit. + my $in = ''; + my $out = ''; + + my $timer = IPC::Run::timeout($PostgreSQL::Test::Utils::timeout_default); + + my $h = $node_publisher->background_psql('postgres', \$in, \$out, $timer, + on_error_stop => 0); + + $in .= q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + }; + $h->pump_nb; + + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001, 9999) s(i); + DELETE FROM test_tab WHERE a > 5000; + COMMIT; + }); + + $in .= q{ + COMMIT; + \q + }; + $h->finish; # errors make the next test fail, so ignore them here + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), 'check extra columns contain local defaults'); + + # Test the streaming in binary mode + $node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET (binary = on)"); + + # Insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001, 10000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + COMMIT; + }); + + $node_publisher->wait_for_catchup($appname); + + $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(6667|6667|6667), 'check extra columns contain local defaults'); + + # Change the local values of the extra columns on the subscriber, + # update publisher, and check that subscriber retains the expected + # values. This is to ensure that non-streaming transactions behave + # properly after a streaming transaction. + $node_subscriber->safe_psql('postgres', + "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'" + ); + $node_publisher->safe_psql('postgres', + "UPDATE test_tab SET b = md5(a::text)"); + + $node_publisher->wait_for_catchup($appname); + + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab" + ); + is($result, qq(6667|6667|6667), + 'check extra columns contain locally changed data'); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -37,99 +119,39 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +# Test streaming mode on $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); - $node_publisher->wait_for_catchup($appname); - # Also wait for initial table sync to finish my $synced_query = "SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');"; $node_subscriber->poll_query_until('postgres', $synced_query) or die "Timed out while waiting for subscriber to synchronize data"; - my $result = $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab"); is($result, qq(2|2|2), 'check initial data was copied to subscriber'); -# Interleave a pair of transactions, each exceeding the 64kB limit. -my $in = ''; -my $out = ''; - -my $timer = IPC::Run::timeout($PostgreSQL::Test::Utils::timeout_default); - -my $h = $node_publisher->background_psql('postgres', \$in, \$out, $timer, - on_error_stop => 0); - -$in .= q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -}; -$h->pump_nb; - -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001, 9999) s(i); -DELETE FROM test_tab WHERE a > 5000; -COMMIT; -}); - -$in .= q{ -COMMIT; -\q -}; -$h->finish; # errors make the next test fail, so ignore them here - -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), 'check extra columns contain local defaults'); - -# Test the streaming in binary mode -$node_subscriber->safe_psql('postgres', - "ALTER SUBSCRIPTION tap_sub SET (binary = on)"); - -# Insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001, 10000) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -COMMIT; -}); +test_streaming($node_publisher, $node_subscriber, $appname); +# Test streaming mode apply +$node_publisher->safe_psql('postgres', "DELETE FROM test_tab WHERE (a > 2)"); $node_publisher->wait_for_catchup($appname); -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(6667|6667|6667), 'check extra columns contain local defaults'); - -# Change the local values of the extra columns on the subscriber, -# update publisher, and check that subscriber retains the expected -# values. This is to ensure that non-streaming transactions behave -# properly after a streaming transaction. +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" +); $node_subscriber->safe_psql('postgres', - "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'" + "ALTER SUBSCRIPTION tap_sub SET(streaming = apply, binary = off)" ); -$node_publisher->safe_psql('postgres', - "UPDATE test_tab SET b = md5(a::text)"); +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" +) or die "Timed out while waiting for apply to restart after changing PUBLICATION"; -$node_publisher->wait_for_catchup($appname); - -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab" -); -is($result, qq(6667|6667|6667), - 'check extra columns contain locally changed data'); +test_streaming($node_publisher, $node_subscriber, $appname); $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/016_stream_subxact.pl b/src/test/subscription/t/016_stream_subxact.pl index f27f1694f2..bc49bf9e40 100644 --- a/src/test/subscription/t/016_stream_subxact.pl +++ b/src/test/subscription/t/016_stream_subxact.pl @@ -8,6 +8,46 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname) = @_; + + # Insert, update and delete enough rows to exceed 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series( 3, 500) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501, 1000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001, 1500) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + SAVEPOINT s3; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501, 2000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + SAVEPOINT s4; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001, 2500) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + COMMIT; + }); + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(1667|1667|1667), + 'check data was copied to subscriber in streaming mode and extra columns contain local defaults' + ); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -37,6 +77,8 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +# Test streaming mode on $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -54,40 +96,22 @@ my $result = "SELECT count(*), count(c), count(d = 999) FROM test_tab"); is($result, qq(2|2|2), 'check initial data was copied to subscriber'); -# Insert, update and delete enough rows to exceed 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series( 3, 500) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501, 1000) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001, 1500) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -SAVEPOINT s3; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501, 2000) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -SAVEPOINT s4; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001, 2500) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -COMMIT; -}); +test_streaming($node_publisher, $node_subscriber, $appname); +# Test streaming mode apply +$node_publisher->safe_psql('postgres', "DELETE FROM test_tab WHERE (a > 2)"); $node_publisher->wait_for_catchup($appname); -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(1667|1667|1667), - 'check data was copied to subscriber in streaming mode and extra columns contain local defaults' +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" +); +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = apply)" ); +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" +) or die "Timed out while waiting for apply to restart after changing PUBLICATION"; +test_streaming($node_publisher, $node_subscriber, $appname); $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/017_stream_ddl.pl b/src/test/subscription/t/017_stream_ddl.pl index 0bce63b716..fccac95f69 100644 --- a/src/test/subscription/t/017_stream_ddl.pl +++ b/src/test/subscription/t/017_stream_ddl.pl @@ -8,6 +8,82 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname) = @_; + + # a small (non-streamed) transaction with DDL and DML + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab VALUES (3, md5(3::text)); + ALTER TABLE test_tab ADD COLUMN c INT; + SAVEPOINT s1; + INSERT INTO test_tab VALUES (4, md5(4::text), -4); + COMMIT; + }); + + # large (streamed) transaction with DDL and DML + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(5, 1000) s(i); + ALTER TABLE test_tab ADD COLUMN d INT; + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001, 2000) s(i); + COMMIT; + }); + + # a small (non-streamed) transaction with DDL and DML + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab VALUES (2001, md5(2001::text), -2001, 2*2001); + ALTER TABLE test_tab ADD COLUMN e INT; + SAVEPOINT s1; + INSERT INTO test_tab VALUES (2002, md5(2002::text), -2002, 2*2002, -3*2002); + COMMIT; + }); + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d), count(e) FROM test_tab"); + is($result, qq(2002|1999|1002|1), + 'check data was copied to subscriber in streaming mode and extra columns contain local defaults' + ); + + # A large (streamed) transaction with DDL and DML. One of the DDL is performed + # after DML to ensure that we invalidate the schema sent for test_tab so that + # the next transaction has to send the schema again. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(2003,5000) s(i); + ALTER TABLE test_tab ADD COLUMN f INT; + COMMIT; + }); + + # A small transaction that won't get streamed. This is just to ensure that we + # send the schema again to reflect the last column added in the previous test. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i, 4*i FROM generate_series(5001,5005) s (i); + COMMIT; + }); + + $node_publisher->wait_for_catchup($appname); + + $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d), count(e), count(f) FROM test_tab"); + is($result, qq(5005|5002|4005|3004|5), + 'check data was copied to subscriber for both streaming and non-streaming transactions' + ); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -37,6 +113,8 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +# Test streaming mode on $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -54,76 +132,28 @@ my $result = "SELECT count(*), count(c), count(d = 999) FROM test_tab"); is($result, qq(2|0|0), 'check initial data was copied to subscriber'); -# a small (non-streamed) transaction with DDL and DML -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab VALUES (3, md5(3::text)); -ALTER TABLE test_tab ADD COLUMN c INT; -SAVEPOINT s1; -INSERT INTO test_tab VALUES (4, md5(4::text), -4); -COMMIT; -}); - -# large (streamed) transaction with DDL and DML -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(5, 1000) s(i); -ALTER TABLE test_tab ADD COLUMN d INT; -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001, 2000) s(i); -COMMIT; -}); +test_streaming($node_publisher, $node_subscriber, $appname); -# a small (non-streamed) transaction with DDL and DML +# Test streaming mode apply $node_publisher->safe_psql( 'postgres', q{ -BEGIN; -INSERT INTO test_tab VALUES (2001, md5(2001::text), -2001, 2*2001); -ALTER TABLE test_tab ADD COLUMN e INT; -SAVEPOINT s1; -INSERT INTO test_tab VALUES (2002, md5(2002::text), -2002, 2*2002, -3*2002); -COMMIT; +DELETE FROM test_tab WHERE (a > 2); +ALTER TABLE test_tab DROP COLUMN c, DROP COLUMN d, DROP COLUMN e, DROP COLUMN f; }); $node_publisher->wait_for_catchup($appname); -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d), count(e) FROM test_tab"); -is($result, qq(2002|1999|1002|1), - 'check data was copied to subscriber in streaming mode and extra columns contain local defaults' +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" ); - -# A large (streamed) transaction with DDL and DML. One of the DDL is performed -# after DML to ensure that we invalidate the schema sent for test_tab so that -# the next transaction has to send the schema again. -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(2003,5000) s(i); -ALTER TABLE test_tab ADD COLUMN f INT; -COMMIT; -}); - -# A small transaction that won't get streamed. This is just to ensure that we -# send the schema again to reflect the last column added in the previous test. -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i, 4*i FROM generate_series(5001,5005) s(i); -COMMIT; -}); - -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d), count(e), count(f) FROM test_tab"); -is($result, qq(5005|5002|4005|3004|5), - 'check data was copied to subscriber for both streaming and non-streaming transactions' +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = apply)" ); +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" +) or die "Timed out while waiting for apply to restart after changing PUBLICATION"; + +test_streaming($node_publisher, $node_subscriber, $appname); $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/018_stream_subxact_abort.pl b/src/test/subscription/t/018_stream_subxact_abort.pl index 7155442e76..8ba38d40b3 100644 --- a/src/test/subscription/t/018_stream_subxact_abort.pl +++ b/src/test/subscription/t/018_stream_subxact_abort.pl @@ -8,6 +8,87 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname) = @_; + + # large (streamed) transaction with DDL, DML and ROLLBACKs + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i); + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501,1000) s(i); + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001,1500) s(i); + SAVEPOINT s3; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501,2000) s(i); + ROLLBACK TO s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001,2500) s(i); + ROLLBACK TO s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2501,3000) s(i); + SAVEPOINT s4; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3001,3500) s(i); + SAVEPOINT s5; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3501,4000) s(i); + COMMIT; + }); + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c) FROM test_tab"); + is($result, qq(2000|0), + 'check rollback to savepoint was reflected on subscriber and extra columns contain local defaults' + ); + + # large (streamed) transaction with subscriber receiving out of order + # subtransaction ROLLBACKs + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(4001,4500) s(i); + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001,5500) s(i); + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(6001,6500) s(i); + SAVEPOINT s3; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(7001,7500) s(i); + RELEASE s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(8001,8500) s(i); + ROLLBACK TO s1; + COMMIT; + }); + + $node_publisher->wait_for_catchup($appname); + + $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c) FROM test_tab"); + is($result, qq(2500|0), + 'check rollback to savepoint was reflected on subscriber'); + + # large (streamed) transaction with subscriber receiving rollback + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(8501,9000) s(i); + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(9001,9500) s(i); + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(9501,10000) s(i); + ROLLBACK; + }); + + $node_publisher->wait_for_catchup($appname); + + $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c) FROM test_tab"); + is($result, qq(2500|0), 'check rollback was reflected on subscriber'); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -36,6 +117,8 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +# Test streaming mode on $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -53,81 +136,23 @@ my $result = "SELECT count(*), count(c) FROM test_tab"); is($result, qq(2|0), 'check initial data was copied to subscriber'); -# large (streamed) transaction with DDL, DML and ROLLBACKs -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i); -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501,1000) s(i); -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001,1500) s(i); -SAVEPOINT s3; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501,2000) s(i); -ROLLBACK TO s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001,2500) s(i); -ROLLBACK TO s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2501,3000) s(i); -SAVEPOINT s4; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3001,3500) s(i); -SAVEPOINT s5; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3501,4000) s(i); -COMMIT; -}); +test_streaming($node_publisher, $node_subscriber, $appname); +# Test streaming mode apply +$node_publisher->safe_psql('postgres', "DELETE FROM test_tab WHERE (a > 2)"); $node_publisher->wait_for_catchup($appname); -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c) FROM test_tab"); -is($result, qq(2000|0), - 'check rollback to savepoint was reflected on subscriber and extra columns contain local defaults' +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" ); +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = apply)" +); +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" +) or die "Timed out while waiting for apply to restart after changing PUBLICATION"; -# large (streamed) transaction with subscriber receiving out of order -# subtransaction ROLLBACKs -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(4001,4500) s(i); -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001,5500) s(i); -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(6001,6500) s(i); -SAVEPOINT s3; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(7001,7500) s(i); -RELEASE s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(8001,8500) s(i); -ROLLBACK TO s1; -COMMIT; -}); - -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c) FROM test_tab"); -is($result, qq(2500|0), - 'check rollback to savepoint was reflected on subscriber'); - -# large (streamed) transaction with subscriber receiving rollback -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(8501,9000) s(i); -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(9001,9500) s(i); -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(9501,10000) s(i); -ROLLBACK; -}); - -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c) FROM test_tab"); -is($result, qq(2500|0), 'check rollback was reflected on subscriber'); +test_streaming($node_publisher, $node_subscriber, $appname); $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/019_stream_subxact_ddl_abort.pl b/src/test/subscription/t/019_stream_subxact_ddl_abort.pl index dbd0fca4d1..c6150d3155 100644 --- a/src/test/subscription/t/019_stream_subxact_ddl_abort.pl +++ b/src/test/subscription/t/019_stream_subxact_ddl_abort.pl @@ -9,6 +9,40 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname) = @_; + + # large (streamed) transaction with DDL, DML and ROLLBACKs + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i); + ALTER TABLE test_tab ADD COLUMN c INT; + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(501,1000) s(i); + ALTER TABLE test_tab ADD COLUMN d INT; + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001,1500) s(i); + ALTER TABLE test_tab ADD COLUMN e INT; + SAVEPOINT s3; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(1501,2000) s(i); + ALTER TABLE test_tab DROP COLUMN c; + ROLLBACK TO s1; + INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(501,1000) s(i); + COMMIT; + }); + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c) FROM test_tab"); + is($result, qq(1000|500), + 'check rollback to savepoint was reflected on subscriber and extra columns contain local defaults' + ); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -37,6 +71,8 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +# Test streaming mode on $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -54,34 +90,27 @@ my $result = "SELECT count(*), count(c) FROM test_tab"); is($result, qq(2|0), 'check initial data was copied to subscriber'); -# large (streamed) transaction with DDL, DML and ROLLBACKs +test_streaming($node_publisher, $node_subscriber, $appname); + +# Test streaming mode apply $node_publisher->safe_psql( 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i); -ALTER TABLE test_tab ADD COLUMN c INT; -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(501,1000) s(i); -ALTER TABLE test_tab ADD COLUMN d INT; -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001,1500) s(i); -ALTER TABLE test_tab ADD COLUMN e INT; -SAVEPOINT s3; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(1501,2000) s(i); +DELETE FROM test_tab WHERE (a > 2); ALTER TABLE test_tab DROP COLUMN c; -ROLLBACK TO s1; -INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(501,1000) s(i); -COMMIT; }); - $node_publisher->wait_for_catchup($appname); -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c) FROM test_tab"); -is($result, qq(1000|500), - 'check rollback to savepoint was reflected on subscriber and extra columns contain local defaults' +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" +); +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = apply)" ); +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" +) or die "Timed out while waiting for apply to restart after changing PUBLICATION"; + +test_streaming($node_publisher, $node_subscriber, $appname); $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/022_twophase_cascade.pl b/src/test/subscription/t/022_twophase_cascade.pl index 900c25d5ce..bb20a3640c 100644 --- a/src/test/subscription/t/022_twophase_cascade.pl +++ b/src/test/subscription/t/022_twophase_cascade.pl @@ -11,6 +11,148 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# --------------------- +# 2PC + STREAMING TESTS +# --------------------- +sub test_streaming +{ + my ($node_A, $node_B, $node_C, $appname_B, $appname_C, $streaming) = @_; + + my $oldpid_B = $node_A->safe_psql('postgres', " + SELECT pid FROM pg_stat_replication + WHERE application_name = '$appname_B' AND state = 'streaming';"); + my $oldpid_C = $node_B->safe_psql('postgres', " + SELECT pid FROM pg_stat_replication + WHERE application_name = '$appname_C' AND state = 'streaming';"); + + # Setup logical replication streaming mode + + $node_B->safe_psql('postgres', " + ALTER SUBSCRIPTION tap_sub_B + SET (streaming = $streaming);"); + $node_C->safe_psql('postgres', " + ALTER SUBSCRIPTION tap_sub_C + SET (streaming = $streaming)"); + + # Wait for subscribers to finish initialization + + $node_A->poll_query_until('postgres', " + SELECT pid != $oldpid_B FROM pg_stat_replication + WHERE application_name = '$appname_B' AND state = 'streaming';" + ) or die "Timed out while waiting for apply to restart"; + $node_B->poll_query_until('postgres', " + SELECT pid != $oldpid_C FROM pg_stat_replication + WHERE application_name = '$appname_C' AND state = 'streaming';" + ) or die "Timed out while waiting for apply to restart"; + + ############################### + # Test 2PC PREPARE / COMMIT PREPARED. + # 1. Data is streamed as a 2PC transaction. + # 2. Then do commit prepared. + # + # Expect all data is replicated on subscriber(s) after the commit. + ############################### + + # First, delete the data except for 2 rows (delete will be replicated) + $node_A->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); + + # Insert, update and delete enough rows to exceed the 64kB limit. + # Then 2PC PREPARE + $node_A->safe_psql('postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); + + # check the transaction state is prepared on subscriber(s) + my $result = $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber B'); + $result = $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber C'); + + # 2PC COMMIT + $node_A->safe_psql('postgres', "COMMIT PREPARED 'test_prepared_tab';"); + + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); + + # check that transaction was committed on subscriber(s) + $result = $node_B->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), 'Rows inserted by 2PC have committed on subscriber B, and extra columns have local defaults'); + $result = $node_C->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), 'Rows inserted by 2PC have committed on subscriber C, and extra columns have local defaults'); + + # check the transaction state is ended on subscriber(s) + $result = $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is committed on subscriber B'); + $result = $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is committed on subscriber C'); + + ############################### + # Test 2PC PREPARE with a nested ROLLBACK TO SAVEPOINT. + # 0. Cleanup from previous test leaving only 2 rows. + # 1. Insert one more row. + # 2. Record a SAVEPOINT. + # 3. Data is streamed using 2PC. + # 4. Do rollback to SAVEPOINT prior to the streamed inserts. + # 5. Then COMMIT PREPARED. + # + # Expect data after the SAVEPOINT is aborted leaving only 3 rows (= 2 original + 1 from step 1). + ############################### + + # First, delete the data except for 2 rows (delete will be replicated) + $node_A->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); + + # 2PC PREPARE with a nested ROLLBACK TO SAVEPOINT + $node_A->safe_psql('postgres', " + BEGIN; + INSERT INTO test_tab VALUES (9999, 'foobar'); + SAVEPOINT sp_inner; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + ROLLBACK TO SAVEPOINT sp_inner; + PREPARE TRANSACTION 'outer'; + "); + + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); + + # check the transaction state prepared on subscriber(s) + $result = $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber B'); + $result = $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber C'); + + # 2PC COMMIT + $node_A->safe_psql('postgres', "COMMIT PREPARED 'outer';"); + + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); + + # check the transaction state is ended on subscriber + $result = $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is ended on subscriber B'); + $result = $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is ended on subscriber C'); + + # check inserts are visible at subscriber(s). + # All the streamed data (prior to the SAVEPOINT) should be rolled back. + # (9999, 'foobar') should be committed. + $result = $node_B->safe_psql('postgres', "SELECT count(*) FROM test_tab where b = 'foobar';"); + is($result, qq(1), 'Rows committed are present on subscriber B'); + $result = $node_B->safe_psql('postgres', "SELECT count(*) FROM test_tab;"); + is($result, qq(3), 'Rows committed are present on subscriber B'); + $result = $node_C->safe_psql('postgres', "SELECT count(*) FROM test_tab where b = 'foobar';"); + is($result, qq(1), 'Rows committed are present on subscriber C'); + $result = $node_C->safe_psql('postgres', "SELECT count(*) FROM test_tab;"); + is($result, qq(3), 'Rows committed are present on subscriber C'); +} + ############################### # Setup a cascade of pub/sub nodes. # node_A -> node_B -> node_C @@ -229,140 +371,8 @@ is($result, qq(21), 'Rows committed are present on subscriber B'); $result = $node_C->safe_psql('postgres', "SELECT a FROM tab_full where a IN (21,22);"); is($result, qq(21), 'Rows committed are present on subscriber C'); -# --------------------- -# 2PC + STREAMING TESTS -# --------------------- - -my $oldpid_B = $node_A->safe_psql('postgres', " - SELECT pid FROM pg_stat_replication - WHERE application_name = '$appname_B' AND state = 'streaming';"); -my $oldpid_C = $node_B->safe_psql('postgres', " - SELECT pid FROM pg_stat_replication - WHERE application_name = '$appname_C' AND state = 'streaming';"); - -# Setup logical replication (streaming = on) - -$node_B->safe_psql('postgres', " - ALTER SUBSCRIPTION tap_sub_B - SET (streaming = on);"); -$node_C->safe_psql('postgres', " - ALTER SUBSCRIPTION tap_sub_C - SET (streaming = on)"); - -# Wait for subscribers to finish initialization - -$node_A->poll_query_until('postgres', " - SELECT pid != $oldpid_B FROM pg_stat_replication - WHERE application_name = '$appname_B' AND state = 'streaming';" -) or die "Timed out while waiting for apply to restart"; -$node_B->poll_query_until('postgres', " - SELECT pid != $oldpid_C FROM pg_stat_replication - WHERE application_name = '$appname_C' AND state = 'streaming';" -) or die "Timed out while waiting for apply to restart"; - -############################### -# Test 2PC PREPARE / COMMIT PREPARED. -# 1. Data is streamed as a 2PC transaction. -# 2. Then do commit prepared. -# -# Expect all data is replicated on subscriber(s) after the commit. -############################### - -# Insert, update and delete enough rows to exceed the 64kB limit. -# Then 2PC PREPARE -$node_A->safe_psql('postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_A->wait_for_catchup($appname_B); -$node_B->wait_for_catchup($appname_C); - -# check the transaction state is prepared on subscriber(s) -$result = $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber B'); -$result = $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber C'); - -# 2PC COMMIT -$node_A->safe_psql('postgres', "COMMIT PREPARED 'test_prepared_tab';"); - -$node_A->wait_for_catchup($appname_B); -$node_B->wait_for_catchup($appname_C); - -# check that transaction was committed on subscriber(s) -$result = $node_B->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), 'Rows inserted by 2PC have committed on subscriber B, and extra columns have local defaults'); -$result = $node_C->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), 'Rows inserted by 2PC have committed on subscriber C, and extra columns have local defaults'); - -# check the transaction state is ended on subscriber(s) -$result = $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is committed on subscriber B'); -$result = $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is committed on subscriber C'); - -############################### -# Test 2PC PREPARE with a nested ROLLBACK TO SAVEPOINT. -# 0. Cleanup from previous test leaving only 2 rows. -# 1. Insert one more row. -# 2. Record a SAVEPOINT. -# 3. Data is streamed using 2PC. -# 4. Do rollback to SAVEPOINT prior to the streamed inserts. -# 5. Then COMMIT PREPARED. -# -# Expect data after the SAVEPOINT is aborted leaving only 3 rows (= 2 original + 1 from step 1). -############################### - -# First, delete the data except for 2 rows (delete will be replicated) -$node_A->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); - -# 2PC PREPARE with a nested ROLLBACK TO SAVEPOINT -$node_A->safe_psql('postgres', " - BEGIN; - INSERT INTO test_tab VALUES (9999, 'foobar'); - SAVEPOINT sp_inner; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - ROLLBACK TO SAVEPOINT sp_inner; - PREPARE TRANSACTION 'outer'; - "); - -$node_A->wait_for_catchup($appname_B); -$node_B->wait_for_catchup($appname_C); - -# check the transaction state prepared on subscriber(s) -$result = $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber B'); -$result = $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber C'); - -# 2PC COMMIT -$node_A->safe_psql('postgres', "COMMIT PREPARED 'outer';"); - -$node_A->wait_for_catchup($appname_B); -$node_B->wait_for_catchup($appname_C); - -# check the transaction state is ended on subscriber -$result = $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is ended on subscriber B'); -$result = $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is ended on subscriber C'); - -# check inserts are visible at subscriber(s). -# All the streamed data (prior to the SAVEPOINT) should be rolled back. -# (9999, 'foobar') should be committed. -$result = $node_B->safe_psql('postgres', "SELECT count(*) FROM test_tab where b = 'foobar';"); -is($result, qq(1), 'Rows committed are present on subscriber B'); -$result = $node_B->safe_psql('postgres', "SELECT count(*) FROM test_tab;"); -is($result, qq(3), 'Rows committed are present on subscriber B'); -$result = $node_C->safe_psql('postgres', "SELECT count(*) FROM test_tab where b = 'foobar';"); -is($result, qq(1), 'Rows committed are present on subscriber C'); -$result = $node_C->safe_psql('postgres', "SELECT count(*) FROM test_tab;"); -is($result, qq(3), 'Rows committed are present on subscriber C'); +test_streaming($node_A, $node_B, $node_C, $appname_B, $appname_C, 'on'); +test_streaming($node_A, $node_B, $node_C, $appname_B, $appname_C, 'apply'); ############################### # check all the cleanup diff --git a/src/test/subscription/t/023_twophase_stream.pl b/src/test/subscription/t/023_twophase_stream.pl index 93ce3ef132..f5c4016480 100644 --- a/src/test/subscription/t/023_twophase_stream.pl +++ b/src/test/subscription/t/023_twophase_stream.pl @@ -8,6 +8,203 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname) = @_; + ############################### + # Test 2PC PREPARE / COMMIT PREPARED. + # 1. Data is streamed as a 2PC transaction. + # 2. Then do commit prepared. + # + # Expect all data is replicated on subscriber side after the commit. + ############################### + + # check that 2PC gets replicated to subscriber + # Insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql('postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is in prepared state on subscriber + my $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber'); + + # 2PC transaction gets committed + $node_publisher->safe_psql('postgres', "COMMIT PREPARED 'test_prepared_tab';"); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is committed on subscriber + $result = $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab") ; + is($result, qq(3334|3334|3334), 'Rows inserted by 2PC have committed on subscriber, and extra columns contain local defaults'); + $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is committed on subscriber'); + + ############################### + # Test 2PC PREPARE / ROLLBACK PREPARED. + # 1. Table is deleted back to 2 rows which are replicated on subscriber. + # 2. Data is streamed using 2PC. + # 3. Do rollback prepared. + # + # Expect data rolls back leaving only the original 2 rows. + ############################### + + # First, delete the data except for 2 rows (will be replicated) + $node_publisher->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); + + # Then insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql('postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is in prepared state on subscriber + $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber'); + + # 2PC transaction gets aborted + $node_publisher->safe_psql('postgres', "ROLLBACK PREPARED 'test_prepared_tab';"); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is aborted on subscriber + $result = $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab") ; + is($result, qq(2|2|2), 'Rows inserted by 2PC are rolled back, leaving only the original 2 rows'); + + $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is aborted on subscriber'); + + ############################### + # Check that 2PC COMMIT PREPARED is decoded properly on crash restart. + # 1. insert, update and delete enough rows to exceed the 64kB limit. + # 2. Then server crashes before the 2PC transaction is committed. + # 3. After servers are restarted the pending transaction is committed. + # + # Expect all data is replicated on subscriber side after the commit. + # Note: both publisher and subscriber do crash/restart. + ############################### + + $node_publisher->safe_psql('postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + $node_subscriber->stop('immediate'); + $node_publisher->stop('immediate'); + + $node_publisher->start; + $node_subscriber->start; + + # commit post the restart + $node_publisher->safe_psql('postgres', "COMMIT PREPARED 'test_prepared_tab';"); + $node_publisher->wait_for_catchup($appname); + + # check inserts are visible + $result = $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab") ; + is($result, qq(3334|3334|3334), 'Rows inserted by 2PC have committed on subscriber, and extra columns contain local defaults'); + + ############################### + # Do INSERT after the PREPARE but before ROLLBACK PREPARED. + # 1. Table is deleted back to 2 rows which are replicated on subscriber. + # 2. Data is streamed using 2PC. + # 3. A single row INSERT is done which is after the PREPARE. + # 4. Then do a ROLLBACK PREPARED. + # + # Expect the 2PC data rolls back leaving only 3 rows on the subscriber + # (the original 2 + inserted 1). + ############################### + + # First, delete the data except for 2 rows (will be replicated) + $node_publisher->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); + + # Then insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql('postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is in prepared state on subscriber + $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber'); + + # Insert a different record (now we are outside of the 2PC transaction) + # Note: the 2PC transaction still holds row locks so make sure this insert is for a separate primary key + $node_publisher->safe_psql('postgres', "INSERT INTO test_tab VALUES (99999, 'foobar')"); + + # 2PC transaction gets aborted + $node_publisher->safe_psql('postgres', "ROLLBACK PREPARED 'test_prepared_tab';"); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is aborted on subscriber, + # but the extra INSERT outside of the 2PC still was replicated + $result = $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab") ; + is($result, qq(3|3|3), 'check the outside insert was copied to subscriber'); + + $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is aborted on subscriber'); + + ############################### + # Do INSERT after the PREPARE but before COMMIT PREPARED. + # 1. Table is deleted back to 2 rows which are replicated on subscriber. + # 2. Data is streamed using 2PC. + # 3. A single row INSERT is done which is after the PREPARE. + # 4. Then do a COMMIT PREPARED. + # + # Expect 2PC data + the extra row are on the subscriber + # (the 3334 + inserted 1 = 3335). + ############################### + + # First, delete the data except for 2 rows (will be replicated) + $node_publisher->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); + + # Then insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql('postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is in prepared state on subscriber + $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber'); + + # Insert a different record (now we are outside of the 2PC transaction) + # Note: the 2PC transaction still holds row locks so make sure this insert is for a separare primary key + $node_publisher->safe_psql('postgres', "INSERT INTO test_tab VALUES (99999, 'foobar')"); + + # 2PC transaction gets committed + $node_publisher->safe_psql('postgres', "COMMIT PREPARED 'test_prepared_tab';"); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is committed on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3335|3335|3335), 'Rows inserted by 2PC (as well as outside insert) have committed on subscriber, and extra columns contain local defaults'); + + $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is committed on subscriber'); +} + ############################### # Setup ############################### @@ -42,6 +239,8 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres'; $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +# Test streaming mode on $node_subscriber->safe_psql('postgres', " CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' @@ -69,198 +268,23 @@ $node_subscriber->poll_query_until('postgres', $twophase_query) my $result = $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab"); is($result, qq(2|2|2), 'check initial data was copied to subscriber'); -############################### -# Test 2PC PREPARE / COMMIT PREPARED. -# 1. Data is streamed as a 2PC transaction. -# 2. Then do commit prepared. -# -# Expect all data is replicated on subscriber side after the commit. -############################### - -# check that 2PC gets replicated to subscriber -# Insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql('postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is in prepared state on subscriber -$result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber'); +test_streaming($node_publisher, $node_subscriber, $appname); -# 2PC transaction gets committed -$node_publisher->safe_psql('postgres', "COMMIT PREPARED 'test_prepared_tab';"); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is committed on subscriber -$result = $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), 'Rows inserted by 2PC have committed on subscriber, and extra columns contain local defaults'); -$result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is committed on subscriber'); - -############################### -# Test 2PC PREPARE / ROLLBACK PREPARED. -# 1. Table is deleted back to 2 rows which are replicated on subscriber. -# 2. Data is streamed using 2PC. -# 3. Do rollback prepared. -# -# Expect data rolls back leaving only the original 2 rows. -############################### - -# First, delete the data except for 2 rows (will be replicated) +# Test streaming mode apply $node_publisher->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); - -# Then insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql('postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is in prepared state on subscriber -$result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber'); - -# 2PC transaction gets aborted -$node_publisher->safe_psql('postgres', "ROLLBACK PREPARED 'test_prepared_tab';"); - $node_publisher->wait_for_catchup($appname); -# check that transaction is aborted on subscriber -$result = $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(2|2|2), 'Rows inserted by 2PC are rolled back, leaving only the original 2 rows'); - -$result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is aborted on subscriber'); - -############################### -# Check that 2PC COMMIT PREPARED is decoded properly on crash restart. -# 1. insert, update and delete enough rows to exceed the 64kB limit. -# 2. Then server crashes before the 2PC transaction is committed. -# 3. After servers are restarted the pending transaction is committed. -# -# Expect all data is replicated on subscriber side after the commit. -# Note: both publisher and subscriber do crash/restart. -############################### - -$node_publisher->safe_psql('postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_subscriber->stop('immediate'); -$node_publisher->stop('immediate'); - -$node_publisher->start; -$node_subscriber->start; - -# commit post the restart -$node_publisher->safe_psql('postgres', "COMMIT PREPARED 'test_prepared_tab';"); -$node_publisher->wait_for_catchup($appname); - -# check inserts are visible -$result = $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), 'Rows inserted by 2PC have committed on subscriber, and extra columns contain local defaults'); - -############################### -# Do INSERT after the PREPARE but before ROLLBACK PREPARED. -# 1. Table is deleted back to 2 rows which are replicated on subscriber. -# 2. Data is streamed using 2PC. -# 3. A single row INSERT is done which is after the PREPARE. -# 4. Then do a ROLLBACK PREPARED. -# -# Expect the 2PC data rolls back leaving only 3 rows on the subscriber -# (the original 2 + inserted 1). -############################### - -# First, delete the data except for 2 rows (will be replicated) -$node_publisher->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); - -# Then insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql('postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is in prepared state on subscriber -$result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber'); - -# Insert a different record (now we are outside of the 2PC transaction) -# Note: the 2PC transaction still holds row locks so make sure this insert is for a separate primary key -$node_publisher->safe_psql('postgres', "INSERT INTO test_tab VALUES (99999, 'foobar')"); - -# 2PC transaction gets aborted -$node_publisher->safe_psql('postgres', "ROLLBACK PREPARED 'test_prepared_tab';"); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is aborted on subscriber, -# but the extra INSERT outside of the 2PC still was replicated -$result = $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3|3|3), 'check the outside insert was copied to subscriber'); - -$result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is aborted on subscriber'); - -############################### -# Do INSERT after the PREPARE but before COMMIT PREPARED. -# 1. Table is deleted back to 2 rows which are replicated on subscriber. -# 2. Data is streamed using 2PC. -# 3. A single row INSERT is done which is after the PREPARE. -# 4. Then do a COMMIT PREPARED. -# -# Expect 2PC data + the extra row are on the subscriber -# (the 3334 + inserted 1 = 3335). -############################### - -# First, delete the data except for 2 rows (will be replicated) -$node_publisher->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); - -# Then insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql('postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is in prepared state on subscriber -$result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber'); - -# Insert a different record (now we are outside of the 2PC transaction) -# Note: the 2PC transaction still holds row locks so make sure this insert is for a separare primary key -$node_publisher->safe_psql('postgres', "INSERT INTO test_tab VALUES (99999, 'foobar')"); - -# 2PC transaction gets committed -$node_publisher->safe_psql('postgres', "COMMIT PREPARED 'test_prepared_tab';"); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is committed on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3335|3335|3335), 'Rows inserted by 2PC (as well as outside insert) have committed on subscriber, and extra columns contain local defaults'); +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" +); +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = apply)" +); +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" +) or die "Timed out while waiting for apply to restart after changing PUBLICATION"; -$result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is committed on subscriber'); +test_streaming($node_publisher, $node_subscriber, $appname); ############################### # check all the cleanup -- 2.18.4 [application/octet-stream] v5-0001-Perform-streaming-logical-transactions-by-background.patch (81.8K, ../../OSZPR01MB6310F0FABB05F8E5BB31A5B0FDFC9@OSZPR01MB6310.jpnprd01.prod.outlook.com/3-v5-0001-Perform-streaming-logical-transactions-by-background.patch) download | inline diff: From ec33a270eb80330b0103d6051ae9bea0b2d6f7fd Mon Sep 17 00:00:00 2001 From: "houzj.fnst" <[email protected]> Date: Wed, 20 Apr 2022 16:45:07 +0800 Subject: [PATCH] Perform streaming logical transactions by background workers Currently, for large transactions, the publisher sends the data in multiple streams (changes divided into chunks depending upon logical_decoding_work_mem), and then on the subscriber-side, the apply worker writes the changes into temporary files and once it receives the commit, it reads from the file and applies the entire transaction. To improve the performance of such transactions, we can instead allow them to be applied via background workers. In this approach, we assign a new apply background (if available) as soon as the xact's first stream is received and the main apply worker will send changes to this new worker via shared memory. The apply background will directly apply the change instead of writing it to temporary files. We keep this worker assigned till the transaction commit is received and also wait for the worker to finish at commit. This preserves commit ordering and avoids writing to and reading from file in most cases. We still need to spill if there is no worker available. We also need to allow stream_stop to complete by the apply background to finish it to avoid deadlocks because T-1's current stream of changes can update rows in conflicting order with T-2's next stream of changes. This patch also extends the subscription streaming option so that user can control whether apply the streaming transaction in a apply background or spill the change to disk. User can set the streaming option to 'on/off', 'apply'. For now, 'apply' means the streaming will be applied via a apply background if available. 'on' means the streaming transaction will be spilled to disk. --- doc/src/sgml/catalogs.sgml | 10 +- doc/src/sgml/ref/create_subscription.sgml | 24 +- src/backend/commands/subscriptioncmds.c | 66 +- src/backend/postmaster/bgworker.c | 3 + src/backend/replication/logical/launcher.c | 84 +- src/backend/replication/logical/origin.c | 10 +- src/backend/replication/logical/proto.c | 7 +- src/backend/replication/logical/tablesync.c | 10 +- src/backend/replication/logical/worker.c | 1454 ++++++++++++++++++++++++--- src/backend/utils/activity/wait_event.c | 3 + src/bin/pg_dump/pg_dump.c | 6 +- src/include/catalog/pg_subscription.h | 16 +- src/include/replication/logicalproto.h | 4 +- src/include/replication/logicalworker.h | 1 + src/include/replication/origin.h | 2 +- src/include/replication/worker_internal.h | 7 +- src/include/utils/wait_event.h | 1 + src/test/regress/expected/subscription.out | 8 +- src/tools/pgindent/typedefs.list | 2 + 19 files changed, 1506 insertions(+), 212 deletions(-) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index a533a21..ed61f99 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -7863,11 +7863,15 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l <row> <entry role="catalog_table_entry"><para role="column_definition"> - <structfield>substream</structfield> <type>bool</type> + <structfield>substream</structfield> <type>char</type> </para> <para> - If true, the subscription will allow streaming of in-progress - transactions + Controls how to handle the streaming of in-progress transactions. + <literal>f</literal> = disallow streaming of in-progress transactions + <literal>o</literal> = spill the changes of in-progress transactions to + disk and apply at once after the transaction is committed on the + publisher. + <literal>a</literal> = apply changes directly using a background worker </para></entry> </row> diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml index 203bb41..8441aa0 100644 --- a/doc/src/sgml/ref/create_subscription.sgml +++ b/doc/src/sgml/ref/create_subscription.sgml @@ -217,13 +217,29 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl </varlistentry> <varlistentry> - <term><literal>streaming</literal> (<type>boolean</type>)</term> + <term><literal>streaming</literal> (<type>enum</type>)</term> <listitem> <para> Specifies whether to enable streaming of in-progress transactions - for this subscription. By default, all transactions - are fully decoded on the publisher and only then sent to the - subscriber as a whole. + for this subscription. The default value is <literal>off</literal>, + all transactions are fully decoded on the publisher and only then + sent to the subscriber as a whole. + </para> + + <para> + If set to <literal>on</literal>, the changes of transaction are + written to temporary files and then applied at once after the + transaction is committed on the publisher. + </para> + + <para> + If set to <literal>apply</literal> incoming + changes are directly applied via one of the background worker, if + available. If no background worker is free to handle streaming + transaction then the changes are written to a file and applied after + the transaction is committed. Note that if error happen when applying + changes in background worker, it might not report the finish LSN of + the remote transaction in server log. </para> </listitem> </varlistentry> diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c index b94236f..fe1d373 100644 --- a/src/backend/commands/subscriptioncmds.c +++ b/src/backend/commands/subscriptioncmds.c @@ -83,7 +83,7 @@ typedef struct SubOpts bool copy_data; bool refresh; bool binary; - bool streaming; + char streaming; bool twophase; bool disableonerr; XLogRecPtr lsn; @@ -96,6 +96,62 @@ static void ReportSlotConnectionError(List *rstates, Oid subid, char *slotname, /* + * Extract the streaming mode value from a DefElem. This is like + * defGetBoolean() but also accepts the special value and "apply". + */ +static char +defGetStreamingMode(DefElem *def) +{ + /* + * If no parameter given, assume "true" is meant. + */ + if (def->arg == NULL) + return SUBSTREAM_ON; + + /* + * Allow 0, 1, "true", "false", "on", "off" or "apply". + */ + switch (nodeTag(def->arg)) + { + case T_Integer: + switch (intVal(def->arg)) + { + case 0: + return SUBSTREAM_OFF; + case 1: + return SUBSTREAM_ON; + default: + /* otherwise, error out below */ + break; + } + break; + default: + { + char *sval = defGetString(def); + + /* + * The set of strings accepted here should match up with the + * grammar's opt_boolean_or_string production. + */ + if (pg_strcasecmp(sval, "true") == 0 || + pg_strcasecmp(sval, "on") == 0) + return SUBSTREAM_ON; + if (pg_strcasecmp(sval, "apply") == 0) + return SUBSTREAM_APPLY; + if (pg_strcasecmp(sval, "false") == 0 || + pg_strcasecmp(sval, "off") == 0) + return SUBSTREAM_OFF; + } + break; + } + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s requires a Boolean value or \"apply\"", + def->defname))); + return SUBSTREAM_OFF; /* keep compiler quiet */ +} + +/* * Common option parsing function for CREATE and ALTER SUBSCRIPTION commands. * * Since not all options can be specified in both commands, this function @@ -132,7 +188,7 @@ parse_subscription_options(ParseState *pstate, List *stmt_options, if (IsSet(supported_opts, SUBOPT_BINARY)) opts->binary = false; if (IsSet(supported_opts, SUBOPT_STREAMING)) - opts->streaming = false; + opts->streaming = SUBSTREAM_OFF; if (IsSet(supported_opts, SUBOPT_TWOPHASE_COMMIT)) opts->twophase = false; if (IsSet(supported_opts, SUBOPT_DISABLE_ON_ERR)) @@ -233,7 +289,7 @@ parse_subscription_options(ParseState *pstate, List *stmt_options, errorConflictingDefElem(defel, pstate); opts->specified_opts |= SUBOPT_STREAMING; - opts->streaming = defGetBoolean(defel); + opts->streaming = defGetStreamingMode(defel); } else if (strcmp(defel->defname, "two_phase") == 0) { @@ -601,7 +657,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt, values[Anum_pg_subscription_subowner - 1] = ObjectIdGetDatum(owner); values[Anum_pg_subscription_subenabled - 1] = BoolGetDatum(opts.enabled); values[Anum_pg_subscription_subbinary - 1] = BoolGetDatum(opts.binary); - values[Anum_pg_subscription_substream - 1] = BoolGetDatum(opts.streaming); + values[Anum_pg_subscription_substream - 1] = CharGetDatum(opts.streaming); values[Anum_pg_subscription_subtwophasestate - 1] = CharGetDatum(opts.twophase ? LOGICALREP_TWOPHASE_STATE_PENDING : @@ -1060,7 +1116,7 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt, if (IsSet(opts.specified_opts, SUBOPT_STREAMING)) { values[Anum_pg_subscription_substream - 1] = - BoolGetDatum(opts.streaming); + CharGetDatum(opts.streaming); replaces[Anum_pg_subscription_substream - 1] = true; } diff --git a/src/backend/postmaster/bgworker.c b/src/backend/postmaster/bgworker.c index 30682b6..070d778 100644 --- a/src/backend/postmaster/bgworker.c +++ b/src/backend/postmaster/bgworker.c @@ -128,6 +128,9 @@ static const struct }, { "ApplyWorkerMain", ApplyWorkerMain + }, + { + "ApplyBgworkerMain", ApplyBgworkerMain } }; diff --git a/src/backend/replication/logical/launcher.c b/src/backend/replication/logical/launcher.c index 0adb2d1..299634c 100644 --- a/src/backend/replication/logical/launcher.c +++ b/src/backend/replication/logical/launcher.c @@ -72,6 +72,7 @@ static void logicalrep_launcher_onexit(int code, Datum arg); static void logicalrep_worker_onexit(int code, Datum arg); static void logicalrep_worker_detach(void); static void logicalrep_worker_cleanup(LogicalRepWorker *worker); +static void logicalrep_worker_stop_internal(LogicalRepWorker *worker); static bool on_commit_launcher_wakeup = false; @@ -224,6 +225,10 @@ logicalrep_worker_find(Oid subid, Oid relid, bool only_running) { LogicalRepWorker *w = &LogicalRepCtx->workers[i]; + /* We only need main apply worker or table sync worker here */ + if (w->subworker) + continue; + if (w->in_use && w->subid == subid && w->relid == relid && (!only_running || w->proc)) { @@ -262,9 +267,9 @@ logicalrep_workers_find(Oid subid, bool only_running) /* * Start new apply background worker, if possible. */ -void +bool logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, Oid userid, - Oid relid) + Oid relid, dsm_handle subworker_dsm) { BackgroundWorker bgw; BackgroundWorkerHandle *bgw_handle; @@ -275,6 +280,9 @@ logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, Oid userid, int nsyncworkers; TimestampTz now; + /* We don't support table sync in subworker */ + Assert(!((subworker_dsm != DSM_HANDLE_INVALID) && OidIsValid(relid))); + ereport(DEBUG1, (errmsg_internal("starting logical replication worker for subscription \"%s\"", subname))); @@ -351,7 +359,7 @@ retry: if (OidIsValid(relid) && nsyncworkers >= max_sync_workers_per_subscription) { LWLockRelease(LogicalRepWorkerLock); - return; + return false; } /* @@ -365,7 +373,7 @@ retry: (errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED), errmsg("out of logical replication worker slots"), errhint("You might need to increase max_logical_replication_workers."))); - return; + return false; } /* Prepare the worker slot. */ @@ -380,6 +388,7 @@ retry: worker->relstate = SUBREL_STATE_UNKNOWN; worker->relstate_lsn = InvalidXLogRecPtr; worker->stream_fileset = NULL; + worker->subworker = (subworker_dsm != DSM_HANDLE_INVALID); worker->last_lsn = InvalidXLogRecPtr; TIMESTAMP_NOBEGIN(worker->last_send_time); TIMESTAMP_NOBEGIN(worker->last_recv_time); @@ -397,19 +406,31 @@ retry: BGWORKER_BACKEND_DATABASE_CONNECTION; bgw.bgw_start_time = BgWorkerStart_RecoveryFinished; snprintf(bgw.bgw_library_name, BGW_MAXLEN, "postgres"); - snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyWorkerMain"); + + if (subworker_dsm == DSM_HANDLE_INVALID) + snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyWorkerMain"); + else + snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyBgworkerMain"); + + if (OidIsValid(relid)) snprintf(bgw.bgw_name, BGW_MAXLEN, "logical replication worker for subscription %u sync %u", subid, relid); - else + else if (subworker_dsm == DSM_HANDLE_INVALID) snprintf(bgw.bgw_name, BGW_MAXLEN, "logical replication worker for subscription %u", subid); + else + snprintf(bgw.bgw_name, BGW_MAXLEN, + "logical replication apply worker for subscription %u", subid); snprintf(bgw.bgw_type, BGW_MAXLEN, "logical replication worker"); bgw.bgw_restart_time = BGW_NEVER_RESTART; bgw.bgw_notify_pid = MyProcPid; bgw.bgw_main_arg = Int32GetDatum(slot); + if (subworker_dsm != DSM_HANDLE_INVALID) + memcpy(bgw.bgw_extra, &subworker_dsm, sizeof(dsm_handle)); + if (!RegisterDynamicBackgroundWorker(&bgw, &bgw_handle)) { /* Failed to start worker, so clean up the worker slot. */ @@ -422,11 +443,13 @@ retry: (errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED), errmsg("out of background worker slots"), errhint("You might need to increase max_worker_processes."))); - return; + return false; } /* Now wait until it attaches. */ WaitForReplicationWorkerAttach(worker, generation, bgw_handle); + + return true; } /* @@ -437,7 +460,6 @@ void logicalrep_worker_stop(Oid subid, Oid relid) { LogicalRepWorker *worker; - uint16 generation; LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); @@ -450,6 +472,22 @@ logicalrep_worker_stop(Oid subid, Oid relid) return; } + logicalrep_worker_stop_internal(worker); + + LWLockRelease(LogicalRepWorkerLock); +} + +/* + * Workhorse for logicalrep_worker_stop() and logicalrep_worker_detach(). Stop + * the worker and wait for wait for it to die. + */ +static void +logicalrep_worker_stop_internal(LogicalRepWorker *worker) +{ + uint16 generation; + + Assert(LWLockHeldByMe(LogicalRepWorkerLock)); + /* * Remember which generation was our worker so we can check if what we see * is still the same one. @@ -486,10 +524,7 @@ logicalrep_worker_stop(Oid subid, Oid relid) * different, meaning that a different worker has taken the slot. */ if (!worker->in_use || worker->generation != generation) - { - LWLockRelease(LogicalRepWorkerLock); return; - } /* Worker has assigned proc, so it has started. */ if (worker->proc) @@ -523,8 +558,6 @@ logicalrep_worker_stop(Oid subid, Oid relid) LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); } - - LWLockRelease(LogicalRepWorkerLock); } /* @@ -600,6 +633,28 @@ logicalrep_worker_attach(int slot) static void logicalrep_worker_detach(void) { + /* + * If we are the main apply worker, stop all the apply background workers + * we started before. + */ + if (!MyLogicalRepWorker->subworker) + { + List *workers; + ListCell *lc; + + LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); + + workers = logicalrep_workers_find(MyLogicalRepWorker->subid, true); + foreach(lc, workers) + { + LogicalRepWorker *w = (LogicalRepWorker *) lfirst(lc); + if (w->subworker) + logicalrep_worker_stop_internal(w); + } + + LWLockRelease(LogicalRepWorkerLock); + } + /* Block concurrent access. */ LWLockAcquire(LogicalRepWorkerLock, LW_EXCLUSIVE); @@ -622,6 +677,7 @@ logicalrep_worker_cleanup(LogicalRepWorker *worker) worker->userid = InvalidOid; worker->subid = InvalidOid; worker->relid = InvalidOid; + worker->subworker = false; } /* @@ -869,7 +925,7 @@ ApplyLauncherMain(Datum main_arg) wait_time = wal_retrieve_retry_interval; logicalrep_worker_launch(sub->dbid, sub->oid, sub->name, - sub->owner, InvalidOid); + sub->owner, InvalidOid, DSM_HANDLE_INVALID); } } diff --git a/src/backend/replication/logical/origin.c b/src/backend/replication/logical/origin.c index b0c8b6c..c6ea68a 100644 --- a/src/backend/replication/logical/origin.c +++ b/src/backend/replication/logical/origin.c @@ -1068,7 +1068,7 @@ ReplicationOriginExitCleanup(int code, Datum arg) * with replorigin_session_reset(). */ void -replorigin_session_setup(RepOriginId node) +replorigin_session_setup(RepOriginId node, bool acquire) { static bool registered_cleanup; int i; @@ -1110,7 +1110,11 @@ replorigin_session_setup(RepOriginId node) if (curstate->roident != node) continue; - else if (curstate->acquired_by != 0) + /* + * We allow the apply worker to get the slot which is acquired by its + * leader process. + */ + else if (curstate->acquired_by != 0 && acquire) { ereport(ERROR, (errcode(ERRCODE_OBJECT_IN_USE), @@ -1321,7 +1325,7 @@ pg_replication_origin_session_setup(PG_FUNCTION_ARGS) name = text_to_cstring((text *) DatumGetPointer(PG_GETARG_DATUM(0))); origin = replorigin_by_name(name, false); - replorigin_session_setup(origin); + replorigin_session_setup(origin, true); replorigin_session_origin = origin; diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c index ff8513e..0409fde 100644 --- a/src/backend/replication/logical/proto.c +++ b/src/backend/replication/logical/proto.c @@ -1138,14 +1138,11 @@ logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn, /* * Read STREAM COMMIT from the output stream. */ -TransactionId +void logicalrep_read_stream_commit(StringInfo in, LogicalRepCommitData *commit_data) { - TransactionId xid; uint8 flags; - xid = pq_getmsgint(in, 4); - /* read flags (unused for now) */ flags = pq_getmsgbyte(in); @@ -1156,8 +1153,6 @@ logicalrep_read_stream_commit(StringInfo in, LogicalRepCommitData *commit_data) commit_data->commit_lsn = pq_getmsgint64(in); commit_data->end_lsn = pq_getmsgint64(in); commit_data->committime = pq_getmsgint64(in); - - return xid; } /* diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c index 49ceec3..04c9f96 100644 --- a/src/backend/replication/logical/tablesync.c +++ b/src/backend/replication/logical/tablesync.c @@ -568,7 +568,8 @@ process_syncing_tables_for_apply(XLogRecPtr current_lsn) MySubscription->oid, MySubscription->name, MyLogicalRepWorker->userid, - rstate->relid); + rstate->relid, + DSM_HANDLE_INVALID); hentry->last_start_time = now; } } @@ -589,6 +590,9 @@ process_syncing_tables_for_apply(XLogRecPtr current_lsn) void process_syncing_tables(XLogRecPtr current_lsn) { + if (MyLogicalRepWorker->subworker) + return; + if (am_tablesync_worker()) process_syncing_tables_for_sync(current_lsn); else @@ -1275,7 +1279,7 @@ LogicalRepSyncTableStart(XLogRecPtr *origin_startpos) * time this tablesync was launched. */ originid = replorigin_by_name(originname, false); - replorigin_session_setup(originid); + replorigin_session_setup(originid, true); replorigin_session_origin = originid; *origin_startpos = replorigin_session_get_progress(false); @@ -1386,7 +1390,7 @@ LogicalRepSyncTableStart(XLogRecPtr *origin_startpos) true /* go backward */ , true /* WAL log */ ); UnlockRelationOid(ReplicationOriginRelationId, RowExclusiveLock); - replorigin_session_setup(originid); + replorigin_session_setup(originid, true); replorigin_session_origin = originid; } else diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c index 7da7823..a317f72 100644 --- a/src/backend/replication/logical/worker.c +++ b/src/backend/replication/logical/worker.c @@ -22,8 +22,25 @@ * STREAMED TRANSACTIONS * --------------------- * Streamed transactions (large transactions exceeding a memory limit on the - * upstream) are not applied immediately, but instead, the data is written - * to temporary files and then applied at once when the final commit arrives. + * upstream) are applied via one of two approaches. + * + * 1) Separate background workers + * + * Assign a new bgworker (if available) as soon as the xact's first stream is + * received and the main apply worker will send changes to this new worker via + * shared memory. We keep this worker assigned till the transaction commit is + * received and also wait for the worker to finish at commit. This preserves + * commit ordering and avoids writing to and reading from file in most cases. + * We still need to spill if there is no worker available. We also need to + * allow stream_stop to complete by the background worker to finish it to avoid + * deadlocks because T-1's current stream of changes can update rows in + * conflicting order with T-2's next stream of changes. + * + * 2) Write to temporary files and apply when the final commit arrives + * + * If no worker is available to handle streamed transaction, the data is + * written to temporary files and then applied at once when the final commit + * arrives. * * Unlike the regular (non-streamed) case, handling streamed transactions has * to handle aborts of both the toplevel transaction and subtransactions. This @@ -174,11 +191,15 @@ #include "rewrite/rewriteHandler.h" #include "storage/buffile.h" #include "storage/bufmgr.h" +#include "storage/dsm.h" #include "storage/fd.h" #include "storage/ipc.h" #include "storage/lmgr.h" #include "storage/proc.h" #include "storage/procarray.h" +#include "storage/shm_mq.h" +#include "storage/shm_toc.h" +#include "storage/spin.h" #include "tcop/tcopprot.h" #include "utils/acl.h" #include "utils/builtins.h" @@ -198,6 +219,75 @@ #define NAPTIME_PER_CYCLE 1000 /* max sleep time between cycles (1s) */ +#define PG_LOGICAL_APPLY_SHM_MAGIC 0x79fb2447 // TODO Consider change + +/* + * Shared information among apply workers. + */ +typedef struct ParallelState +{ + slock_t mutex; + + /* State for apply background worker. */ + char state; + + TransactionId stream_xid; + uint32 n; /* id of apply background worker */ +} ParallelState; + +/* + * States for apply background worker. + */ +#define APPLY_BGWORKER_ATTACHED 'a' +#define APPLY_BGWORKER_READY 'r' +#define APPLY_BGWORKER_BUSY 'b' +#define APPLY_BGWORKER_FINISHED 'f' +#define APPLY_BGWORKER_EXIT 'e' + +/* + * Struct for maintaining a apply background worker. + */ +typedef struct WorkerState +{ + shm_mq_handle *mq_handle; + dsm_segment *dsm_seg; + ParallelState volatile *pstate; +} WorkerState; + +typedef struct WorkerEntry +{ + TransactionId xid; + WorkerState *wstate; +} WorkerEntry; + +/* Apply background workers hash table (initialized on first use) */ +static HTAB *ApplyWorkersHash = NULL; +static List *ApplyWorkersIdleList = NIL; +static List *ApplyWorkersList = NIL; + +/* Fields valid only for apply background workers */ +volatile ParallelState *MyParallelState = NULL; +static List *subxactlist = NIL; + +/* The number of changes during one streaming block */ +static uint32 nchanges = 0; + +/* Worker setup and interactions */ +static WorkerState *apply_bgworker_setup(void); +static WorkerState *find_or_start_apply_bgworker(TransactionId xid, + bool start); +static void apply_bgworker_setup_dsm(WorkerState *wstate); +static void apply_bgworker_wait_for(WorkerState *wstate, + char wait_for_state); +static void apply_bgworker_send_data(WorkerState *wstate, Size nbytes, + const void *data); +static void apply_bgworker_free(WorkerState *wstate); +static void apply_bgworker_check_status(void); +static void apply_bgworker_set_state(char state); + +#define am_apply_bgworker() (MyLogicalRepWorker->subworker) +#define applying_changes_in_bgworker() (in_streamed_transaction && stream_apply_worker != NULL) + typedef struct FlushPosition { dlist_node node; @@ -260,18 +350,20 @@ static XLogRecPtr remote_final_lsn = InvalidXLogRecPtr; static bool in_streamed_transaction = false; static TransactionId stream_xid = InvalidTransactionId; +static WorkerState *stream_apply_worker = NULL; /* * We enable skipping all data modification changes (INSERT, UPDATE, etc.) for * the subscription if the remote transaction's finish LSN matches the subskiplsn. * Once we start skipping changes, we don't stop it until we skip all changes of * the transaction even if pg_subscription is updated and MySubscription->skiplsn - * gets changed or reset during that. Also, in streaming transaction cases, we - * don't skip receiving and spooling the changes since we decide whether or not + * gets changed or reset during that. Also, in streaming transaction cases (streaming = on), + * we don't skip receiving and spooling the changes since we decide whether or not * to skip applying the changes when starting to apply changes. The subskiplsn is * cleared after successfully skipping the transaction or applying non-empty * transaction. The latter prevents the mistakenly specified subskiplsn from - * being left. + * being left. Note that we cannot skip the streaming transaction in apply mode, + * because we cannot get the finish LSN before applying the changes. */ static XLogRecPtr skip_xact_finish_lsn = InvalidXLogRecPtr; #define is_skipping_changes() (unlikely(!XLogRecPtrIsInvalid(skip_xact_finish_lsn))) @@ -426,43 +518,103 @@ end_replication_step(void) } /* - * Handle streamed transactions. + * Handle streamed transactions for both main apply worker and apply background + * worker. * - * If in streaming mode (receiving a block of streamed transaction), we - * simply redirect it to a file for the proper toplevel transaction. + * For the main apply worker, if in streaming mode (receiving a block of + * streamed transaction), we send the data to the apply background worker. + * + * For the apply background worker, define a savepoint if new subtransaction + * was started. * * Returns true for streamed transactions, false otherwise (regular mode). */ static bool handle_streamed_transaction(LogicalRepMsgType action, StringInfo s) { - TransactionId xid; + TransactionId current_xid = InvalidTransactionId; - /* not in streaming mode */ - if (!in_streamed_transaction) + /* + * Return if we are not in streaming mode and are not in a apply background + * worker. + */ + if (!in_streamed_transaction && !am_apply_bgworker()) return false; - Assert(stream_fd != NULL); Assert(TransactionIdIsValid(stream_xid)); /* * We should have received XID of the subxact as the first part of the * message, so extract it. */ - xid = pq_getmsgint(s, 4); + current_xid = pq_getmsgint(s, 4); - if (!TransactionIdIsValid(xid)) + if (!TransactionIdIsValid(current_xid)) ereport(ERROR, (errcode(ERRCODE_PROTOCOL_VIOLATION), errmsg_internal("invalid transaction ID in streamed replication transaction"))); - /* Add the new subxact to the array (unless already there). */ - subxact_info_add(xid); + if (am_apply_bgworker()) + { + /* + * Inside apply background worker we can figure out that new + * subtransaction was started if new change arrived with different xid. + * In that case we can define named savepoint, so that we were able to + * commit/rollback it separately later. + * + * Special case is if the first change comes from subtransaction, then + * we check that current_xid differs from stream_xid. + */ + if (current_xid != stream_xid && + !list_member_int(subxactlist, (int) current_xid)) + { + MemoryContext oldctx; + char spname[MAXPGPATH]; + + snprintf(spname, MAXPGPATH, "savepoint_for_xid_%u", current_xid); + + elog(LOG, "[Apply BGW #%u] defining savepoint %s", + MyParallelState->n, spname); + + DefineSavepoint(spname); + CommitTransactionCommand(); + + oldctx = MemoryContextSwitchTo(ApplyContext); + subxactlist = lappend_int(subxactlist, (int) current_xid); + MemoryContextSwitchTo(oldctx); + } + } + else if (applying_changes_in_bgworker()) + { + /* + * If we decided to apply the changes of this transaction in a apply + * background worker, pass the data to the worker. + */ + apply_bgworker_send_data(stream_apply_worker, s->len, s->data); + nchanges += 1; + + /* + * XXX The publisher side doesn't always send relation update message + * after the streaming transaction, so update the relation in main + * apply worker here. + */ + if (action == LOGICAL_REP_MSG_RELATION) + { + LogicalRepRelation *rel = logicalrep_read_rel(s); + logicalrep_relmap_update(rel); + } + + } + else + { + /* Add the new subxact to the array (unless already there). */ + subxact_info_add(current_xid); - /* write the change to the current file */ - stream_write_change(action, s); + /* write the change to the current file */ + stream_write_change(action, s); + } - return true; + return !am_apply_bgworker(); } /* @@ -844,6 +996,9 @@ apply_handle_commit(StringInfo s) apply_handle_commit_internal(&commit_data); + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); + /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(commit_data.end_lsn); @@ -898,7 +1053,9 @@ apply_handle_prepare_internal(LogicalRepPreparedTxnData *prepare_data) * BeginTransactionBlock is necessary to balance the EndTransactionBlock * called within the PrepareTransactionBlock below. */ - BeginTransactionBlock(); + if (!IsTransactionBlock()) + BeginTransactionBlock(); + CommitTransactionCommand(); /* Completes the preceding Begin command. */ /* @@ -950,6 +1107,9 @@ apply_handle_prepare(StringInfo s) in_remote_transaction = false; + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); + /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(prepare_data.end_lsn); @@ -974,32 +1134,57 @@ apply_handle_commit_prepared(StringInfo s) { LogicalRepCommitPreparedTxnData prepare_data; char gid[GIDSIZE]; + WorkerState *wstate; logicalrep_read_commit_prepared(s, &prepare_data); set_apply_error_context_xact(prepare_data.xid, prepare_data.commit_lsn); - /* Compute GID for two_phase transactions. */ - TwoPhaseTransactionGid(MySubscription->oid, prepare_data.xid, - gid, sizeof(gid)); + /* Check if we have prepared transaction in another bgworker */ + wstate = find_or_start_apply_bgworker(prepare_data.xid, false); - /* There is no transaction when COMMIT PREPARED is called */ - begin_replication_step(); + if (wstate) + { + elog(DEBUG1, "received commit for streamed transaction %u", prepare_data.xid); - /* - * Update origin state so we can restart streaming from correct position - * in case of crash. - */ - replorigin_session_origin_lsn = prepare_data.end_lsn; - replorigin_session_origin_timestamp = prepare_data.commit_time; + /* Send commit message */ + apply_bgworker_send_data(wstate, s->len, s->data); + + /* Wait for bgworker to finish */ + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_FINISHED); + + apply_bgworker_free(wstate); + } + else + { + /* Compute GID for two_phase transactions. */ + TwoPhaseTransactionGid(MySubscription->oid, prepare_data.xid, + gid, sizeof(gid)); + + /* There is no transaction when COMMIT PREPARED is called */ + begin_replication_step(); + + /* + * Update origin state so we can restart streaming from correct position + * in case of crash. + */ + replorigin_session_origin_lsn = prepare_data.end_lsn; + replorigin_session_origin_timestamp = prepare_data.commit_time; + + FinishPreparedTransaction(gid, true); + end_replication_step(); + CommitTransactionCommand(); + + apply_bgworker_set_state(APPLY_BGWORKER_FINISHED); + } - FinishPreparedTransaction(gid, true); - end_replication_step(); - CommitTransactionCommand(); pgstat_report_stat(false); store_flush_position(prepare_data.end_lsn); in_remote_transaction = false; + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); + /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(prepare_data.end_lsn); @@ -1017,37 +1202,57 @@ apply_handle_rollback_prepared(StringInfo s) { LogicalRepRollbackPreparedTxnData rollback_data; char gid[GIDSIZE]; + WorkerState *wstate; logicalrep_read_rollback_prepared(s, &rollback_data); set_apply_error_context_xact(rollback_data.xid, rollback_data.rollback_end_lsn); - /* Compute GID for two_phase transactions. */ - TwoPhaseTransactionGid(MySubscription->oid, rollback_data.xid, - gid, sizeof(gid)); + /* Check if we are processing the prepared transaction in a bgworker */ + wstate = find_or_start_apply_bgworker(rollback_data.xid, false); - /* - * It is possible that we haven't received prepare because it occurred - * before walsender reached a consistent point or the two_phase was still - * not enabled by that time, so in such cases, we need to skip rollback - * prepared. - */ - if (LookupGXact(gid, rollback_data.prepare_end_lsn, - rollback_data.prepare_time)) + if (wstate) + { + apply_bgworker_send_data(wstate, s->len, s->data); + + /* Wait for bgworker to finish */ + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_FINISHED); + + elog(LOG, "rollback prepared streaming of xid %u", rollback_data.xid); + + apply_bgworker_free(wstate); + } + else { + /* Compute GID for two_phase transactions. */ + TwoPhaseTransactionGid(MySubscription->oid, rollback_data.xid, + gid, sizeof(gid)); + /* - * Update origin state so we can restart streaming from correct - * position in case of crash. + * It is possible that we haven't received prepare because it occurred + * before walsender reached a consistent point or the two_phase was still + * not enabled by that time, so in such cases, we need to skip rollback + * prepared. */ - replorigin_session_origin_lsn = rollback_data.rollback_end_lsn; - replorigin_session_origin_timestamp = rollback_data.rollback_time; + if (LookupGXact(gid, rollback_data.prepare_end_lsn, + rollback_data.prepare_time)) + { + /* + * Update origin state so we can restart streaming from correct + * position in case of crash. + */ + replorigin_session_origin_lsn = rollback_data.rollback_end_lsn; + replorigin_session_origin_timestamp = rollback_data.rollback_time; - /* There is no transaction when ABORT/ROLLBACK PREPARED is called */ - begin_replication_step(); - FinishPreparedTransaction(gid, false); - end_replication_step(); - CommitTransactionCommand(); + /* There is no transaction when ABORT/ROLLBACK PREPARED is called */ + begin_replication_step(); + FinishPreparedTransaction(gid, false); + end_replication_step(); + CommitTransactionCommand(); + + clear_subscription_skip_lsn(rollback_data.rollback_end_lsn); + } - clear_subscription_skip_lsn(rollback_data.rollback_end_lsn); + apply_bgworker_set_state(APPLY_BGWORKER_FINISHED); } pgstat_report_stat(false); @@ -1055,6 +1260,9 @@ apply_handle_rollback_prepared(StringInfo s) store_flush_position(rollback_data.rollback_end_lsn); in_remote_transaction = false; + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); + /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(rollback_data.rollback_end_lsn); @@ -1063,11 +1271,131 @@ apply_handle_rollback_prepared(StringInfo s) } /* - * Handle STREAM PREPARE. + * Look up worker inside ApplyWorkersHash for requested xid. * - * Logic is in two parts: - * 1. Replay all the spooled operations - * 2. Mark the transaction as prepared + * If start flag is true, try to start a new worker if not found in hash table. + */ +static WorkerState * +find_or_start_apply_bgworker(TransactionId xid, bool start) +{ + bool found; + WorkerState *wstate; + WorkerEntry *entry = NULL; + + if (!TransactionIdIsValid(xid)) + return NULL; + + if (!start && ApplyWorkersHash == NULL) + return NULL; + + /* + * We don't start new background worker if we are not in streaming apply + * mode. + */ + if (MySubscription->stream != SUBSTREAM_APPLY) + return NULL; + + /* + * We don't start new background worker if user has set skiplsn as it's + * possible that user want to skip the streaming transaction. For streaming + * transaction, we need to spill the transaction to disk so that we can get + * the last LSN of the transaction to judge whether to skip before starting + * to apply the change. + */ + if (start && !XLogRecPtrIsInvalid(MySubscription->skiplsn)) + return NULL; + + /* + * For streaming transactions that are being applied in bgworker, we cannot + * decide whether to apply the change for a relation that is not in the + * READY state (see should_apply_changes_for_rel) as we won't know + * remote_final_lsn by that time. So, we don't start new bgworker in this + * case. + */ + if (start && !AllTablesyncsReady()) + return NULL; + + /* First time through, initialize apply workers hashtable */ + if (ApplyWorkersHash == NULL) + { + HASHCTL ctl; + + MemSet(&ctl, 0, sizeof(ctl)); + ctl.keysize = sizeof(TransactionId); + ctl.entrysize = sizeof(WorkerEntry); + ctl.hcxt = ApplyContext; + + ApplyWorkersHash = hash_create("logical apply workers hash", 8, &ctl, + HASH_ELEM | HASH_BLOBS | HASH_CONTEXT); + } + + /* + * Find entry for requested transaction. + */ + entry = hash_search(ApplyWorkersHash, &xid, start ? HASH_ENTER : HASH_FIND, + &found); + if (found) + { + entry->wstate->pstate->state = APPLY_BGWORKER_BUSY; + return entry->wstate; + } + else if (!start) + return NULL; + + /* If there is at least one worker in the idle list, then take one. */ + if (list_length(ApplyWorkersIdleList) > 0) + { + wstate = (WorkerState *) llast(ApplyWorkersIdleList); + ApplyWorkersIdleList = list_delete_last(ApplyWorkersIdleList); + } + else + { + wstate = apply_bgworker_setup(); + + if (wstate == NULL) + { + /* + * If the bgworker cannot be launched, remove entry in hash table. + */ + hash_search(ApplyWorkersHash, &xid, HASH_REMOVE, &found); + return NULL; + } + } + + /* Fill up the hash entry */ + wstate->pstate->state = APPLY_BGWORKER_BUSY; + wstate->pstate->stream_xid = xid; + entry->wstate = wstate; + entry->xid = xid; + + return wstate; +} + +/* + * Add the worker to the freelist and remove the entry from hash table. + */ +static void +apply_bgworker_free(WorkerState *wstate) +{ + bool found; + MemoryContext oldctx; + TransactionId xid = wstate->pstate->stream_xid; + + oldctx = MemoryContextSwitchTo(ApplyContext); + + hash_search(ApplyWorkersHash, &xid, + HASH_REMOVE, &found); + + elog(LOG, "adding finished apply worker #%u for xid %u to the idle list", + wstate->pstate->n, wstate->pstate->stream_xid); + + ApplyWorkersIdleList = lappend(ApplyWorkersIdleList, wstate); + + MemoryContextSwitchTo(oldctx); +} + +/* + * Handle STREAM PREPARE. */ static void apply_handle_stream_prepare(StringInfo s) @@ -1088,24 +1416,71 @@ apply_handle_stream_prepare(StringInfo s) logicalrep_read_stream_prepare(s, &prepare_data); set_apply_error_context_xact(prepare_data.xid, prepare_data.prepare_lsn); - elog(DEBUG1, "received prepare for streamed transaction %u", prepare_data.xid); + /* + * If we are in a bgworker, just prepare the transaction. + */ + if (am_apply_bgworker()) + { + elog(LOG, "received prepare for streamed transaction %u", prepare_data.xid); - /* Replay all the spooled operations. */ - apply_spooled_messages(prepare_data.xid, prepare_data.prepare_lsn); + /* Mark the transaction as prepared. */ + apply_handle_prepare_internal(&prepare_data); - /* Mark the transaction as prepared. */ - apply_handle_prepare_internal(&prepare_data); + CommitTransactionCommand(); - CommitTransactionCommand(); + pgstat_report_stat(false); - pgstat_report_stat(false); + list_free(subxactlist); + subxactlist = NIL; - store_flush_position(prepare_data.end_lsn); + apply_bgworker_set_state(APPLY_BGWORKER_READY); + } + else + { + WorkerState *wstate = find_or_start_apply_bgworker(prepare_data.xid, false); + + /* + * If we are in main apply worker, check if we are processing this + * transaction in a bgworker. + */ + if (wstate) + { + apply_bgworker_send_data(wstate, s->len, s->data); + + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_READY); + + pgstat_report_stat(false); + store_flush_position(prepare_data.end_lsn); + } + else + { + /* + * If we are in main apply worker and the transaction has been + * serialized to file, replay all the spooled operations. + */ + apply_spooled_messages(prepare_data.xid, prepare_data.prepare_lsn); + + /* Mark the transaction as prepared. */ + apply_handle_prepare_internal(&prepare_data); + + CommitTransactionCommand(); + + pgstat_report_stat(false); + + store_flush_position(prepare_data.end_lsn); + + in_remote_transaction = false; + + /* unlink the files with serialized changes and subxact info. */ + stream_cleanup_files(MyLogicalRepWorker->subid, prepare_data.xid); + } + } in_remote_transaction = false; + stream_apply_worker = NULL; - /* unlink the files with serialized changes and subxact info. */ - stream_cleanup_files(MyLogicalRepWorker->subid, prepare_data.xid); + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(prepare_data.end_lsn); @@ -1155,15 +1530,6 @@ apply_handle_stream_start(StringInfo s) (errcode(ERRCODE_PROTOCOL_VIOLATION), errmsg_internal("duplicate STREAM START message"))); - /* - * Start a transaction on stream start, this transaction will be committed - * on the stream stop unless it is a tablesync worker in which case it - * will be committed after processing all the messages. We need the - * transaction for handling the buffile, used for serializing the - * streaming data and subxact info. - */ - begin_replication_step(); - /* notify handle methods we're processing a remote transaction */ in_streamed_transaction = true; @@ -1177,36 +1543,90 @@ apply_handle_stream_start(StringInfo s) set_apply_error_context_xact(stream_xid, InvalidXLogRecPtr); + if (am_apply_bgworker()) + { + /* + * Make sure the handle apply_dispatch methods are aware we're in a + * remote transaction. + */ + in_remote_transaction = true; + + /* If we are in a bgworker, begin the transaction */ + AcceptInvalidationMessages(); + maybe_reread_subscription(); + + StartTransactionCommand(); + BeginTransactionBlock(); + CommitTransactionCommand(); + + return; + } + /* - * Initialize the worker's stream_fileset if we haven't yet. This will be - * used for the entire duration of the worker so create it in a permanent - * context. We create this on the very first streaming message from any - * transaction and then use it for this and other streaming transactions. - * Now, we could create a fileset at the start of the worker as well but - * then we won't be sure that it will ever be used. + * If we are in main apply worker, check if there is any free bgworker + * we can use to process this transaction. */ - if (MyLogicalRepWorker->stream_fileset == NULL) + stream_apply_worker = find_or_start_apply_bgworker(stream_xid, first_segment); + + if (applying_changes_in_bgworker()) { - MemoryContext oldctx; + /* + * If we have free worker or we already started to apply this + * transaction in bgworker, we pass the data to worker. + */ + if (first_segment) + apply_bgworker_send_data(stream_apply_worker, s->len, s->data); - oldctx = MemoryContextSwitchTo(ApplyContext); + nchanges = 0; + elog(LOG, "starting streaming of xid %u", stream_xid); + } - MyLogicalRepWorker->stream_fileset = palloc(sizeof(FileSet)); - FileSetInit(MyLogicalRepWorker->stream_fileset); + /* + * If no worker is available for the first stream start, we start to + * serialize all the changes of the transaction. + */ + else + { + /* + * Start a transaction on stream start, this transaction will be committed + * on the stream stop unless it is a tablesync worker in which case it + * will be committed after processing all the messages. We need the + * transaction for handling the buffile, used for serializing the + * streaming data and subxact info. + */ + begin_replication_step(); - MemoryContextSwitchTo(oldctx); - } + /* + * Initialize the worker's stream_fileset if we haven't yet. This will be + * used for the entire duration of the worker so create it in a permanent + * context. We create this on the very first streaming message from any + * transaction and then use it for this and other streaming transactions. + * Now, we could create a fileset at the start of the worker as well but + * then we won't be sure that it will ever be used. + */ + if (MyLogicalRepWorker->stream_fileset == NULL) + { + MemoryContext oldctx; - /* open the spool file for this transaction */ - stream_open_file(MyLogicalRepWorker->subid, stream_xid, first_segment); + oldctx = MemoryContextSwitchTo(ApplyContext); - /* if this is not the first segment, open existing subxact file */ - if (!first_segment) - subxact_info_read(MyLogicalRepWorker->subid, stream_xid); + MyLogicalRepWorker->stream_fileset = palloc(sizeof(FileSet)); + FileSetInit(MyLogicalRepWorker->stream_fileset); - pgstat_report_activity(STATE_RUNNING, NULL); + MemoryContextSwitchTo(oldctx); + } - end_replication_step(); + /* open the spool file for this transaction */ + stream_open_file(MyLogicalRepWorker->subid, stream_xid, first_segment); + + /* if this is not the first segment, open existing subxact file */ + if (!first_segment) + subxact_info_read(MyLogicalRepWorker->subid, stream_xid); + + end_replication_step(); + } + + pgstat_report_activity(STATE_RUNNING, NULL); } /* @@ -1220,44 +1640,47 @@ apply_handle_stream_stop(StringInfo s) (errcode(ERRCODE_PROTOCOL_VIOLATION), errmsg_internal("STREAM STOP message without STREAM START"))); - /* - * Close the file with serialized changes, and serialize information about - * subxacts for the toplevel transaction. - */ - subxact_info_write(MyLogicalRepWorker->subid, stream_xid); - stream_close_file(); + if (applying_changes_in_bgworker()) + { + char action = LOGICAL_REP_MSG_STREAM_STOP; - /* We must be in a valid transaction state */ - Assert(IsTransactionState()); + apply_bgworker_send_data(stream_apply_worker, 1, &action); + apply_bgworker_wait_for(stream_apply_worker, APPLY_BGWORKER_READY); - /* Commit the per-stream transaction */ - CommitTransactionCommand(); + elog(LOG, "stopped streaming of xid %u, %u changes streamed", stream_xid, nchanges); + } + else + { + /* + * Close the file with serialized changes, and serialize information + * about subxacts for the toplevel transaction. + */ + subxact_info_write(MyLogicalRepWorker->subid, stream_xid); + stream_close_file(); - in_streamed_transaction = false; + /* We must be in a valid transaction state */ + Assert(IsTransactionState()); + + /* Commit the per-stream transaction */ + CommitTransactionCommand(); - /* Reset per-stream context */ - MemoryContextReset(LogicalStreamingContext); + /* Reset per-stream context */ + MemoryContextReset(LogicalStreamingContext); + } + + in_streamed_transaction = false; + stream_apply_worker = NULL; pgstat_report_activity(STATE_IDLE, NULL); reset_apply_error_context_info(); } /* - * Handle STREAM abort message. + * Handle STREAM ABORT message when the transaction was spilled to disk. */ static void -apply_handle_stream_abort(StringInfo s) +serialize_stream_abort(TransactionId xid, TransactionId subxid) { - TransactionId xid; - TransactionId subxid; - - if (in_streamed_transaction) - ereport(ERROR, - (errcode(ERRCODE_PROTOCOL_VIOLATION), - errmsg_internal("STREAM ABORT message without STREAM STOP"))); - - logicalrep_read_stream_abort(s, &xid, &subxid); - /* * If the two XIDs are the same, it's in fact abort of toplevel xact, so * just delete the files with serialized info. @@ -1339,8 +1762,120 @@ apply_handle_stream_abort(StringInfo s) end_replication_step(); CommitTransactionCommand(); } +} - reset_apply_error_context_info(); +/* + * Handle STREAM ABORT message. + */ +static void +apply_handle_stream_abort(StringInfo s) +{ + TransactionId xid; + TransactionId subxid; + + if (in_streamed_transaction) + ereport(ERROR, + (errcode(ERRCODE_PROTOCOL_VIOLATION), + errmsg_internal("STREAM COMMIT message without STREAM STOP"))); + + logicalrep_read_stream_abort(s, &xid, &subxid); + + if (am_apply_bgworker()) + { + ereport(LOG, + (errmsg("[Apply BGW #%u] aborting current transaction xid=%u, subxid=%u", + MyParallelState->n, GetCurrentTransactionIdIfAny(), GetCurrentSubTransactionId()))); + + /* + * If the two XIDs are the same, it's in fact abort of toplevel xact, + * so just free the subxactlist. + */ + if (subxid == xid) + { + set_apply_error_context_xact(subxid, InvalidXLogRecPtr); + + AbortCurrentTransaction(); + + EndTransactionBlock(false); + CommitTransactionCommand(); + + in_remote_transaction = false; + + list_free(subxactlist); + subxactlist = NIL; + + apply_bgworker_set_state(APPLY_BGWORKER_FINISHED); + } + else + { + /* + * OK, so it's a subxact. Rollback to the savepoint. + * + * We also need to read the subxactlist, determine the offset + * tracked for the subxact, and truncate the list. + */ + int i; + bool found = false; + char spname[MAXPGPATH]; + + set_apply_error_context_xact(subxid, InvalidXLogRecPtr); + snprintf(spname, MAXPGPATH, "savepoint_for_xid_%u", subxid); + + ereport(LOG, + (errmsg("[Apply BGW #%u] rolling back to savepoint %s", + MyParallelState->n, spname))); + + for(i = list_length(subxactlist) - 1; i >= 0; i--) + { + xid = (TransactionId) list_nth_int(subxactlist, i); + if (xid == subxid) + { + found = true; + break; + } + } + + if (found) + { + elog(LOG, "rolled back to savepoint %s", spname); + RollbackToSavepoint(spname); + CommitTransactionCommand(); + subxactlist = list_truncate(subxactlist, i + 1); + } + + apply_bgworker_set_state(APPLY_BGWORKER_READY); + } + + reset_apply_error_context_info(); + } + else + { + WorkerState *wstate = find_or_start_apply_bgworker(xid, false); + + /* + * If we are in main apply worker, check if we are processing this + * transaction in a bgworker. + */ + if (wstate) + { + apply_bgworker_send_data(wstate, s->len, s->data); + + if (subxid == xid) + { + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_FINISHED); + apply_bgworker_free(wstate); + } + else + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_READY); + } + + /* + * We are in main apply worker and the transaction has been serialized + * to file. + */ + else + serialize_stream_abort(xid, subxid); + } } /* @@ -1463,40 +1998,6 @@ apply_spooled_messages(TransactionId xid, XLogRecPtr lsn) } /* - * Handle STREAM COMMIT message. - */ -static void -apply_handle_stream_commit(StringInfo s) -{ - TransactionId xid; - LogicalRepCommitData commit_data; - - if (in_streamed_transaction) - ereport(ERROR, - (errcode(ERRCODE_PROTOCOL_VIOLATION), - errmsg_internal("STREAM COMMIT message without STREAM STOP"))); - - xid = logicalrep_read_stream_commit(s, &commit_data); - set_apply_error_context_xact(xid, commit_data.commit_lsn); - - elog(DEBUG1, "received commit for streamed transaction %u", xid); - - apply_spooled_messages(xid, commit_data.commit_lsn); - - apply_handle_commit_internal(&commit_data); - - /* unlink the files with serialized changes and subxact info */ - stream_cleanup_files(MyLogicalRepWorker->subid, xid); - - /* Process any tables that are being synchronized in parallel. */ - process_syncing_tables(commit_data.end_lsn); - - pgstat_report_activity(STATE_IDLE, NULL); - - reset_apply_error_context_info(); -} - -/* * Helper function for apply_handle_commit and apply_handle_stream_commit. */ static void @@ -2445,6 +2946,101 @@ apply_handle_truncate(StringInfo s) end_replication_step(); } +/* + * Handle STREAM COMMIT message. + */ +static void +apply_handle_stream_commit(StringInfo s) +{ + LogicalRepCommitData commit_data; + TransactionId xid; + + if (in_streamed_transaction) + ereport(ERROR, + (errcode(ERRCODE_PROTOCOL_VIOLATION), + errmsg_internal("STREAM COMMIT message without STREAM STOP"))); + + xid = pq_getmsgint(s, 4); + logicalrep_read_stream_commit(s, &commit_data); + set_apply_error_context_xact(xid, commit_data.commit_lsn); + + elog(DEBUG1, "received commit for streamed transaction %u", xid); + + if (am_apply_bgworker()) + { + /* + * Update origin state so we can restart streaming from correct + * position in case of crash. + */ + replorigin_session_origin_lsn = commit_data.end_lsn; + replorigin_session_origin_timestamp = commit_data.committime; + + CommitTransactionCommand(); + EndTransactionBlock(false); + CommitTransactionCommand(); + + in_remote_transaction = false; + + pgstat_report_stat(false); + + list_free(subxactlist); + subxactlist = NIL; + + apply_bgworker_set_state(APPLY_BGWORKER_FINISHED); + } + else + { + /* + * If we are in main apply worker, check if we are processing this + * transaction in a apply background worker. + */ + WorkerState *wstate = find_or_start_apply_bgworker(xid, false); + + if (wstate) + { + /* Send commit message */ + apply_bgworker_send_data(wstate, s->len, s->data); + + /* Wait for apply background worker to finish */ + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_FINISHED); + + pgstat_report_stat(false); + store_flush_position(commit_data.end_lsn); + stop_skipping_changes(); + + apply_bgworker_free(wstate); + + /* + * The transaction is either non-empty or skipped, so we clear the + * subskiplsn. + */ + clear_subscription_skip_lsn(commit_data.commit_lsn); + } + else + { + /* + * If we are in main apply worker and the transaction has been + * serialized to file, replay all the spooled operations. + */ + apply_spooled_messages(xid, commit_data.commit_lsn); + + apply_handle_commit_internal(&commit_data); + + /* unlink the files with serialized changes and subxact info */ + stream_cleanup_files(MyLogicalRepWorker->subid, xid); + } + } + + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); + + /* Process any tables that are being synchronized in parallel. */ + process_syncing_tables(commit_data.end_lsn); + + pgstat_report_activity(STATE_IDLE, NULL); + + reset_apply_error_context_info(); +} /* * Logical replication protocol message dispatcher. @@ -2511,6 +3107,8 @@ apply_dispatch(StringInfo s) break; case LOGICAL_REP_MSG_STREAM_START: + /* FIXME : log for debugging here. */ + elog(LOG, "LOGICAL_REP_MSG_STREAM_START"); apply_handle_stream_start(s); break; @@ -2618,6 +3216,10 @@ store_flush_position(XLogRecPtr remote_lsn) { FlushPosition *flushpos; + /* We only need to collect the LSN in main apply worker */ + if (am_apply_bgworker()) + return; + /* Need to do this in permanent context */ MemoryContextSwitchTo(ApplyContext); @@ -2794,6 +3396,9 @@ LogicalRepApplyLoop(XLogRecPtr last_received) AcceptInvalidationMessages(); maybe_reread_subscription(); + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); + /* Process any table synchronization changes. */ process_syncing_tables(last_received); } @@ -3686,7 +4291,7 @@ ApplyWorkerMain(Datum main_arg) originid = replorigin_by_name(originname, true); if (!OidIsValid(originid)) originid = replorigin_create(originname); - replorigin_session_setup(originid); + replorigin_session_setup(originid, true); replorigin_session_origin = originid; origin_startpos = replorigin_session_get_progress(false); CommitTransactionCommand(); @@ -3733,7 +4338,7 @@ ApplyWorkerMain(Datum main_arg) options.proto.logical.publication_names = MySubscription->publications; options.proto.logical.binary = MySubscription->binary; - options.proto.logical.streaming = MySubscription->stream; + options.proto.logical.streaming = (MySubscription->stream != SUBSTREAM_OFF); options.proto.logical.twophase = false; if (!am_tablesync_worker()) @@ -3891,7 +4496,8 @@ clear_subscription_skip_lsn(XLogRecPtr finish_lsn) XLogRecPtr myskiplsn = MySubscription->skiplsn; bool started_tx = false; - if (likely(XLogRecPtrIsInvalid(myskiplsn))) + if (likely(XLogRecPtrIsInvalid(myskiplsn)) || + am_apply_bgworker()) return; if (!IsTransactionState()) @@ -4027,3 +4633,529 @@ reset_apply_error_context_info(void) apply_error_callback_arg.remote_attnum = -1; set_apply_error_context_xact(InvalidTransactionId, InvalidXLogRecPtr); } + +/* Apply Background Worker main loop */ +static void +LogicalApplyBgwLoop(shm_mq_handle *mqh, volatile ParallelState *pst) +{ + shm_mq_result shmq_res; + PGPROC *registrant; + ErrorContextCallback errcallback; + XLogRecPtr last_received = InvalidXLogRecPtr; + + registrant = BackendPidGetProc(MyBgworkerEntry->bgw_notify_pid); + SetLatch(®istrant->procLatch); + + /* + * Push apply error context callback. Fields will be filled during + * applying a change. + */ + errcallback.callback = apply_error_callback; + errcallback.previous = error_context_stack; + error_context_stack = &errcallback; + + for (;;) + { + void *data; + Size len; + StringInfoData s; + MemoryContext oldctx; + XLogRecPtr start_lsn; + XLogRecPtr end_lsn; + TimestampTz send_time; + + CHECK_FOR_INTERRUPTS(); + + /* Ensure we are reading the data into our memory context. */ + oldctx = MemoryContextSwitchTo(ApplyMessageContext); + + shmq_res = shm_mq_receive(mqh, &len, &data, false); + + if (shmq_res != SHM_MQ_SUCCESS) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("lost connection to the main apply worker"))); + + if (len == 0) + { + elog(LOG, "[Apply BGW #%u] got zero-length message, stopping", pst->n); + break; + } + + s.cursor = 0; + s.maxlen = -1; + s.data = (char *) data; + s.len = len; + + /* + * We use first byte of message for additional communication between + * main Logical replication worker and apply bgworkers, so if it + * differs from 'w', then process it first. + */ + switch (pq_getmsgbyte(&s)) + { + /* End message of streaming chunk */ + case LOGICAL_REP_MSG_STREAM_STOP: + elog(LOG, "[Apply BGW #%u] ended processing streaming chunk," + "waiting on shm_mq_receive", pst->n); + + apply_bgworker_set_state(APPLY_BGWORKER_READY); + + SetLatch(®istrant->procLatch); + + in_streamed_transaction = false; + pgstat_report_activity(STATE_IDLE, NULL); + continue; + case 'w': + break; + default: + elog(ERROR, "unexpected message"); + break; + } + + start_lsn = pq_getmsgint64(&s); + end_lsn = pq_getmsgint64(&s); + send_time = pq_getmsgint64(&s); + + if (last_received < start_lsn) + last_received = start_lsn; + + if (last_received < end_lsn) + last_received = end_lsn; + + /* + * TO IMPROVE: Do we need to display the bgworker's information in + * pg_stat_replication ? + */ + UpdateWorkerStats(last_received, send_time, false); + + apply_dispatch(&s); + + if (ConfigReloadPending) + { + ConfigReloadPending = false; + ProcessConfigFile(PGC_SIGHUP); + } + + MemoryContextSwitchTo(oldctx); + MemoryContextReset(ApplyMessageContext); + } + + MemoryContextSwitchTo(TopMemoryContext); + MemoryContextReset(ApplyContext); + + /* Pop the error context stack */ + error_context_stack = errcallback.previous; + + elog(LOG, "[Apply BGW #%u] exiting", pst->n); + + /* Signal main process that we are done. */ + SetLatch(®istrant->procLatch); +} + +/* + * Set the failed flag so that the main apply worker can realize we have + * shutdown. + */ +static void +ApplyBgwShutdown(int code, Datum arg) +{ + SpinLockAcquire(&MyParallelState->mutex); + MyParallelState->state = APPLY_BGWORKER_EXIT; + SpinLockRelease(&MyParallelState->mutex); + + dsm_detach((dsm_segment *) DatumGetPointer(arg)); +} + +/* + * Apply Background Worker entry point + */ +void +ApplyBgworkerMain(Datum main_arg) +{ + volatile ParallelState *pst; + + dsm_handle handle; + dsm_segment *seg; + shm_toc *toc; + shm_mq *mq; + shm_mq_handle *mqh; + MemoryContext oldcontext; + RepOriginId originid; + int worker_slot = DatumGetInt32(main_arg); + char originname[NAMEDATALEN]; + + /* Attach to slot */ + logicalrep_worker_attach(worker_slot); + + MemoryContextSwitchTo(TopMemoryContext); + + /* Load the subscription into persistent memory context. */ + ApplyContext = AllocSetContextCreate(TopMemoryContext, + "ApplyContext", + ALLOCSET_DEFAULT_SIZES); + + /* + * Init the ApplyMessageContext which we clean up after each replication + * protocol message. + */ + ApplyMessageContext = AllocSetContextCreate(ApplyContext, + "ApplyMessageContext", + ALLOCSET_DEFAULT_SIZES); + + /* Setup signal handling */ + pqsignal(SIGHUP, SignalHandlerForConfigReload); + pqsignal(SIGTERM, die); + BackgroundWorkerUnblockSignals(); + + /* + * Connect to the dynamic shared memory segment. + * + * The backend that registered this worker passed us the ID of a shared + * memory segment to which we must attach for further instructions. In + * order to attach to dynamic shared memory, we need a resource owner. + * Once we've mapped the segment in our address space, attach to the table + * of contents so we can locate the various data structures we'll need to + * find within the segment. + */ + CurrentResourceOwner = ResourceOwnerCreate(NULL, "Logical apply worker"); + memcpy(&handle, MyBgworkerEntry->bgw_extra, sizeof(dsm_handle)); + seg = dsm_attach(handle); + if (seg == NULL) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("unable to map dynamic shared memory segment"))); + toc = shm_toc_attach(PG_LOGICAL_APPLY_SHM_MAGIC, dsm_segment_address(seg)); + if (toc == NULL) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("bad magic number in dynamic shared memory segment"))); + + before_shmem_exit(ApplyBgwShutdown, PointerGetDatum(seg)); + + /* Look up the parallel state. */ + pst = shm_toc_lookup(toc, 0, false); + MyParallelState = pst; + + /* + * Attach to the message queue. + */ + mq = shm_toc_lookup(toc, 1, false); + shm_mq_set_receiver(mq, MyProc); + mqh = shm_mq_attach(mq, seg, NULL); + + /* Connect to our database. */ + BackgroundWorkerInitializeConnectionByOid(MyLogicalRepWorker->dbid, + MyLogicalRepWorker->userid, + 0); + + /* + * Set the client encoding to the database encoding, since that is what + * the leader will expect. + */ + SetClientEncoding(GetDatabaseEncoding()); + + stream_xid = pst->stream_xid; + + StartTransactionCommand(); + oldcontext = MemoryContextSwitchTo(ApplyContext); + + MySubscription = GetSubscription(MyLogicalRepWorker->subid, true); + if (!MySubscription) + { + ereport(LOG, + (errmsg("logical replication apply worker for subscription %u will not " + "start because the subscription was removed during startup", + MyLogicalRepWorker->subid))); + proc_exit(0); + } + + MySubscriptionValid = true; + MemoryContextSwitchTo(oldcontext); + + /* Setup synchronous commit according to the user's wishes */ + SetConfigOption("synchronous_commit", MySubscription->synccommit, + PGC_BACKEND, PGC_S_OVERRIDE); + + /* Keep us informed about subscription changes. */ + CacheRegisterSyscacheCallback(SUBSCRIPTIONOID, + subscription_change_cb, + (Datum) 0); + + CommitTransactionCommand(); + + /* Setup replication origin tracking. */ + StartTransactionCommand(); + snprintf(originname, sizeof(originname), "pg_%u", MySubscription->oid); + originid = replorigin_by_name(originname, true); + if (!OidIsValid(originid)) + originid = replorigin_create(originname); + replorigin_session_setup(originid, false); + replorigin_session_origin = originid; + CommitTransactionCommand(); + + /* + * Allocate the origin name in long-lived context for error context + * message. + */ + apply_error_callback_arg.origin_name = MemoryContextStrdup(ApplyContext, + originname); + + /* + * Indicate that we're fully initialized and ready to begin the main part + * of the apply operation. + */ + apply_bgworker_set_state(APPLY_BGWORKER_ATTACHED); + + elog(LOG, "[Apply BGW #%u] started", pst->n); + + PG_TRY(); + { + LogicalApplyBgwLoop(mqh, pst); + } + PG_CATCH(); + { + /* + * Report the worker failed while applying streaming transaction + * changes. Abort the current transaction so that the stats message is + * sent in an idle state. + */ + AbortOutOfAnyTransaction(); + pgstat_report_subscription_error(MySubscription->oid, false); + + PG_RE_THROW(); + } + PG_END_TRY(); + + /* + * We're done. Explicitly detach the shared memory segment so that we + * don't get a resource leak warning at commit time. This will fire any + * on_dsm_detach callbacks we've registered, as well. Once that's done, + * we can go ahead and exit. + */ + dsm_detach(seg); + proc_exit(0); +} + +/* + * Set up a dynamic shared memory segment. + * + * We set up a control region that contains a ParallelState, + * plus one region per message queue. There are as many message queues as + * the number of workers. + */ +static void +apply_bgworker_setup_dsm(WorkerState *wstate) +{ + shm_toc_estimator e; + int toc_key = 0; + Size segsize; + dsm_segment *seg; + shm_toc *toc; + ParallelState *pst; + shm_mq *mq; + int64 queue_size = 160000000; /* 16 MB for now */ + + /* + * Estimate how much shared memory we need. + * + * Because the TOC machinery may choose to insert padding of oddly-sized + * requests, we must estimate each chunk separately. + * + * We need one key to register the location of the header, and we need + * nworkers keys to track the locations of the message queues. + */ + shm_toc_initialize_estimator(&e); + shm_toc_estimate_chunk(&e, sizeof(ParallelState)); + shm_toc_estimate_chunk(&e, (Size) queue_size); + + shm_toc_estimate_keys(&e, 1 + 1); + segsize = shm_toc_estimate(&e); + + /* Create the shared memory segment and establish a table of contents. */ + seg = dsm_create(shm_toc_estimate(&e), 0); + toc = shm_toc_create(PG_LOGICAL_APPLY_SHM_MAGIC, dsm_segment_address(seg), + segsize); + + /* Set up the header region. */ + pst = shm_toc_allocate(toc, sizeof(ParallelState)); + SpinLockInit(&pst->mutex); + pst->state = APPLY_BGWORKER_BUSY; + pst->stream_xid = stream_xid; + pst->n = list_length(ApplyWorkersList) + 1; + + shm_toc_insert(toc, toc_key++, pst); + + /* Set up one message queue per worker, plus one. */ + mq = shm_mq_create(shm_toc_allocate(toc, (Size) queue_size), + (Size) queue_size); + shm_toc_insert(toc, toc_key++, mq); + shm_mq_set_sender(mq, MyProc); + + /* Attach the queues. */ + wstate->mq_handle = shm_mq_attach(mq, seg, NULL); + + /* Return results to caller. */ + wstate->dsm_seg = seg; + wstate->pstate = pst; +} + +/* + * Start apply worker background worker process and allocat shared memory for + * it. + */ +static WorkerState * +apply_bgworker_setup(void) +{ + MemoryContext oldcontext; + bool launched; + WorkerState *wstate; + + elog(LOG, "setting up apply worker #%u", list_length(ApplyWorkersList) + 1); + + oldcontext = MemoryContextSwitchTo(ApplyContext); + + wstate = (WorkerState *) palloc0(sizeof(WorkerState)); + + /* Setup shared memory */ + apply_bgworker_setup_dsm(wstate); + + launched = logicalrep_worker_launch(MyLogicalRepWorker->dbid, + MySubscription->oid, + MySubscription->name, + MyLogicalRepWorker->userid, + InvalidOid, + dsm_segment_handle(wstate->dsm_seg)); + + if (launched) + { + /* Wait for worker to become ready. */ + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_ATTACHED); + + ApplyWorkersList = lappend(ApplyWorkersList, wstate); + } + else + { + pfree(wstate); + wstate = NULL; + } + + MemoryContextSwitchTo(oldcontext); + + return wstate; +} + +/* + * Send the data to the specified apply bgworker via shared-memory queue. + */ +static void +apply_bgworker_send_data(WorkerState *wstate, Size nbytes, const void *data) +{ + shm_mq_result result; + + result = shm_mq_send(wstate->mq_handle, nbytes, data, false, true); + + if (result != SHM_MQ_SUCCESS) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("could not send tuples to shared-memory queue"))); +} + +/* + * Wait until the state of apply background worker reach the 'wait_for_state' + */ +static void +apply_bgworker_wait_for(WorkerState *wstate, char wait_for_state) +{ + for (;;) + { + char status; + + /* If the worker is ready, we have succeeded. */ + SpinLockAcquire(&wstate->pstate->mutex); + status = wstate->pstate->state; + SpinLockRelease(&wstate->pstate->mutex); + + if (status == wait_for_state) + break; + + /* If any workers (or the postmaster) have died, we have failed. */ + if (status == APPLY_BGWORKER_EXIT) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("Background worker %u failed to apply transaction %u", + wstate->pstate->n, wstate->pstate->stream_xid))); + + /* Wait to be signalled. */ + WaitLatch(MyLatch, WL_LATCH_SET | WL_EXIT_ON_PM_DEATH, 0, + WAIT_EVENT_LOGICAL_APPLY_WORKER_READY); + + /* Reset the latch so we don't spin. */ + ResetLatch(MyLatch); + + /* An interrupt may have occurred while we were waiting. */ + CHECK_FOR_INTERRUPTS(); + } +} + +/* + * Check the status of workers and report an error if any apply background + * worker has exited unexpectedly. + * + * Exit if any relation is not in the READY state and if any worker is handling + * the streaming transaction at the same time. Because for streaming + * transactions that is being applied in apply bgworker, we cannot decide whether to + * apply the change for a relation that is not in the READY state (see + * should_apply_changes_for_rel) as we won't know remote_final_lsn by that + * time. + */ +static void +apply_bgworker_check_status(void) +{ + ListCell *lc; + + if (am_apply_bgworker() || MySubscription->stream != SUBSTREAM_APPLY) + return; + + foreach(lc, ApplyWorkersList) + { + WorkerState *wstate = (WorkerState *) lfirst(lc); + + /* + * We don't lock here as in the worst case we will just detect the + * failure of worker a bit later. + */ + if (wstate->pstate->state == APPLY_BGWORKER_EXIT) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("Background worker %u exited unexpectedly", + wstate->pstate->n))); + } + + if (list_length(ApplyWorkersIdleList) != list_length(ApplyWorkersList) && + !AllTablesyncsReady()) + { + ereport(LOG, + (errmsg("logical replication apply workers for subscription \"%s\" will restart", + MySubscription->name), + errdetail("Cannot start table synchronization while bgworkers are " + "handling streamed replication transaction"))); + + proc_exit(0); + } +} + +/* Set the state of apply background worker */ +static void +apply_bgworker_set_state(char state) +{ + if (!am_apply_bgworker()) + return; + + elog(LOG, "[Apply BGW #%u] set state to %c", + MyParallelState->n, state); + + SpinLockAcquire(&MyParallelState->mutex); + MyParallelState->state = state; + SpinLockRelease(&MyParallelState->mutex); +} diff --git a/src/backend/utils/activity/wait_event.c b/src/backend/utils/activity/wait_event.c index 87c15b9..24a6fac 100644 --- a/src/backend/utils/activity/wait_event.c +++ b/src/backend/utils/activity/wait_event.c @@ -388,6 +388,9 @@ pgstat_get_wait_ipc(WaitEventIPC w) case WAIT_EVENT_HASH_GROW_BUCKETS_REINSERT: event_name = "HashGrowBucketsReinsert"; break; + case WAIT_EVENT_LOGICAL_APPLY_WORKER_READY: + event_name = "LogicalApplyWorkerReady"; + break; case WAIT_EVENT_LOGICAL_SYNC_DATA: event_name = "LogicalSyncData"; break; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 786d592..a476885 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -4449,7 +4449,7 @@ getSubscriptions(Archive *fout) if (fout->remoteVersion >= 140000) appendPQExpBufferStr(query, " s.substream,\n"); else - appendPQExpBufferStr(query, " false AS substream,\n"); + appendPQExpBufferStr(query, " 'f' AS substream,\n"); if (fout->remoteVersion >= 150000) appendPQExpBufferStr(query, @@ -4579,8 +4579,10 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo) if (strcmp(subinfo->subbinary, "t") == 0) appendPQExpBufferStr(query, ", binary = true"); - if (strcmp(subinfo->substream, "f") != 0) + if (strcmp(subinfo->substream, "o") == 0) appendPQExpBufferStr(query, ", streaming = on"); + else if (strcmp(subinfo->substream, "a") == 0) + appendPQExpBufferStr(query, ", streaming = apply"); if (strcmp(subinfo->subtwophasestate, two_phase_disabled) != 0) appendPQExpBufferStr(query, ", two_phase = on"); diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h index d1260f5..b5f270c 100644 --- a/src/include/catalog/pg_subscription.h +++ b/src/include/catalog/pg_subscription.h @@ -68,7 +68,7 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW bool subbinary; /* True if the subscription wants the * publisher to send data in binary */ - bool substream; /* Stream in-progress transactions. */ + char substream; /* Stream in-progress transactions. */ char subtwophasestate; /* Stream two-phase transactions */ @@ -109,7 +109,7 @@ typedef struct Subscription bool enabled; /* Indicates if the subscription is enabled */ bool binary; /* Indicates if the subscription wants data in * binary format */ - bool stream; /* Allow streaming in-progress transactions. */ + char stream; /* Allow streaming in-progress transactions. */ char twophasestate; /* Allow streaming two-phase transactions */ bool disableonerr; /* Indicates if the subscription should be * automatically disabled if a worker error @@ -120,6 +120,18 @@ typedef struct Subscription List *publications; /* List of publication names to subscribe to */ } Subscription; +/* Disallow streaming in-progress transactions */ +#define SUBSTREAM_OFF 'f' + +/* + * Streaming transactions are written to a temporary file and applied only + * after the transaction is committed on upstream. + */ +#define SUBSTREAM_ON 'o' + +/* Streaming transactions are appied immediately via a background worker */ +#define SUBSTREAM_APPLY 'a' + extern Subscription *GetSubscription(Oid subid, bool missing_ok); extern void FreeSubscription(Subscription *sub); extern void DisableSubscription(Oid subid); diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h index a771ab8..1eb8e12 100644 --- a/src/include/replication/logicalproto.h +++ b/src/include/replication/logicalproto.h @@ -243,8 +243,8 @@ extern TransactionId logicalrep_read_stream_start(StringInfo in, extern void logicalrep_write_stream_stop(StringInfo out); extern void logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn, XLogRecPtr commit_lsn); -extern TransactionId logicalrep_read_stream_commit(StringInfo out, - LogicalRepCommitData *commit_data); +extern void logicalrep_read_stream_commit(StringInfo out, + LogicalRepCommitData *commit_data); extern void logicalrep_write_stream_abort(StringInfo out, TransactionId xid, TransactionId subxid); extern void logicalrep_read_stream_abort(StringInfo in, TransactionId *xid, diff --git a/src/include/replication/logicalworker.h b/src/include/replication/logicalworker.h index cd1b6e8..6a1af7f 100644 --- a/src/include/replication/logicalworker.h +++ b/src/include/replication/logicalworker.h @@ -13,6 +13,7 @@ #define LOGICALWORKER_H extern void ApplyWorkerMain(Datum main_arg); +extern void ApplyBgworkerMain(Datum main_arg); extern bool IsLogicalWorker(void); diff --git a/src/include/replication/origin.h b/src/include/replication/origin.h index 14d5c49..1ed51bc 100644 --- a/src/include/replication/origin.h +++ b/src/include/replication/origin.h @@ -53,7 +53,7 @@ extern XLogRecPtr replorigin_get_progress(RepOriginId node, bool flush); extern void replorigin_session_advance(XLogRecPtr remote_commit, XLogRecPtr local_commit); -extern void replorigin_session_setup(RepOriginId node); +extern void replorigin_session_setup(RepOriginId node, bool acquire); extern void replorigin_session_reset(void); extern XLogRecPtr replorigin_session_get_progress(bool flush); diff --git a/src/include/replication/worker_internal.h b/src/include/replication/worker_internal.h index 4485d4e..4d11a13 100644 --- a/src/include/replication/worker_internal.h +++ b/src/include/replication/worker_internal.h @@ -60,6 +60,8 @@ typedef struct LogicalRepWorker */ FileSet *stream_fileset; + bool subworker; + /* Stats. */ XLogRecPtr last_lsn; TimestampTz last_send_time; @@ -84,8 +86,9 @@ extern void logicalrep_worker_attach(int slot); extern LogicalRepWorker *logicalrep_worker_find(Oid subid, Oid relid, bool only_running); extern List *logicalrep_workers_find(Oid subid, bool only_running); -extern void logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, - Oid userid, Oid relid); +extern bool logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, + Oid userid, Oid relid, + dsm_handle subworker_dsm); extern void logicalrep_worker_stop(Oid subid, Oid relid); extern void logicalrep_worker_wakeup(Oid subid, Oid relid); extern void logicalrep_worker_wakeup_ptr(LogicalRepWorker *worker); diff --git a/src/include/utils/wait_event.h b/src/include/utils/wait_event.h index b578e2e..d5081d9 100644 --- a/src/include/utils/wait_event.h +++ b/src/include/utils/wait_event.h @@ -105,6 +105,7 @@ typedef enum WAIT_EVENT_HASH_GROW_BUCKETS_ALLOCATE, WAIT_EVENT_HASH_GROW_BUCKETS_ELECT, WAIT_EVENT_HASH_GROW_BUCKETS_REINSERT, + WAIT_EVENT_LOGICAL_APPLY_WORKER_READY, WAIT_EVENT_LOGICAL_SYNC_DATA, WAIT_EVENT_LOGICAL_SYNC_STATE_CHANGE, WAIT_EVENT_MQ_INTERNAL, diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out index 7fcfad1..e08e4d4 100644 --- a/src/test/regress/expected/subscription.out +++ b/src/test/regress/expected/subscription.out @@ -197,7 +197,7 @@ ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); DROP SUBSCRIPTION regress_testsub; -- fail - streaming must be boolean CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = foo); -ERROR: streaming requires a Boolean value +ERROR: streaming requires a Boolean value or "apply" -- now it works CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = true); WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables @@ -205,7 +205,7 @@ WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION .. List of subscriptions Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable on error | Synchronous commit | Conninfo | Skip LSN -----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------+---------- - regress_testsub | regress_subscription_user | f | {testpub} | f | t | d | f | off | dbname=regress_doesnotexist | 0/0 + regress_testsub | regress_subscription_user | f | {testpub} | f | o | d | f | off | dbname=regress_doesnotexist | 0/0 (1 row) ALTER SUBSCRIPTION regress_testsub SET (streaming = false); @@ -299,7 +299,7 @@ ALTER SUBSCRIPTION regress_testsub SET (streaming = true); List of subscriptions Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable on error | Synchronous commit | Conninfo | Skip LSN -----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------+---------- - regress_testsub | regress_subscription_user | f | {testpub} | f | t | p | f | off | dbname=regress_doesnotexist | 0/0 + regress_testsub | regress_subscription_user | f | {testpub} | f | o | p | f | off | dbname=regress_doesnotexist | 0/0 (1 row) ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); @@ -311,7 +311,7 @@ WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION .. List of subscriptions Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable on error | Synchronous commit | Conninfo | Skip LSN -----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------+---------- - regress_testsub | regress_subscription_user | f | {testpub} | f | t | p | f | off | dbname=regress_doesnotexist | 0/0 + regress_testsub | regress_subscription_user | f | {testpub} | f | o | p | f | off | dbname=regress_doesnotexist | 0/0 (1 row) ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 87ee7bf..9c036d1 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -2918,11 +2918,13 @@ WordEntryPosVector WordEntryPosVector1 WorkTableScan WorkTableScanState +WorkerEntry WorkerInfo WorkerInfoData WorkerInstrumentation WorkerJobDumpPtrType WorkerJobRestorePtrType +WorkerState Working_State WriteBufPtrType WriteBytePtrType -- 2.7.2.windows.1 ^ permalink raw reply [nested|flat] 66+ messages in thread
* Re: Perform streaming logical transactions by background workers and parallel apply @ 2022-05-05 05:45 Peter Smith <[email protected]> parent: [email protected] <[email protected]> 1 sibling, 1 reply; 66+ messages in thread From: Peter Smith @ 2022-05-05 05:45 UTC (permalink / raw) To: [email protected] <[email protected]>; +Cc: Amit Kapila <[email protected]>; [email protected] <[email protected]>; PostgreSQL Hackers <[email protected]> Here are my review comments for v5-0001. I will take a look at the v5-0002 (TAP) patch another time. ====== 1. Commit message The message still refers to "apply background". Should that say "apply background worker"? Other parts just call this the "worker". Personally, I think it might be better to coin some new term for this thing (e.g. "apply-bgworker" or something like that of your choosing) so then you can just concisely *always* refer to that everywhere without any ambiguity. e.g same applies to every comment and every message in this patch. They should all use identical terminology (e.g. "apply-bgworker"). ~~~ 2. Commit message "We also need to allow stream_stop to complete by the apply background to finish it to..." Wording: ??? ~~~ 3. Commit message This patch also extends the subscription streaming option so that user can control whether apply the streaming transaction in a apply background or spill the change to disk. Wording: "user" -> "the user" Typo: "whether apply" -> "whether to apply" Typo: "a apply" -> "an apply" ~~~ 4. Commit message User can set the streaming option to 'on/off', 'apply'. For now, 'apply' means the streaming will be applied via a apply background if available. 'on' means the streaming transaction will be spilled to disk. I think "apply" might not be the best choice of values for this meaning, but I think Hou-san already said [1] that this was being reconsidered. ~~~ 5. doc/src/sgml/catalogs.sgml - formatting @@ -7863,11 +7863,15 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l <row> <entry role="catalog_table_entry"><para role="column_definition"> - <structfield>substream</structfield> <type>bool</type> + <structfield>substream</structfield> <type>char</type> </para> <para> - If true, the subscription will allow streaming of in-progress - transactions + Controls how to handle the streaming of in-progress transactions. + <literal>f</literal> = disallow streaming of in-progress transactions + <literal>o</literal> = spill the changes of in-progress transactions to + disk and apply at once after the transaction is committed on the + publisher. + <literal>a</literal> = apply changes directly using a background worker </para></entry> </row> Needs to be consistent with other value lists on this page. 5a. The first sentence to end with ":" 5b. List items to end with "," ~~~ 6. doc/src/sgml/ref/create_subscription.sgml + <para> + If set to <literal>apply</literal> incoming + changes are directly applied via one of the background worker, if + available. If no background worker is free to handle streaming + transaction then the changes are written to a file and applied after + the transaction is committed. Note that if error happen when applying + changes in background worker, it might not report the finish LSN of + the remote transaction in server log. </para> 6a. Typo: "one of the background worker," -> "one of the background workers," 6b. Wording BEFORE Note that if error happen when applying changes in background worker, it might not report the finish LSN of the remote transaction in server log. SUGGESTION Note that if an error happens when applying changes in a background worker, it might not report the finish LSN of the remote transaction in the server log. ~~~ 7. src/backend/commands/subscriptioncmds.c - defGetStreamingMode +static char +defGetStreamingMode(DefElem *def) +{ + /* + * If no parameter given, assume "true" is meant. + */ + if (def->arg == NULL) + return SUBSTREAM_ON; But is that right? IIUC all the docs said that the default is OFF. ~~~ 8. src/backend/commands/subscriptioncmds.c - defGetStreamingMode + /* + * The set of strings accepted here should match up with the + * grammar's opt_boolean_or_string production. + */ + if (pg_strcasecmp(sval, "true") == 0 || + pg_strcasecmp(sval, "on") == 0) + return SUBSTREAM_ON; + if (pg_strcasecmp(sval, "apply") == 0) + return SUBSTREAM_APPLY; + if (pg_strcasecmp(sval, "false") == 0 || + pg_strcasecmp(sval, "off") == 0) + return SUBSTREAM_OFF; Perhaps should re-order these OFF/ON/APPLY to be consistent with the T_Integer case above here. ~~~ 9. src/backend/replication/logical/launcher.c - logicalrep_worker_launch The "start new apply background worker ..." function comment feels a bit misleading now that seems what you are calling this new kind of worker. E.g. this is also called to start the sync worker. And also for the apply worker (which we are not really calling a "background worker" in other places). This comment is the same as [PSv4] #19. ~~~ 10. src/backend/replication/logical/launcher.c - logicalrep_worker_launch @@ -275,6 +280,9 @@ logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, Oid userid, int nsyncworkers; TimestampTz now; + /* We don't support table sync in subworker */ + Assert(!((subworker_dsm != DSM_HANDLE_INVALID) && OidIsValid(relid))); I think you should declare a new variable like: bool is_subworker = subworker_dsm != DSM_HANDLE_INVALID; Then this Assert can be simplified, and also you can re-use the 'is_subworker' later multiple times in this same function to simplify lots of other code also. ~~~ 11. src/backend/replication/logical/launcher.c - logicalrep_worker_stop_internal +/* + * Workhorse for logicalrep_worker_stop() and logicalrep_worker_detach(). Stop + * the worker and wait for wait for it to die. + */ +static void +logicalrep_worker_stop_internal(LogicalRepWorker *worker) Typo: "wait for" is repeated 2x. ~~~ 12. src/backend/replication/logical/origin.c - replorigin_session_setup @@ -1110,7 +1110,11 @@ replorigin_session_setup(RepOriginId node) if (curstate->roident != node) continue; - else if (curstate->acquired_by != 0) + /* + * We allow the apply worker to get the slot which is acquired by its + * leader process. + */ + else if (curstate->acquired_by != 0 && acquire) I still feel this is overly-cofusing. Shouldn't comment say "Allow the apply bgworker to get the slot...". Also the parameter name 'acquire' is hard to reconcile with the comment. E.g. I feel all this would be easier to understand if the param was was refactored with a name like 'bgworker' and the code was changed to: else if (curstate->acquired_by != 0 && !bgworker) Of course, the value true/false would need to be flipped on calls too. This is the same as my previous comment [PSv4] #26. ~~~ 13. src/backend/replication/logical/proto.c @@ -1138,14 +1138,11 @@ logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn, /* * Read STREAM COMMIT from the output stream. */ -TransactionId +void logicalrep_read_stream_commit(StringInfo in, LogicalRepCommitData *commit_data) { - TransactionId xid; uint8 flags; - xid = pq_getmsgint(in, 4); - /* read flags (unused for now) */ flags = pq_getmsgbyte(in); There is something incompatible with the read/write functions here. The write writes the txid before the flags, but the read_commit does not read it at all – if only reads the flags (???) if this is really correct then I think there need to be some comments to explain WHY it is correct. NOTE: See also review comment 28 where I proposed another way to write this code. ~~~ 14. src/backend/replication/logical/worker.c - comment The whole comment is similar to the commit message so any changes there should be made here also. ~~~ 15. src/backend/replication/logical/worker.c - ParallelState +/* + * Shared information among apply workers. + */ +typedef struct ParallelState It looks like there is already another typedef called "ParallelState" because it is already in the typedefs.list. Maybe this name should be changed or maybe make it static or something? ~~~ 16. src/backend/replication/logical/worker.c - defines +/* + * States for apply background worker. + */ +#define APPLY_BGWORKER_ATTACHED 'a' +#define APPLY_BGWORKER_READY 'r' +#define APPLY_BGWORKER_BUSY 'b' +#define APPLY_BGWORKER_FINISHED 'f' +#define APPLY_BGWORKER_EXIT 'e' Those char states all look independent. So wouldn’t this be represented better as an enum to reinforce that fact? ~~~ 17. src/backend/replication/logical/worker.c - functions +/* Worker setup and interactions */ +static WorkerState *apply_bgworker_setup(void); +static WorkerState *find_or_start_apply_bgworker(TransactionId xid, + bool start); Maybe rename to apply_bgworker_find_or_start() to match the pattern of the others? ~~~ 18. src/backend/replication/logical/worker.c - macros +#define am_apply_bgworker() (MyLogicalRepWorker->subworker) +#define applying_changes_in_bgworker() (in_streamed_transaction && stream_apply_worker != NULL) 18a. Somehow I felt these are not in the best place. - Maybe am_apply_bgworker() should be in worker_internal.h? - Maybe the applying_changes_in_bgworker() should be nearby the stream_apply_worker declaration 18b. Maybe applying_changes_in_bgworker should be renamed to something else to match the pattern of the others (e.g. "apply_bgworker_active" or something) ~~~ 19. src/backend/replication/logical/worker.c - handle_streamed_transaction + /* + * If we decided to apply the changes of this transaction in a apply + * background worker, pass the data to the worker. + */ Typo: "in a apply" -> "in an apply" ~~~ 20. src/backend/replication/logical/worker.c - handle_streamed_transaction + /* + * XXX The publisher side doesn't always send relation update message + * after the streaming transaction, so update the relation in main + * apply worker here. + */ Wording: "doesn't always send relation update message" -> "doesn't always send relation update messages" ?? ~~~ 21. src/backend/replication/logical/worker.c - apply_handle_commit_prepared + apply_bgworker_set_state(APPLY_BGWORKER_FINISHED); It seems somewhat confusing to see calls to apply_bgworker_set_state() when we may or may not even be an apply bgworker. I know it adds more code, but I somehow feel it is more readable if all these calls were changed to look below. Please consider it. SUGGESTION if (am_bgworker()) apply_bgworker_set_state(XXX); Then you can also change the apply_bgworker_set_state to Assert(am_apply_bgworker()); ~~~ 22. src/backend/replication/logical/worker.c - find_or_start_apply_bgworker + + if (!start && ApplyWorkersHash == NULL) + return NULL; + IIUC maybe this extra check is not really necessary. I see no harm to create the HashTable even if was called in this state. If the 'start' flag is false then nothing is going to be found anyway, so it will return NULL. e.g. Might as well make the code a few lines shorter/simpler by removing this check. ~~~ 23. src/backend/replication/logical/worker.c - apply_bgworker_free +/* + * Add the worker to the freelist and remove the entry from hash table. + */ +static void +apply_bgworker_free(WorkerState *wstate) +{ + bool found; + MemoryContext oldctx; + TransactionId xid = wstate->pstate->stream_xid; If you are not going to check the value of 'found' then why bother to pass this param at all? Can't you just pass NULL? ~~~ 24. src/backend/replication/logical/worker.c - apply_bgworker_free Should there be an Assert that the bgworker state really was FINISHED? I think I asked this already [PSv4] #48. ~~~ 24. src/backend/replication/logical/worker.c - apply_handle_stream_start @@ -1088,24 +1416,71 @@ apply_handle_stream_prepare(StringInfo s) logicalrep_read_stream_prepare(s, &prepare_data); set_apply_error_context_xact(prepare_data.xid, prepare_data.prepare_lsn); - elog(DEBUG1, "received prepare for streamed transaction %u", prepare_data.xid); + /* + * If we are in a bgworker, just prepare the transaction. + */ + if (am_apply_bgworker()) Don’t need to say "If we are..." because the am_apply_worker() condition makes it clear this is true. ~~~ 25. src/backend/replication/logical/worker.c - apply_handle_stream_start - if (MyLogicalRepWorker->stream_fileset == NULL) + stream_apply_worker = find_or_start_apply_bgworker(stream_xid, first_segment); + + if (applying_changes_in_bgworker()) { IIUC this condition seems overkill. I think you can just say if (stream_apply_worker) ~~~ 26. src/backend/replication/logical/worker.c - apply_handle_stream_abort + if (found) + { + elog(LOG, "rolled back to savepoint %s", spname); + RollbackToSavepoint(spname); + CommitTransactionCommand(); + subxactlist = list_truncate(subxactlist, i + 1); + } Should that elog use the "[Apply BGW #%u]" format like the others for BGW? ~~~ 27. src/backend/replication/logical/worker.c - apply_handle_stream_abort Should this function be setting stream_apply_worker = NULL somewhere when all is done? ~~~ 28. src/backend/replication/logical/worker.c - apply_handle_stream_commit +/* + * Handle STREAM COMMIT message. + */ +static void +apply_handle_stream_commit(StringInfo s) +{ + LogicalRepCommitData commit_data; + TransactionId xid; + + if (in_streamed_transaction) + ereport(ERROR, + (errcode(ERRCODE_PROTOCOL_VIOLATION), + errmsg_internal("STREAM COMMIT message without STREAM STOP"))); + + xid = pq_getmsgint(s, 4); + logicalrep_read_stream_commit(s, &commit_data); + set_apply_error_context_xact(xid, commit_data.commit_lsn); There is something a bit odd about this code. I think the logicalrep_read_stream_commit() should take another param and the Txid be extracted/read only INSIDE that logicalrep_read_stream_commit function. See also review comment #13. ~~~ 29. src/backend/replication/logical/worker.c - apply_handle_stream_commit I am unsure, but should something be setting the stream_apply_worker = NULL somewhere when all is done? ~~~ 30. src/backend/replication/logical/worker.c - LogicalApplyBgwLoop 30a. + if (shmq_res != SHM_MQ_SUCCESS) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("lost connection to the main apply worker"))); 30b. + default: + elog(ERROR, "unexpected message"); + break; Should both those error messages have the "[Apply BGW #%u]" prefix like the other BGW messages? ~~~ 31. src/backend/replication/logical/worker.c - ApplyBgwShutdown +/* + * Set the failed flag so that the main apply worker can realize we have + * shutdown. + */ +static void +ApplyBgwShutdown(int code, Datum arg) The comment does not seem to be in sync with the code. E.g. Wording: "failed flag" -> "exit state" ?? ~~~ 32. src/backend/replication/logical/worker.c - ApplyBgwShutdown +/* + * Set the failed flag so that the main apply worker can realize we have + * shutdown. + */ +static void +ApplyBgwShutdown(int code, Datum arg) If the 'code' param is deliberately unused it might be better to say so in the comment... ~~~ 33. src/backend/replication/logical/worker.c - LogicalApplyBgwMain 33a. + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("unable to map dynamic shared memory segment"))); 33b. + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("bad magic number in dynamic shared memory segment"))); + 33c. + ereport(LOG, + (errmsg("logical replication apply worker for subscription %u will not " + "start because the subscription was removed during startup", + MyLogicalRepWorker->subid))); Should all these messages have "[Apply BGW ?]" prefix even though they are not yet attached? ~~~ 34. src/backend/replication/logical/worker.c - setup_dsm + * We need one key to register the location of the header, and we need + * nworkers keys to track the locations of the message queues. + */ This comment about 'nworkers' seems stale because that variable no longer exists. ~~~ 35. src/backend/replication/logical/worker.c - apply_bgworker_setup +/* + * Start apply worker background worker process and allocat shared memory for + * it. + */ +static WorkerState * +apply_bgworker_setup(void) typo: "allocat" -> "allocate" ~~~ 36. src/backend/replication/logical/worker.c - apply_bgworker_setup + elog(LOG, "setting up apply worker #%u", list_length(ApplyWorkersList) + 1) Should this message have the standard "[Apply BGW %u]" pattern? ~~~ 37. src/backend/replication/logical/worker.c - apply_bgworker_setup + if (launched) + { + /* Wait for worker to become ready. */ + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_ATTACHED); + + ApplyWorkersList = lappend(ApplyWorkersList, wstate); + } Since there is a state APPLY_BGWORKER_READY I think either this comment is wrong or this passed parameter ATTACHED must be wrong. ~~~ 38. src/backend/replication/logical/worker.c - apply_bgworker_send_data + if (result != SHM_MQ_SUCCESS) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("could not send tuples to shared-memory queue"))); +} Wording: Is it right to ocall these "tuples" or better just say "data"? I am not sure. Already asked this in [PSv4] #68 ~~~ 39. src/backend/replication/logical/worker.c - apply_bgworker_wait_for +/* + * Wait until the state of apply background worker reach the 'wait_for_state' + */ +static void +apply_bgworker_wait_for(WorkerState *wstate, char wait_for_state) typo: "reach" -> "reaches" ~~~ 40. src/backend/replication/logical/worker.c - apply_bgworker_wait_for + /* If the worker is ready, we have succeeded. */ + SpinLockAcquire(&wstate->pstate->mutex); + status = wstate->pstate->state; + SpinLockRelease(&wstate->pstate->mutex); + + if (status == wait_for_state) + break; 40a. What does this mention "ready". This function might be waiting for a different state to that. 40b. Anyway, I think this comment should be a few lines lower, above the if (status == wait_for_state) ~~~ 41. src/backend/replication/logical/worker.c - apply_bgworker_wait_for + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("Background worker %u failed to apply transaction %u", + wstate->pstate->n, wstate->pstate->stream_xid))); Should this message have the standard "[Apply BGW %u]" pattern? ~~~ 42. src/backend/replication/logical/worker.c - check_workers_status + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("Background worker %u exited unexpectedly", + wstate->pstate->n))); Should this message have the standard "[Apply BGW %u]" pattern? Or if this is just from Apply worker maybe it should be clearer like "Apply worker detected apply bgworker %u exited unexpectedly". ~~~ 43. src/backend/replication/logical/worker.c - check_workers_status + ereport(LOG, + (errmsg("logical replication apply workers for subscription \"%s\" will restart", + MySubscription->name), + errdetail("Cannot start table synchronization while bgworkers are " + "handling streamed replication transaction"))); I am not sure, but isn't the message backwards? e.g. Should it say more like: "Cannot handle streamed transactions using bgworkers while table synchronization is still in progress". ~~~ 44. src/backend/replication/logical/worker.c - apply_bgworker_set_state + elog(LOG, "[Apply BGW #%u] set state to %c", + MyParallelState->n, state); The line wrapping seemed overkill here. ~~~ 45. src/backend/utils/activity/wait_event.c @@ -388,6 +388,9 @@ pgstat_get_wait_ipc(WaitEventIPC w) case WAIT_EVENT_HASH_GROW_BUCKETS_REINSERT: event_name = "HashGrowBucketsReinsert"; break; + case WAIT_EVENT_LOGICAL_APPLY_WORKER_READY: + event_name = "LogicalApplyWorkerReady"; + break; I am not sure this is the best name for this event since the only place it is used (in apply_bgworker_wait_for) is not only waiting for READY state. Maybe a name like WAIT_EVENT_LOGICAL_APPLY_BGWORKER or WAIT_EVENT_LOGICAL_APPLY_WORKER_SYNC would be more appropriate? Need to change the wait_event.h also. ~~~ 46. src/include/catalog/pg_subscription.h +/* Disallow streaming in-progress transactions */ +#define SUBSTREAM_OFF 'f' + +/* + * Streaming transactions are written to a temporary file and applied only + * after the transaction is committed on upstream. + */ +#define SUBSTREAM_ON 'o' + +/* Streaming transactions are appied immediately via a background worker */ +#define SUBSTREAM_APPLY 'a' 46a. There is not really any overarching comment that associates these #defines back to the new 'stream' field so you are just supposed to guess that's what they are for? 46b. I also feel that using 'o' for ON is not consistent with the 'f' of OFF. IMO better to use 't/f' for true/false instead of 'o/f'. Also don't forget update docs, pg_dump.c etc. 46c. Typo: "appied" -> "applied" ~~~~ 47. src/test/regress/expected/subscription.out - missting test Missing some test cases for all new option values? E.g. Where is the test using streaming value is set to 'apply'. Same comment as [PSv4] #81 ------ [1] https://www.postgresql.org/message-id/OS0PR01MB5716E8D536552467EFB512EF94FC9%40OS0PR01MB5716.jpnprd0... [PSv4] https://www.postgresql.org/message-id/CAHut%2BPuqYP5eD5wcSCtk%3Da6KuMjat2UCzqyGoE7sieCaBsVskQ%40mail... Kind Regards, Peter Smith. Fujitsu Australia ^ permalink raw reply [nested|flat] 66+ messages in thread
* Re: Perform streaming logical transactions by background workers and parallel apply @ 2022-05-06 08:56 Peter Smith <[email protected]> parent: [email protected] <[email protected]> 0 siblings, 1 reply; 66+ messages in thread From: Peter Smith @ 2022-05-06 08:56 UTC (permalink / raw) To: [email protected] <[email protected]>; +Cc: [email protected] <[email protected]>; Amit Kapila <[email protected]>; PostgreSQL Hackers <[email protected]> On Fri, Apr 29, 2022 at 3:22 PM [email protected] <[email protected]> wrote: ... > Thanks for your patch. > > The patch modified streaming option in logical replication, it can be set to > 'on', 'off' and 'apply'. The new option 'apply' haven't been tested in the tap test. > Attach a patch which modified the subscription tap test to cover both 'on' and > 'apply' option. (The main patch is also attached to make cfbot happy.) > Here are my review comments for v5-0002 (TAP tests) Your changes followed a similar pattern of refactoring so most of my comments below is repeated for all the files. ====== 1. Commit message For the tap tests about streaming option in logical replication, test both 'on' and 'apply' option. SUGGESTION Change all TAP tests using the PUBLICATION "streaming" option, so they now test both 'on' and 'apply' values. ~~~ 2. src/test/subscription/t/015_stream.pl +sub test_streaming +{ I think the function should have a comment to say that its purpose is to encapsulate all the common (stream related) test steps so the same code can be run both for the streaming=on and streaming=apply cases. ~~~ 3. src/test/subscription/t/015_stream.pl + +# Test streaming mode on +# Test streaming mode apply These comments fell too small. IMO they should both be more prominent like: ################################ # Test using streaming mode 'on' ################################ ################################### # Test using streaming mode 'apply' ################################### ~~~ 4. src/test/subscription/t/015_stream.pl +# Test streaming mode apply +$node_publisher->safe_psql('postgres', "DELETE FROM test_tab WHERE (a > 2)"); $node_publisher->wait_for_catchup($appname); I think those 2 lines do not really belong after the "# Test streaming mode apply" comment. IIUC they are really just doing cleanup from the prior test part so I think they should a) be *above* this comment (and say "# cleanup the test data") or b) maybe it is best to put all the cleanup lines actually inside the 'test_streaming' function so that the last thing the function does is clean up after itself. option b seems tidier to me. ~~~ 5. src/test/subscription/t/016_stream_subxact.pl sub test_streaming should be commented. (same as comment #2) ~~~ 6. src/test/subscription/t/016_stream_subxact.pl The comments for the different streaming nodes should be more prominent. (same as comment #3) ~~~ 7. src/test/subscription/t/016_stream_subxact.pl +# Test streaming mode apply +$node_publisher->safe_psql('postgres', "DELETE FROM test_tab WHERE (a > 2)"); $node_publisher->wait_for_catchup($appname); These don't seem to belong here. They are clean up from the prior test. (same as comment #4) ~~~ 8. src/test/subscription/t/017_stream_ddl.pl sub test_streaming should be commented. (same as comment #2) ~~~ 9. src/test/subscription/t/017_stream_ddl.pl The comments for the different streaming nodes should be more prominent. (same as comment #3) ~~~ 10. src/test/subscription/t/017_stream_ddl.pl +# Test streaming mode apply $node_publisher->safe_psql( 'postgres', q{ -BEGIN; -INSERT INTO test_tab VALUES (2001, md5(2001::text), -2001, 2*2001); -ALTER TABLE test_tab ADD COLUMN e INT; -SAVEPOINT s1; -INSERT INTO test_tab VALUES (2002, md5(2002::text), -2002, 2*2002, -3*2002); -COMMIT; +DELETE FROM test_tab WHERE (a > 2); +ALTER TABLE test_tab DROP COLUMN c, DROP COLUMN d, DROP COLUMN e, DROP COLUMN f; }); $node_publisher->wait_for_catchup($appname); These don't seem to belong here. They are clean up from the prior test. (same as comment #4) ~~~ 11. .../t/018_stream_subxact_abort.pl sub test_streaming should be commented. (same as comment #2) ~~~ 12. .../t/018_stream_subxact_abort.pl The comments for the different streaming nodes should be more prominent. (same as comment #3) ~~~ 13. .../t/018_stream_subxact_abort.pl +# Test streaming mode apply +$node_publisher->safe_psql('postgres', "DELETE FROM test_tab WHERE (a > 2)"); $node_publisher->wait_for_catchup($appname); These don't seem to belong here. They are clean up from the prior test. (same as comment #4) ~~~ 14. .../t/019_stream_subxact_ddl_abort.pl sub test_streaming should be commented. (same as comment #2) ~~~ 15. .../t/019_stream_subxact_ddl_abort.pl The comments for the different streaming nodes should be more prominent. (same as comment #3) ~~~ 16. .../t/019_stream_subxact_ddl_abort.pl +test_streaming($node_publisher, $node_subscriber, $appname); + +# Test streaming mode apply $node_publisher->safe_psql( 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i); -ALTER TABLE test_tab ADD COLUMN c INT; -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(501,1000) s(i); -ALTER TABLE test_tab ADD COLUMN d INT; -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001,1500) s(i); -ALTER TABLE test_tab ADD COLUMN e INT; -SAVEPOINT s3; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(1501,2000) s(i); +DELETE FROM test_tab WHERE (a > 2); ALTER TABLE test_tab DROP COLUMN c; -ROLLBACK TO s1; -INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(501,1000) s(i); -COMMIT; }); - $node_publisher->wait_for_catchup($appname); These don't seem to belong here. They are clean up from the prior test. (same as comment #4) ~~~ 17. .../subscription/t/022_twophase_cascade. +# --------------------- +# 2PC + STREAMING TESTS +# --------------------- +sub test_streaming +{ I think maybe that 2PC comment should not have been moved. IMO it belongs in the main test body... ~~~ 18. .../subscription/t/022_twophase_cascade. sub test_streaming should be commented. (same as comment #2) ~~~ 19. .../subscription/t/022_twophase_cascade. +sub test_streaming +{ + my ($node_A, $node_B, $node_C, $appname_B, $appname_C, $streaming) = @_; If you called that '$streaming' param something more like '$streaming_mode' it would read better I think. ~~~ 20. .../subscription/t/023_twophase_stream.pl sub test_streaming should be commented. (same as comment #2) ~~~ 21. .../subscription/t/023_twophase_stream.pl The comments for the different streaming nodes should be more prominent. (same as comment #3) ~~~ 22. .../subscription/t/023_twophase_stream.pl +# Test streaming mode apply $node_publisher->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); - -# Then insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql('postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is in prepared state on subscriber -$result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber'); - -# 2PC transaction gets aborted -$node_publisher->safe_psql('postgres', "ROLLBACK PREPARED 'test_prepared_tab';"); - $node_publisher->wait_for_catchup($appname); These don't seem to belong here. They are clean up from the prior test. (same as comment #4) ------ Kind Regards, Peter Smith. Fujitsu Australia ^ permalink raw reply [nested|flat] 66+ messages in thread
* RE: Perform streaming logical transactions by background workers and parallel apply @ 2022-05-13 08:48 [email protected] <[email protected]> parent: Peter Smith <[email protected]> 0 siblings, 0 replies; 66+ messages in thread From: [email protected] @ 2022-05-13 08:48 UTC (permalink / raw) To: Peter Smith <[email protected]>; +Cc: Amit Kapila <[email protected]>; [email protected] <[email protected]>; PostgreSQL Hackers <[email protected]> On Thursday, May 5, 2022 1:46 PM Peter Smith <[email protected]> wrote: > Here are my review comments for v5-0001. > I will take a look at the v5-0002 (TAP) patch another time. Thanks for the comments ! > 4. Commit message > > User can set the streaming option to 'on/off', 'apply'. For now, > 'apply' means the streaming will be applied via a apply background if > available. 'on' means the streaming transaction will be spilled to > disk. > > > I think "apply" might not be the best choice of values for this > meaning, but I think Hou-san already said [1] that this was being > reconsidered. Yes, I am thinking over this along with some other related stuff[1] posted by Amit and sawada. Will change this in next version. [1] https://www.postgresql.org/message-id/flat/CAA4eK1%2B7D4qAQUQEE8zzQ0fGCqeBWd3rzTaY5N0jVs-VXFc_Xw%40m... > 7. src/backend/commands/subscriptioncmds.c - defGetStreamingMode > > +static char > +defGetStreamingMode(DefElem *def) > +{ > + /* > + * If no parameter given, assume "true" is meant. > + */ > + if (def->arg == NULL) > + return SUBSTREAM_ON; > > But is that right? IIUC all the docs said that the default is OFF. I think it's right. "arg == NULL" means user specify the streaming option without the value. Like CREATE SUBSCRIPTION xxx WITH(streaming). The value should be 'on' in this case. > 12. src/backend/replication/logical/origin.c - replorigin_session_setup > > @@ -1110,7 +1110,11 @@ replorigin_session_setup(RepOriginId node) > if (curstate->roident != node) > continue; > > - else if (curstate->acquired_by != 0) > + /* > + * We allow the apply worker to get the slot which is acquired by its > + * leader process. > + */ > + else if (curstate->acquired_by != 0 && acquire) > > I still feel this is overly-cofusing. Shouldn't comment say "Allow the > apply bgworker to get the slot...". > > Also the parameter name 'acquire' is hard to reconcile with the > comment. E.g. I feel all this would be easier to understand if the > param was was refactored with a name like 'bgworker' and the code was > changed to: > else if (curstate->acquired_by != 0 && !bgworker) > > Of course, the value true/false would need to be flipped on calls too. > This is the same as my previous comment [PSv4] #26. I feel it's not good idea to mention bgworker in origin.c. I have remove this comment and add some other comments in worker.c. > 26. src/backend/replication/logical/worker.c - apply_handle_stream_abort > > + if (found) > + { > + elog(LOG, "rolled back to savepoint %s", spname); > + RollbackToSavepoint(spname); > + CommitTransactionCommand(); > + subxactlist = list_truncate(subxactlist, i + 1); > + } > > Should that elog use the "[Apply BGW #%u]" format like the others for BGW? I feel the "[Apply BGW #%u]" is a bit hacky and some of them comes from the old patchset. I will recheck these logs and adjust them and change some log level in next version. > 27. src/backend/replication/logical/worker.c - apply_handle_stream_abort > > Should this function be setting stream_apply_worker = NULL somewhere > when all is done? > 29. src/backend/replication/logical/worker.c - apply_handle_stream_commit > > I am unsure, but should something be setting the stream_apply_worker = > NULL somewhere when all is done? I think the worker already be set to NULL in apply_handle_stream_stop. > 32. src/backend/replication/logical/worker.c - ApplyBgwShutdown > > +/* > + * Set the failed flag so that the main apply worker can realize we have > + * shutdown. > + */ > +static void > +ApplyBgwShutdown(int code, Datum arg) > > If the 'code' param is deliberately unused it might be better to say > so in the comment... Not sure about this. After searching the codes, I think most of the callback functions doesn't use and add comments for the 'code' param. > 45. src/backend/utils/activity/wait_event.c > > @@ -388,6 +388,9 @@ pgstat_get_wait_ipc(WaitEventIPC w) > case WAIT_EVENT_HASH_GROW_BUCKETS_REINSERT: > event_name = "HashGrowBucketsReinsert"; > break; > + case WAIT_EVENT_LOGICAL_APPLY_WORKER_READY: > + event_name = "LogicalApplyWorkerReady"; > + break; > > I am not sure this is the best name for this event since the only > place it is used (in apply_bgworker_wait_for) is not only waiting for > READY state. Maybe a name like WAIT_EVENT_LOGICAL_APPLY_BGWORKER or > WAIT_EVENT_LOGICAL_APPLY_WORKER_SYNC would be more appropriate? Need > to change the wait_event.h also. I noticed a similar named "WAIT_EVENT_LOGICAL_SYNC_STATE_CHANGE", so I changed this to WAIT_EVENT_LOGICAL_APPLY_WORKER_STATE_CHANGE. > 47. src/test/regress/expected/subscription.out - missting test > > Missing some test cases for all new option values? E.g. Where is the > test using streaming value is set to 'apply'. Same comment as [PSv4] > #81 The new option is tested in the second patch posted by Shi yu. I addressed other comments from Peter and the 2PC related comment from Shi. Here is the version patch. Best regards, Hou zj Attachments: [application/octet-stream] v6-0001-Perform-streaming-logical-transactions-by-background.patch (74.2K, ../../OS0PR01MB57164393B7F8A9A71C9CEB7394CA9@OS0PR01MB5716.jpnprd01.prod.outlook.com/2-v6-0001-Perform-streaming-logical-transactions-by-background.patch) download | inline diff: From 06cb238bc2acd9e2beaf4d2b3229e8e528088638 Mon Sep 17 00:00:00 2001 From: "houzj.fnst" <[email protected]> Date: Wed, 20 Apr 2022 16:45:07 +0800 Subject: [PATCH] Perform streaming logical transactions by background workers Currently, for large transactions, the publisher sends the data in multiple streams (changes divided into chunks depending upon logical_decoding_work_mem), and then on the subscriber-side, the apply worker writes the changes into temporary files and once it receives the commit, it reads from the file and applies the entire transaction. To improve the performance of such transactions, we can instead allow them to be applied via background workers. In this approach, we assign a new apply background worker (if available) as soon as the xact's first stream is received and the main apply worker will send changes to this new worker via shared memory. The apply background worker will directly apply the change instead of writing it to temporary files. We keep this worker assigned till the transaction commit is received and also wait for the worker to finish at commit. This preserves commit ordering and avoids writing to and reading from file in most cases. We still need to spill if there is no worker available. We also need to allow stream_stop to complete by the apply background worker to finish it to avoid deadlocks because T-1's current stream of changes can update rows in conflicting order with T-2's next stream of changes. This patch also extends the subscription streaming option so that the user can control whether to apply the streaming transaction in an apply background worker or spill the change to disk. User can set the streaming option to 'on/off', 'apply'. For now, 'apply' means the streaming will be applied via a apply background worker if available. 'on' means the streaming transaction will be spilled to disk. --- doc/src/sgml/catalogs.sgml | 10 +- doc/src/sgml/ref/create_subscription.sgml | 24 +- src/backend/commands/subscriptioncmds.c | 66 +- src/backend/postmaster/bgworker.c | 3 + src/backend/replication/logical/launcher.c | 87 +- src/backend/replication/logical/origin.c | 6 +- src/backend/replication/logical/tablesync.c | 10 +- src/backend/replication/logical/worker.c | 1396 ++++++++++++++++++++++++--- src/backend/utils/activity/wait_event.c | 3 + src/bin/pg_dump/pg_dump.c | 6 +- src/include/catalog/pg_subscription.h | 17 +- src/include/replication/logicalworker.h | 1 + src/include/replication/origin.h | 2 +- src/include/replication/worker_internal.h | 13 +- src/include/utils/wait_event.h | 1 + src/test/regress/expected/subscription.out | 2 +- src/tools/pgindent/typedefs.list | 4 + 17 files changed, 1455 insertions(+), 196 deletions(-) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index a533a21..5e46c4b 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -7863,11 +7863,15 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l <row> <entry role="catalog_table_entry"><para role="column_definition"> - <structfield>substream</structfield> <type>bool</type> + <structfield>substream</structfield> <type>char</type> </para> <para> - If true, the subscription will allow streaming of in-progress - transactions + Controls how to handle the streaming of in-progress transactions: + <literal>f</literal> = disallow streaming of in-progress transactions, + <literal>t</literal> = spill the changes of in-progress transactions to + disk and apply at once after the transaction is committed on the + publisher, + <literal>a</literal> = apply changes directly using a background worker </para></entry> </row> diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml index 203bb41..22bd818 100644 --- a/doc/src/sgml/ref/create_subscription.sgml +++ b/doc/src/sgml/ref/create_subscription.sgml @@ -217,13 +217,29 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl </varlistentry> <varlistentry> - <term><literal>streaming</literal> (<type>boolean</type>)</term> + <term><literal>streaming</literal> (<type>enum</type>)</term> <listitem> <para> Specifies whether to enable streaming of in-progress transactions - for this subscription. By default, all transactions - are fully decoded on the publisher and only then sent to the - subscriber as a whole. + for this subscription. The default value is <literal>off</literal>, + all transactions are fully decoded on the publisher and only then + sent to the subscriber as a whole. + </para> + + <para> + If set to <literal>on</literal>, the changes of transaction are + written to temporary files and then applied at once after the + transaction is committed on the publisher. + </para> + + <para> + If set to <literal>apply</literal> incoming + changes are directly applied via one of the background workers, if + available. If no background worker is free to handle streaming + transaction then the changes are written to a file and applied after + the transaction is committed. Note that if an error happens when + applying changes in a background worker, it might not report the + finish LSN of the remote transaction in the server log. </para> </listitem> </varlistentry> diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c index 690cdaa..f154762 100644 --- a/src/backend/commands/subscriptioncmds.c +++ b/src/backend/commands/subscriptioncmds.c @@ -83,7 +83,7 @@ typedef struct SubOpts bool copy_data; bool refresh; bool binary; - bool streaming; + char streaming; bool twophase; bool disableonerr; XLogRecPtr lsn; @@ -96,6 +96,62 @@ static void ReportSlotConnectionError(List *rstates, Oid subid, char *slotname, /* + * Extract the streaming mode value from a DefElem. This is like + * defGetBoolean() but also accepts the special value and "apply". + */ +static char +defGetStreamingMode(DefElem *def) +{ + /* + * If no parameter given, assume "true" is meant. + */ + if (def->arg == NULL) + return SUBSTREAM_ON; + + /* + * Allow 0, 1, "true", "false", "on", "off" or "apply". + */ + switch (nodeTag(def->arg)) + { + case T_Integer: + switch (intVal(def->arg)) + { + case 0: + return SUBSTREAM_OFF; + case 1: + return SUBSTREAM_ON; + default: + /* otherwise, error out below */ + break; + } + break; + default: + { + char *sval = defGetString(def); + + /* + * The set of strings accepted here should match up with the + * grammar's opt_boolean_or_string production. + */ + if (pg_strcasecmp(sval, "false") == 0 || + pg_strcasecmp(sval, "off") == 0) + return SUBSTREAM_OFF; + if (pg_strcasecmp(sval, "true") == 0 || + pg_strcasecmp(sval, "on") == 0) + return SUBSTREAM_ON; + if (pg_strcasecmp(sval, "apply") == 0) + return SUBSTREAM_APPLY; + } + break; + } + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s requires a Boolean value or \"apply\"", + def->defname))); + return SUBSTREAM_OFF; /* keep compiler quiet */ +} + +/* * Common option parsing function for CREATE and ALTER SUBSCRIPTION commands. * * Since not all options can be specified in both commands, this function @@ -132,7 +188,7 @@ parse_subscription_options(ParseState *pstate, List *stmt_options, if (IsSet(supported_opts, SUBOPT_BINARY)) opts->binary = false; if (IsSet(supported_opts, SUBOPT_STREAMING)) - opts->streaming = false; + opts->streaming = SUBSTREAM_OFF; if (IsSet(supported_opts, SUBOPT_TWOPHASE_COMMIT)) opts->twophase = false; if (IsSet(supported_opts, SUBOPT_DISABLE_ON_ERR)) @@ -233,7 +289,7 @@ parse_subscription_options(ParseState *pstate, List *stmt_options, errorConflictingDefElem(defel, pstate); opts->specified_opts |= SUBOPT_STREAMING; - opts->streaming = defGetBoolean(defel); + opts->streaming = defGetStreamingMode(defel); } else if (strcmp(defel->defname, "two_phase") == 0) { @@ -601,7 +657,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt, values[Anum_pg_subscription_subowner - 1] = ObjectIdGetDatum(owner); values[Anum_pg_subscription_subenabled - 1] = BoolGetDatum(opts.enabled); values[Anum_pg_subscription_subbinary - 1] = BoolGetDatum(opts.binary); - values[Anum_pg_subscription_substream - 1] = BoolGetDatum(opts.streaming); + values[Anum_pg_subscription_substream - 1] = CharGetDatum(opts.streaming); values[Anum_pg_subscription_subtwophasestate - 1] = CharGetDatum(opts.twophase ? LOGICALREP_TWOPHASE_STATE_PENDING : @@ -1060,7 +1116,7 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt, if (IsSet(opts.specified_opts, SUBOPT_STREAMING)) { values[Anum_pg_subscription_substream - 1] = - BoolGetDatum(opts.streaming); + CharGetDatum(opts.streaming); replaces[Anum_pg_subscription_substream - 1] = true; } diff --git a/src/backend/postmaster/bgworker.c b/src/backend/postmaster/bgworker.c index 40601ae..40ccb89 100644 --- a/src/backend/postmaster/bgworker.c +++ b/src/backend/postmaster/bgworker.c @@ -128,6 +128,9 @@ static const struct }, { "ApplyWorkerMain", ApplyWorkerMain + }, + { + "ApplyBgworkerMain", ApplyBgworkerMain } }; diff --git a/src/backend/replication/logical/launcher.c b/src/backend/replication/logical/launcher.c index bd5f78c..f8f0490 100644 --- a/src/backend/replication/logical/launcher.c +++ b/src/backend/replication/logical/launcher.c @@ -73,6 +73,7 @@ static void logicalrep_launcher_onexit(int code, Datum arg); static void logicalrep_worker_onexit(int code, Datum arg); static void logicalrep_worker_detach(void); static void logicalrep_worker_cleanup(LogicalRepWorker *worker); +static void logicalrep_worker_stop_internal(LogicalRepWorker *worker); static bool on_commit_launcher_wakeup = false; @@ -223,6 +224,10 @@ logicalrep_worker_find(Oid subid, Oid relid, bool only_running) { LogicalRepWorker *w = &LogicalRepCtx->workers[i]; + /* We only need main apply worker or table sync worker here */ + if (w->subworker) + continue; + if (w->in_use && w->subid == subid && w->relid == relid && (!only_running || w->proc)) { @@ -259,11 +264,11 @@ logicalrep_workers_find(Oid subid, bool only_running) } /* - * Start new apply background worker, if possible. + * Start new background worker, if possible. */ -void +bool logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, Oid userid, - Oid relid) + Oid relid, dsm_handle subworker_dsm) { BackgroundWorker bgw; BackgroundWorkerHandle *bgw_handle; @@ -273,6 +278,10 @@ logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, Oid userid, LogicalRepWorker *worker = NULL; int nsyncworkers; TimestampTz now; + bool is_subworker = (subworker_dsm != DSM_HANDLE_INVALID); + + /* We don't support table sync in subworker */ + Assert(!(is_subworker && OidIsValid(relid))); ereport(DEBUG1, (errmsg_internal("starting logical replication worker for subscription \"%s\"", @@ -350,7 +359,7 @@ retry: if (OidIsValid(relid) && nsyncworkers >= max_sync_workers_per_subscription) { LWLockRelease(LogicalRepWorkerLock); - return; + return false; } /* @@ -364,7 +373,7 @@ retry: (errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED), errmsg("out of logical replication worker slots"), errhint("You might need to increase max_logical_replication_workers."))); - return; + return false; } /* Prepare the worker slot. */ @@ -379,6 +388,7 @@ retry: worker->relstate = SUBREL_STATE_UNKNOWN; worker->relstate_lsn = InvalidXLogRecPtr; worker->stream_fileset = NULL; + worker->subworker = is_subworker; worker->last_lsn = InvalidXLogRecPtr; TIMESTAMP_NOBEGIN(worker->last_send_time); TIMESTAMP_NOBEGIN(worker->last_recv_time); @@ -396,19 +406,31 @@ retry: BGWORKER_BACKEND_DATABASE_CONNECTION; bgw.bgw_start_time = BgWorkerStart_RecoveryFinished; snprintf(bgw.bgw_library_name, BGW_MAXLEN, "postgres"); - snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyWorkerMain"); + + if (subworker_dsm == DSM_HANDLE_INVALID) + snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyWorkerMain"); + else + snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyBgworkerMain"); + + if (OidIsValid(relid)) snprintf(bgw.bgw_name, BGW_MAXLEN, "logical replication worker for subscription %u sync %u", subid, relid); - else + else if (subworker_dsm == DSM_HANDLE_INVALID) snprintf(bgw.bgw_name, BGW_MAXLEN, "logical replication worker for subscription %u", subid); + else + snprintf(bgw.bgw_name, BGW_MAXLEN, + "logical replication apply worker for subscription %u", subid); snprintf(bgw.bgw_type, BGW_MAXLEN, "logical replication worker"); bgw.bgw_restart_time = BGW_NEVER_RESTART; bgw.bgw_notify_pid = MyProcPid; bgw.bgw_main_arg = Int32GetDatum(slot); + if (is_subworker) + memcpy(bgw.bgw_extra, &subworker_dsm, sizeof(dsm_handle)); + if (!RegisterDynamicBackgroundWorker(&bgw, &bgw_handle)) { /* Failed to start worker, so clean up the worker slot. */ @@ -421,11 +443,13 @@ retry: (errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED), errmsg("out of background worker slots"), errhint("You might need to increase max_worker_processes."))); - return; + return false; } /* Now wait until it attaches. */ WaitForReplicationWorkerAttach(worker, generation, bgw_handle); + + return true; } /* @@ -436,7 +460,6 @@ void logicalrep_worker_stop(Oid subid, Oid relid) { LogicalRepWorker *worker; - uint16 generation; LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); @@ -449,6 +472,22 @@ logicalrep_worker_stop(Oid subid, Oid relid) return; } + logicalrep_worker_stop_internal(worker); + + LWLockRelease(LogicalRepWorkerLock); +} + +/* + * Workhorse for logicalrep_worker_stop() and logicalrep_worker_detach(). Stop + * the worker and wait for it to die. + */ +static void +logicalrep_worker_stop_internal(LogicalRepWorker *worker) +{ + uint16 generation; + + Assert(LWLockHeldByMe(LogicalRepWorkerLock)); + /* * Remember which generation was our worker so we can check if what we see * is still the same one. @@ -485,10 +524,7 @@ logicalrep_worker_stop(Oid subid, Oid relid) * different, meaning that a different worker has taken the slot. */ if (!worker->in_use || worker->generation != generation) - { - LWLockRelease(LogicalRepWorkerLock); return; - } /* Worker has assigned proc, so it has started. */ if (worker->proc) @@ -522,8 +558,6 @@ logicalrep_worker_stop(Oid subid, Oid relid) LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); } - - LWLockRelease(LogicalRepWorkerLock); } /* @@ -599,6 +633,28 @@ logicalrep_worker_attach(int slot) static void logicalrep_worker_detach(void) { + /* + * If we are the main apply worker, stop all the apply background workers + * we started before. + */ + if (!MyLogicalRepWorker->subworker) + { + List *workers; + ListCell *lc; + + LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); + + workers = logicalrep_workers_find(MyLogicalRepWorker->subid, true); + foreach(lc, workers) + { + LogicalRepWorker *w = (LogicalRepWorker *) lfirst(lc); + if (w->subworker) + logicalrep_worker_stop_internal(w); + } + + LWLockRelease(LogicalRepWorkerLock); + } + /* Block concurrent access. */ LWLockAcquire(LogicalRepWorkerLock, LW_EXCLUSIVE); @@ -621,6 +677,7 @@ logicalrep_worker_cleanup(LogicalRepWorker *worker) worker->userid = InvalidOid; worker->subid = InvalidOid; worker->relid = InvalidOid; + worker->subworker = false; } /* @@ -868,7 +925,7 @@ ApplyLauncherMain(Datum main_arg) wait_time = wal_retrieve_retry_interval; logicalrep_worker_launch(sub->dbid, sub->oid, sub->name, - sub->owner, InvalidOid); + sub->owner, InvalidOid, DSM_HANDLE_INVALID); } } diff --git a/src/backend/replication/logical/origin.c b/src/backend/replication/logical/origin.c index 21937ab..aaa6694 100644 --- a/src/backend/replication/logical/origin.c +++ b/src/backend/replication/logical/origin.c @@ -1068,7 +1068,7 @@ ReplicationOriginExitCleanup(int code, Datum arg) * with replorigin_session_reset(). */ void -replorigin_session_setup(RepOriginId node) +replorigin_session_setup(RepOriginId node, bool acquire) { static bool registered_cleanup; int i; @@ -1110,7 +1110,7 @@ replorigin_session_setup(RepOriginId node) if (curstate->roident != node) continue; - else if (curstate->acquired_by != 0) + else if (curstate->acquired_by != 0 && acquire) { ereport(ERROR, (errcode(ERRCODE_OBJECT_IN_USE), @@ -1321,7 +1321,7 @@ pg_replication_origin_session_setup(PG_FUNCTION_ARGS) name = text_to_cstring((text *) DatumGetPointer(PG_GETARG_DATUM(0))); origin = replorigin_by_name(name, false); - replorigin_session_setup(origin); + replorigin_session_setup(origin, true); replorigin_session_origin = origin; diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c index b03e0f5..b9be9d0 100644 --- a/src/backend/replication/logical/tablesync.c +++ b/src/backend/replication/logical/tablesync.c @@ -568,7 +568,8 @@ process_syncing_tables_for_apply(XLogRecPtr current_lsn) MySubscription->oid, MySubscription->name, MyLogicalRepWorker->userid, - rstate->relid); + rstate->relid, + DSM_HANDLE_INVALID); hentry->last_start_time = now; } } @@ -589,6 +590,9 @@ process_syncing_tables_for_apply(XLogRecPtr current_lsn) void process_syncing_tables(XLogRecPtr current_lsn) { + if (MyLogicalRepWorker->subworker) + return; + if (am_tablesync_worker()) process_syncing_tables_for_sync(current_lsn); else @@ -1275,7 +1279,7 @@ LogicalRepSyncTableStart(XLogRecPtr *origin_startpos) * time this tablesync was launched. */ originid = replorigin_by_name(originname, false); - replorigin_session_setup(originid); + replorigin_session_setup(originid, true); replorigin_session_origin = originid; *origin_startpos = replorigin_session_get_progress(false); @@ -1386,7 +1390,7 @@ LogicalRepSyncTableStart(XLogRecPtr *origin_startpos) true /* go backward */ , true /* WAL log */ ); UnlockRelationOid(ReplicationOriginRelationId, RowExclusiveLock); - replorigin_session_setup(originid); + replorigin_session_setup(originid, true); replorigin_session_origin = originid; } else diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c index 3b80ed9..862ad41 100644 --- a/src/backend/replication/logical/worker.c +++ b/src/backend/replication/logical/worker.c @@ -22,8 +22,25 @@ * STREAMED TRANSACTIONS * --------------------- * Streamed transactions (large transactions exceeding a memory limit on the - * upstream) are not applied immediately, but instead, the data is written - * to temporary files and then applied at once when the final commit arrives. + * upstream) are applied via one of two approaches. + * + * 1) Separate background workers + * + * Assign a new bgworker (if available) as soon as the xact's first stream is + * received and the main apply worker will send changes to this new worker via + * shared memory. We keep this worker assigned till the transaction commit is + * received and also wait for the worker to finish at commit. This preserves + * commit ordering and avoids writing to and reading from file in most cases. + * We still need to spill if there is no worker available. We also need to + * allow stream_stop to complete by the background worker to finish it to avoid + * deadlocks because T-1's current stream of changes can update rows in + * conflicting order with T-2's next stream of changes. + * + * 2) Write to temporary files and apply when the final commit arrives + * + * If no worker is available to handle streamed transaction, the data is + * written to temporary files and then applied at once when the final commit + * arrives. * * Unlike the regular (non-streamed) case, handling streamed transactions has * to handle aborts of both the toplevel transaction and subtransactions. This @@ -174,11 +191,15 @@ #include "rewrite/rewriteHandler.h" #include "storage/buffile.h" #include "storage/bufmgr.h" +#include "storage/dsm.h" #include "storage/fd.h" #include "storage/ipc.h" #include "storage/lmgr.h" #include "storage/proc.h" #include "storage/procarray.h" +#include "storage/shm_mq.h" +#include "storage/shm_toc.h" +#include "storage/spin.h" #include "tcop/tcopprot.h" #include "utils/acl.h" #include "utils/builtins.h" @@ -198,6 +219,75 @@ #define NAPTIME_PER_CYCLE 1000 /* max sleep time between cycles (1s) */ +#define PG_LOGICAL_APPLY_SHM_MAGIC 0x79fb2447 // TODO Consider change + +/* + * States for apply background worker. + */ +typedef enum ApplyBgworkerState +{ + APPLY_BGWORKER_ATTACHED = 0, + APPLY_BGWORKER_READY, + APPLY_BGWORKER_BUSY, + APPLY_BGWORKER_FINISHED, + APPLY_BGWORKER_EXIT +} ApplyBgworkerState; + +/* + * Shared information among apply workers. + */ +typedef struct ApplyBgworkerShared +{ + slock_t mutex; + + /* State for apply background worker. */ + ApplyBgworkerState state; + + TransactionId stream_xid; + uint32 n; /* id of apply background worker */ +} ApplyBgworkerShared; + +/* + * Struct for maintaining an apply background worker. + */ +typedef struct WorkerState +{ + shm_mq_handle *mq_handle; + dsm_segment *dsm_seg; + ApplyBgworkerShared volatile *pstate; +} WorkerState; + +typedef struct WorkerEntry +{ + TransactionId xid; + WorkerState *wstate; +} WorkerEntry; + +/* Apply background workers hash table (initialized on first use) */ +static HTAB *ApplyWorkersHash = NULL; +static List *ApplyWorkersIdleList = NIL; +static List *ApplyWorkersList = NIL; + +/* Fields valid only for apply background workers */ +volatile ApplyBgworkerShared *MyParallelState = NULL; +static List *subxactlist = NIL; + +/* The number of changes during one streaming block */ +static uint32 nchanges = 0; + +/* Worker setup and interactions */ +static WorkerState *apply_bgworker_setup(void); +static WorkerState *apply_bgworker_find_or_start(TransactionId xid, + bool start); +static void apply_bgworker_setup_dsm(WorkerState *wstate); +static void apply_bgworker_wait_for(WorkerState *wstate, + char wait_for_state); +static void apply_bgworker_send_data(WorkerState *wstate, Size nbytes, + const void *data); +static void apply_bgworker_free(WorkerState *wstate); +static void apply_bgworker_check_status(void); +static void apply_bgworker_set_state(char state); + typedef struct FlushPosition { dlist_node node; @@ -260,18 +350,23 @@ static XLogRecPtr remote_final_lsn = InvalidXLogRecPtr; static bool in_streamed_transaction = false; static TransactionId stream_xid = InvalidTransactionId; +static WorkerState *stream_apply_worker = NULL; + +/* check if we apply transaction in apply bgworker */ +#define apply_bgworker_active() (in_streamed_transaction && stream_apply_worker != NULL) /* * We enable skipping all data modification changes (INSERT, UPDATE, etc.) for * the subscription if the remote transaction's finish LSN matches the subskiplsn. * Once we start skipping changes, we don't stop it until we skip all changes of * the transaction even if pg_subscription is updated and MySubscription->skiplsn - * gets changed or reset during that. Also, in streaming transaction cases, we - * don't skip receiving and spooling the changes since we decide whether or not + * gets changed or reset during that. Also, in streaming transaction cases (streaming = on), + * we don't skip receiving and spooling the changes since we decide whether or not * to skip applying the changes when starting to apply changes. The subskiplsn is * cleared after successfully skipping the transaction or applying non-empty * transaction. The latter prevents the mistakenly specified subskiplsn from - * being left. + * being left. Note that we cannot skip the streaming transaction in apply mode, + * because we cannot get the finish LSN before applying the changes. */ static XLogRecPtr skip_xact_finish_lsn = InvalidXLogRecPtr; #define is_skipping_changes() (unlikely(!XLogRecPtrIsInvalid(skip_xact_finish_lsn))) @@ -426,43 +521,103 @@ end_replication_step(void) } /* - * Handle streamed transactions. + * Handle streamed transactions for both main apply worker and apply background + * worker. * - * If in streaming mode (receiving a block of streamed transaction), we - * simply redirect it to a file for the proper toplevel transaction. + * For the main apply worker, if in streaming mode (receiving a block of + * streamed transaction), we send the data to the apply background worker. + * + * For the apply background worker, define a savepoint if new subtransaction + * was started. * * Returns true for streamed transactions, false otherwise (regular mode). */ static bool handle_streamed_transaction(LogicalRepMsgType action, StringInfo s) { - TransactionId xid; + TransactionId current_xid = InvalidTransactionId; - /* not in streaming mode */ - if (!in_streamed_transaction) + /* + * Return if we are not in streaming mode and are not in an apply + * background worker. + */ + if (!in_streamed_transaction && !am_apply_bgworker()) return false; - Assert(stream_fd != NULL); Assert(TransactionIdIsValid(stream_xid)); /* * We should have received XID of the subxact as the first part of the * message, so extract it. */ - xid = pq_getmsgint(s, 4); + current_xid = pq_getmsgint(s, 4); - if (!TransactionIdIsValid(xid)) + if (!TransactionIdIsValid(current_xid)) ereport(ERROR, (errcode(ERRCODE_PROTOCOL_VIOLATION), errmsg_internal("invalid transaction ID in streamed replication transaction"))); - /* Add the new subxact to the array (unless already there). */ - subxact_info_add(xid); + if (am_apply_bgworker()) + { + /* + * Inside apply background worker we can figure out that new + * subtransaction was started if new change arrived with different xid. + * In that case we can define named savepoint, so that we were able to + * commit/rollback it separately later. + * + * Special case is if the first change comes from subtransaction, then + * we check that current_xid differs from stream_xid. + */ + if (current_xid != stream_xid && + !list_member_int(subxactlist, (int) current_xid)) + { + MemoryContext oldctx; + char spname[MAXPGPATH]; + + snprintf(spname, MAXPGPATH, "savepoint_for_xid_%u", current_xid); + + elog(LOG, "[Apply BGW #%u] defining savepoint %s", + MyParallelState->n, spname); + + DefineSavepoint(spname); + CommitTransactionCommand(); + + oldctx = MemoryContextSwitchTo(ApplyContext); + subxactlist = lappend_int(subxactlist, (int) current_xid); + MemoryContextSwitchTo(oldctx); + } + } + else if (apply_bgworker_active()) + { + /* + * If we decided to apply the changes of this transaction in an apply + * background worker, pass the data to the worker. + */ + apply_bgworker_send_data(stream_apply_worker, s->len, s->data); + nchanges += 1; + + /* + * XXX The publisher side doesn't always send relation update messages + * after the streaming transaction, so update the relation in main + * apply worker here. + */ + if (action == LOGICAL_REP_MSG_RELATION) + { + LogicalRepRelation *rel = logicalrep_read_rel(s); + logicalrep_relmap_update(rel); + } + + } + else + { + /* Add the new subxact to the array (unless already there). */ + subxact_info_add(current_xid); - /* write the change to the current file */ - stream_write_change(action, s); + /* write the change to the current file */ + stream_write_change(action, s); + } - return true; + return !am_apply_bgworker(); } /* @@ -844,6 +999,9 @@ apply_handle_commit(StringInfo s) apply_handle_commit_internal(&commit_data); + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); + /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(commit_data.end_lsn); @@ -898,7 +1056,9 @@ apply_handle_prepare_internal(LogicalRepPreparedTxnData *prepare_data) * BeginTransactionBlock is necessary to balance the EndTransactionBlock * called within the PrepareTransactionBlock below. */ - BeginTransactionBlock(); + if (!IsTransactionBlock()) + BeginTransactionBlock(); + CommitTransactionCommand(); /* Completes the preceding Begin command. */ /* @@ -950,6 +1110,9 @@ apply_handle_prepare(StringInfo s) in_remote_transaction = false; + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); + /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(prepare_data.end_lsn); @@ -1063,11 +1226,130 @@ apply_handle_rollback_prepared(StringInfo s) } /* - * Handle STREAM PREPARE. + * Look up worker inside ApplyWorkersHash for requested xid. * - * Logic is in two parts: - * 1. Replay all the spooled operations - * 2. Mark the transaction as prepared + * If start flag is true, try to start a new worker if not found in hash table. + */ +static WorkerState * +apply_bgworker_find_or_start(TransactionId xid, bool start) +{ + bool found; + WorkerState *wstate; + WorkerEntry *entry = NULL; + + if (!TransactionIdIsValid(xid)) + return NULL; + + /* + * We don't start new background worker if we are not in streaming apply + * mode. + */ + if (MySubscription->stream != SUBSTREAM_APPLY) + return NULL; + + /* + * We don't start new background worker if user has set skiplsn as it's + * possible that user want to skip the streaming transaction. For streaming + * transaction, we need to spill the transaction to disk so that we can get + * the last LSN of the transaction to judge whether to skip before starting + * to apply the change. + */ + if (start && !XLogRecPtrIsInvalid(MySubscription->skiplsn)) + return NULL; + + /* + * For streaming transactions that are being applied in bgworker, we cannot + * decide whether to apply the change for a relation that is not in the + * READY state (see should_apply_changes_for_rel) as we won't know + * remote_final_lsn by that time. So, we don't start new bgworker in this + * case. + */ + if (start && !AllTablesyncsReady()) + return NULL; + + /* First time through, initialize apply workers hashtable */ + if (ApplyWorkersHash == NULL) + { + HASHCTL ctl; + + MemSet(&ctl, 0, sizeof(ctl)); + ctl.keysize = sizeof(TransactionId); + ctl.entrysize = sizeof(WorkerEntry); + ctl.hcxt = ApplyContext; + + ApplyWorkersHash = hash_create("logical apply workers hash", 8, &ctl, + HASH_ELEM | HASH_BLOBS | HASH_CONTEXT); + } + + /* + * Find entry for requested transaction. + */ + entry = hash_search(ApplyWorkersHash, &xid, start ? HASH_ENTER : HASH_FIND, + &found); + if (found) + { + entry->wstate->pstate->state = APPLY_BGWORKER_BUSY; + return entry->wstate; + } + else if (!start) + return NULL; + + /* If there is at least one worker in the idle list, then take one. */ + if (list_length(ApplyWorkersIdleList) > 0) + { + wstate = (WorkerState *) llast(ApplyWorkersIdleList); + ApplyWorkersIdleList = list_delete_last(ApplyWorkersIdleList); + } + else + { + wstate = apply_bgworker_setup(); + + if (wstate == NULL) + { + /* + * If the bgworker cannot be launched, remove entry in hash table. + */ + hash_search(ApplyWorkersHash, &xid, HASH_REMOVE, &found); + return NULL; + } + } + + /* Fill up the hash entry */ + wstate->pstate->state = APPLY_BGWORKER_BUSY; + wstate->pstate->stream_xid = xid; + entry->wstate = wstate; + entry->xid = xid; + + return wstate; +} + +/* + * Add the worker to the freelist and remove the entry from hash table. + */ +static void +apply_bgworker_free(WorkerState *wstate) +{ + bool found; + MemoryContext oldctx; + TransactionId xid = wstate->pstate->stream_xid; + + Assert(wstate->pstate->state == APPLY_BGWORKER_FINISHED); + + oldctx = MemoryContextSwitchTo(ApplyContext); + + hash_search(ApplyWorkersHash, &xid, + HASH_REMOVE, &found); + + elog(LOG, "adding finished apply worker #%u for xid %u to the idle list", + wstate->pstate->n, wstate->pstate->stream_xid); + + ApplyWorkersIdleList = lappend(ApplyWorkersIdleList, wstate); + + MemoryContextSwitchTo(oldctx); +} + +/* + * Handle STREAM PREPARE. */ static void apply_handle_stream_prepare(StringInfo s) @@ -1088,24 +1370,68 @@ apply_handle_stream_prepare(StringInfo s) logicalrep_read_stream_prepare(s, &prepare_data); set_apply_error_context_xact(prepare_data.xid, prepare_data.prepare_lsn); - elog(DEBUG1, "received prepare for streamed transaction %u", prepare_data.xid); + if (am_apply_bgworker()) + { + elog(LOG, "received prepare for streamed transaction %u", prepare_data.xid); - /* Replay all the spooled operations. */ - apply_spooled_messages(prepare_data.xid, prepare_data.prepare_lsn); + /* Mark the transaction as prepared. */ + apply_handle_prepare_internal(&prepare_data); - /* Mark the transaction as prepared. */ - apply_handle_prepare_internal(&prepare_data); + CommitTransactionCommand(); - CommitTransactionCommand(); + pgstat_report_stat(false); - pgstat_report_stat(false); + list_free(subxactlist); + subxactlist = NIL; - store_flush_position(prepare_data.end_lsn); + apply_bgworker_set_state(APPLY_BGWORKER_READY); + } + else + { + WorkerState *wstate = apply_bgworker_find_or_start(prepare_data.xid, false); + + /* + * If we are in main apply worker, check if we are processing this + * transaction in a bgworker. + */ + if (wstate) + { + apply_bgworker_send_data(wstate, s->len, s->data); + + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_READY); + + pgstat_report_stat(false); + store_flush_position(prepare_data.end_lsn); + } + else + { + /* + * If we are in main apply worker and the transaction has been + * serialized to file, replay all the spooled operations. + */ + apply_spooled_messages(prepare_data.xid, prepare_data.prepare_lsn); + + /* Mark the transaction as prepared. */ + apply_handle_prepare_internal(&prepare_data); + + CommitTransactionCommand(); + + pgstat_report_stat(false); + + store_flush_position(prepare_data.end_lsn); + + in_remote_transaction = false; + + /* unlink the files with serialized changes and subxact info. */ + stream_cleanup_files(MyLogicalRepWorker->subid, prepare_data.xid); + } + } in_remote_transaction = false; + stream_apply_worker = NULL; - /* unlink the files with serialized changes and subxact info. */ - stream_cleanup_files(MyLogicalRepWorker->subid, prepare_data.xid); + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(prepare_data.end_lsn); @@ -1155,15 +1481,6 @@ apply_handle_stream_start(StringInfo s) (errcode(ERRCODE_PROTOCOL_VIOLATION), errmsg_internal("duplicate STREAM START message"))); - /* - * Start a transaction on stream start, this transaction will be committed - * on the stream stop unless it is a tablesync worker in which case it - * will be committed after processing all the messages. We need the - * transaction for handling the buffile, used for serializing the - * streaming data and subxact info. - */ - begin_replication_step(); - /* notify handle methods we're processing a remote transaction */ in_streamed_transaction = true; @@ -1177,36 +1494,90 @@ apply_handle_stream_start(StringInfo s) set_apply_error_context_xact(stream_xid, InvalidXLogRecPtr); + if (am_apply_bgworker()) + { + /* + * Make sure the handle apply_dispatch methods are aware we're in a + * remote transaction. + */ + in_remote_transaction = true; + + /* If we are in a bgworker, begin the transaction */ + AcceptInvalidationMessages(); + maybe_reread_subscription(); + + StartTransactionCommand(); + BeginTransactionBlock(); + CommitTransactionCommand(); + + return; + } + /* - * Initialize the worker's stream_fileset if we haven't yet. This will be - * used for the entire duration of the worker so create it in a permanent - * context. We create this on the very first streaming message from any - * transaction and then use it for this and other streaming transactions. - * Now, we could create a fileset at the start of the worker as well but - * then we won't be sure that it will ever be used. + * If we are in main apply worker, check if there is any free bgworker + * we can use to process this transaction. */ - if (MyLogicalRepWorker->stream_fileset == NULL) + stream_apply_worker = apply_bgworker_find_or_start(stream_xid, first_segment); + + if (stream_apply_worker) { - MemoryContext oldctx; + /* + * If we have free worker or we already started to apply this + * transaction in bgworker, we pass the data to worker. + */ + if (first_segment) + apply_bgworker_send_data(stream_apply_worker, s->len, s->data); - oldctx = MemoryContextSwitchTo(ApplyContext); + nchanges = 0; + elog(LOG, "starting streaming of xid %u", stream_xid); + } - MyLogicalRepWorker->stream_fileset = palloc(sizeof(FileSet)); - FileSetInit(MyLogicalRepWorker->stream_fileset); + /* + * If no worker is available for the first stream start, we start to + * serialize all the changes of the transaction. + */ + else + { + /* + * Start a transaction on stream start, this transaction will be committed + * on the stream stop unless it is a tablesync worker in which case it + * will be committed after processing all the messages. We need the + * transaction for handling the buffile, used for serializing the + * streaming data and subxact info. + */ + begin_replication_step(); - MemoryContextSwitchTo(oldctx); - } + /* + * Initialize the worker's stream_fileset if we haven't yet. This will be + * used for the entire duration of the worker so create it in a permanent + * context. We create this on the very first streaming message from any + * transaction and then use it for this and other streaming transactions. + * Now, we could create a fileset at the start of the worker as well but + * then we won't be sure that it will ever be used. + */ + if (MyLogicalRepWorker->stream_fileset == NULL) + { + MemoryContext oldctx; - /* open the spool file for this transaction */ - stream_open_file(MyLogicalRepWorker->subid, stream_xid, first_segment); + oldctx = MemoryContextSwitchTo(ApplyContext); - /* if this is not the first segment, open existing subxact file */ - if (!first_segment) - subxact_info_read(MyLogicalRepWorker->subid, stream_xid); + MyLogicalRepWorker->stream_fileset = palloc(sizeof(FileSet)); + FileSetInit(MyLogicalRepWorker->stream_fileset); - pgstat_report_activity(STATE_RUNNING, NULL); + MemoryContextSwitchTo(oldctx); + } - end_replication_step(); + /* open the spool file for this transaction */ + stream_open_file(MyLogicalRepWorker->subid, stream_xid, first_segment); + + /* if this is not the first segment, open existing subxact file */ + if (!first_segment) + subxact_info_read(MyLogicalRepWorker->subid, stream_xid); + + end_replication_step(); + } + + pgstat_report_activity(STATE_RUNNING, NULL); } /* @@ -1220,44 +1591,47 @@ apply_handle_stream_stop(StringInfo s) (errcode(ERRCODE_PROTOCOL_VIOLATION), errmsg_internal("STREAM STOP message without STREAM START"))); - /* - * Close the file with serialized changes, and serialize information about - * subxacts for the toplevel transaction. - */ - subxact_info_write(MyLogicalRepWorker->subid, stream_xid); - stream_close_file(); + if (apply_bgworker_active()) + { + char action = LOGICAL_REP_MSG_STREAM_STOP; - /* We must be in a valid transaction state */ - Assert(IsTransactionState()); + apply_bgworker_send_data(stream_apply_worker, 1, &action); + apply_bgworker_wait_for(stream_apply_worker, APPLY_BGWORKER_READY); - /* Commit the per-stream transaction */ - CommitTransactionCommand(); + elog(LOG, "stopped streaming of xid %u, %u changes streamed", stream_xid, nchanges); + } + else + { + /* + * Close the file with serialized changes, and serialize information + * about subxacts for the toplevel transaction. + */ + subxact_info_write(MyLogicalRepWorker->subid, stream_xid); + stream_close_file(); - in_streamed_transaction = false; + /* We must be in a valid transaction state */ + Assert(IsTransactionState()); + + /* Commit the per-stream transaction */ + CommitTransactionCommand(); + + /* Reset per-stream context */ + MemoryContextReset(LogicalStreamingContext); + } - /* Reset per-stream context */ - MemoryContextReset(LogicalStreamingContext); + in_streamed_transaction = false; + stream_apply_worker = NULL; pgstat_report_activity(STATE_IDLE, NULL); reset_apply_error_context_info(); } /* - * Handle STREAM abort message. + * Handle STREAM ABORT message when the transaction was spilled to disk. */ static void -apply_handle_stream_abort(StringInfo s) +serialize_stream_abort(TransactionId xid, TransactionId subxid) { - TransactionId xid; - TransactionId subxid; - - if (in_streamed_transaction) - ereport(ERROR, - (errcode(ERRCODE_PROTOCOL_VIOLATION), - errmsg_internal("STREAM ABORT message without STREAM STOP"))); - - logicalrep_read_stream_abort(s, &xid, &subxid); - /* * If the two XIDs are the same, it's in fact abort of toplevel xact, so * just delete the files with serialized info. @@ -1339,51 +1713,163 @@ apply_handle_stream_abort(StringInfo s) end_replication_step(); CommitTransactionCommand(); } - - reset_apply_error_context_info(); } /* - * Common spoolfile processing. + * Handle STREAM ABORT message. */ static void -apply_spooled_messages(TransactionId xid, XLogRecPtr lsn) +apply_handle_stream_abort(StringInfo s) { - StringInfoData s2; - int nchanges; - char path[MAXPGPATH]; - char *buffer = NULL; - MemoryContext oldcxt; - BufFile *fd; + TransactionId xid; + TransactionId subxid; - maybe_start_skipping_changes(lsn); + if (in_streamed_transaction) + ereport(ERROR, + (errcode(ERRCODE_PROTOCOL_VIOLATION), + errmsg_internal("STREAM COMMIT message without STREAM STOP"))); - /* Make sure we have an open transaction */ - begin_replication_step(); + logicalrep_read_stream_abort(s, &xid, &subxid); - /* - * Allocate file handle and memory required to process all the messages in - * TopTransactionContext to avoid them getting reset after each message is - * processed. - */ - oldcxt = MemoryContextSwitchTo(TopTransactionContext); + if (am_apply_bgworker()) + { + ereport(LOG, + (errmsg("[Apply BGW #%u] aborting current transaction xid=%u, subxid=%u", + MyParallelState->n, GetCurrentTransactionIdIfAny(), GetCurrentSubTransactionId()))); - /* Open the spool file for the committed/prepared transaction */ - changes_filename(path, MyLogicalRepWorker->subid, xid); - elog(DEBUG1, "replaying changes from file \"%s\"", path); + /* + * If the two XIDs are the same, it's in fact abort of toplevel xact, + * so just free the subxactlist. + */ + if (subxid == xid) + { + set_apply_error_context_xact(subxid, InvalidXLogRecPtr); - fd = BufFileOpenFileSet(MyLogicalRepWorker->stream_fileset, path, O_RDONLY, - false); + AbortCurrentTransaction(); - buffer = palloc(BLCKSZ); - initStringInfo(&s2); + EndTransactionBlock(false); + CommitTransactionCommand(); - MemoryContextSwitchTo(oldcxt); + in_remote_transaction = false; - remote_final_lsn = lsn; + list_free(subxactlist); + subxactlist = NIL; - /* - * Make sure the handle apply_dispatch methods are aware we're in a remote + apply_bgworker_set_state(APPLY_BGWORKER_FINISHED); + } + else + { + /* + * OK, so it's a subxact. Rollback to the savepoint. + * + * We also need to read the subxactlist, determine the offset + * tracked for the subxact, and truncate the list. + */ + int i; + bool found = false; + char spname[MAXPGPATH]; + + set_apply_error_context_xact(subxid, InvalidXLogRecPtr); + snprintf(spname, MAXPGPATH, "savepoint_for_xid_%u", subxid); + + ereport(LOG, + (errmsg("[Apply BGW #%u] rolling back to savepoint %s", + MyParallelState->n, spname))); + + for(i = list_length(subxactlist) - 1; i >= 0; i--) + { + xid = (TransactionId) list_nth_int(subxactlist, i); + if (xid == subxid) + { + found = true; + break; + } + } + + if (found) + { + elog(LOG, "rolled back to savepoint %s", spname); + RollbackToSavepoint(spname); + CommitTransactionCommand(); + subxactlist = list_truncate(subxactlist, i + 1); + } + + apply_bgworker_set_state(APPLY_BGWORKER_READY); + } + + reset_apply_error_context_info(); + } + else + { + WorkerState *wstate = apply_bgworker_find_or_start(xid, false); + + /* + * If we are in main apply worker, check if we are processing this + * transaction in a bgworker. + */ + if (wstate) + { + apply_bgworker_send_data(wstate, s->len, s->data); + + if (subxid == xid) + { + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_FINISHED); + apply_bgworker_free(wstate); + } + else + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_READY); + } + + /* + * We are in main apply worker and the transaction has been serialized + * to file. + */ + else + serialize_stream_abort(xid, subxid); + } +} + +/* + * Common spoolfile processing. + */ +static void +apply_spooled_messages(TransactionId xid, XLogRecPtr lsn) +{ + StringInfoData s2; + int nchanges; + char path[MAXPGPATH]; + char *buffer = NULL; + MemoryContext oldcxt; + BufFile *fd; + + maybe_start_skipping_changes(lsn); + + /* Make sure we have an open transaction */ + begin_replication_step(); + + /* + * Allocate file handle and memory required to process all the messages in + * TopTransactionContext to avoid them getting reset after each message is + * processed. + */ + oldcxt = MemoryContextSwitchTo(TopTransactionContext); + + /* Open the spool file for the committed/prepared transaction */ + changes_filename(path, MyLogicalRepWorker->subid, xid); + elog(DEBUG1, "replaying changes from file \"%s\"", path); + + fd = BufFileOpenFileSet(MyLogicalRepWorker->stream_fileset, path, O_RDONLY, + false); + + buffer = palloc(BLCKSZ); + initStringInfo(&s2); + + MemoryContextSwitchTo(oldcxt); + + remote_final_lsn = lsn; + + /* + * Make sure the handle apply_dispatch methods are aware we're in a remote * transaction. */ in_remote_transaction = true; @@ -1463,40 +1949,6 @@ apply_spooled_messages(TransactionId xid, XLogRecPtr lsn) } /* - * Handle STREAM COMMIT message. - */ -static void -apply_handle_stream_commit(StringInfo s) -{ - TransactionId xid; - LogicalRepCommitData commit_data; - - if (in_streamed_transaction) - ereport(ERROR, - (errcode(ERRCODE_PROTOCOL_VIOLATION), - errmsg_internal("STREAM COMMIT message without STREAM STOP"))); - - xid = logicalrep_read_stream_commit(s, &commit_data); - set_apply_error_context_xact(xid, commit_data.commit_lsn); - - elog(DEBUG1, "received commit for streamed transaction %u", xid); - - apply_spooled_messages(xid, commit_data.commit_lsn); - - apply_handle_commit_internal(&commit_data); - - /* unlink the files with serialized changes and subxact info */ - stream_cleanup_files(MyLogicalRepWorker->subid, xid); - - /* Process any tables that are being synchronized in parallel. */ - process_syncing_tables(commit_data.end_lsn); - - pgstat_report_activity(STATE_IDLE, NULL); - - reset_apply_error_context_info(); -} - -/* * Helper function for apply_handle_commit and apply_handle_stream_commit. */ static void @@ -2445,6 +2897,100 @@ apply_handle_truncate(StringInfo s) end_replication_step(); } +/* + * Handle STREAM COMMIT message. + */ +static void +apply_handle_stream_commit(StringInfo s) +{ + LogicalRepCommitData commit_data; + TransactionId xid; + + if (in_streamed_transaction) + ereport(ERROR, + (errcode(ERRCODE_PROTOCOL_VIOLATION), + errmsg_internal("STREAM COMMIT message without STREAM STOP"))); + + xid = logicalrep_read_stream_commit(s, &commit_data); + set_apply_error_context_xact(xid, commit_data.commit_lsn); + + elog(DEBUG1, "received commit for streamed transaction %u", xid); + + if (am_apply_bgworker()) + { + /* + * Update origin state so we can restart streaming from correct + * position in case of crash. + */ + replorigin_session_origin_lsn = commit_data.end_lsn; + replorigin_session_origin_timestamp = commit_data.committime; + + CommitTransactionCommand(); + EndTransactionBlock(false); + CommitTransactionCommand(); + + in_remote_transaction = false; + + pgstat_report_stat(false); + + list_free(subxactlist); + subxactlist = NIL; + + apply_bgworker_set_state(APPLY_BGWORKER_FINISHED); + } + else + { + /* + * If we are in main apply worker, check if we are processing this + * transaction in an apply background worker. + */ + WorkerState *wstate = apply_bgworker_find_or_start(xid, false); + + if (wstate) + { + /* Send commit message */ + apply_bgworker_send_data(wstate, s->len, s->data); + + /* Wait for apply background worker to finish */ + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_FINISHED); + + pgstat_report_stat(false); + store_flush_position(commit_data.end_lsn); + stop_skipping_changes(); + + apply_bgworker_free(wstate); + + /* + * The transaction is either non-empty or skipped, so we clear the + * subskiplsn. + */ + clear_subscription_skip_lsn(commit_data.commit_lsn); + } + else + { + /* + * If we are in main apply worker and the transaction has been + * serialized to file, replay all the spooled operations. + */ + apply_spooled_messages(xid, commit_data.commit_lsn); + + apply_handle_commit_internal(&commit_data); + + /* unlink the files with serialized changes and subxact info */ + stream_cleanup_files(MyLogicalRepWorker->subid, xid); + } + } + + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); + + /* Process any tables that are being synchronized in parallel. */ + process_syncing_tables(commit_data.end_lsn); + + pgstat_report_activity(STATE_IDLE, NULL); + + reset_apply_error_context_info(); +} /* * Logical replication protocol message dispatcher. @@ -2511,6 +3057,8 @@ apply_dispatch(StringInfo s) break; case LOGICAL_REP_MSG_STREAM_START: + /* FIXME : log for debugging here. */ + elog(LOG, "LOGICAL_REP_MSG_STREAM_START"); apply_handle_stream_start(s); break; @@ -2618,6 +3166,10 @@ store_flush_position(XLogRecPtr remote_lsn) { FlushPosition *flushpos; + /* We only need to collect the LSN in main apply worker */ + if (am_apply_bgworker()) + return; + /* Need to do this in permanent context */ MemoryContextSwitchTo(ApplyContext); @@ -2794,6 +3346,9 @@ LogicalRepApplyLoop(XLogRecPtr last_received) AcceptInvalidationMessages(); maybe_reread_subscription(); + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); + /* Process any table synchronization changes. */ process_syncing_tables(last_received); } @@ -3686,7 +4241,7 @@ ApplyWorkerMain(Datum main_arg) originid = replorigin_by_name(originname, true); if (!OidIsValid(originid)) originid = replorigin_create(originname); - replorigin_session_setup(originid); + replorigin_session_setup(originid, true); replorigin_session_origin = originid; origin_startpos = replorigin_session_get_progress(false); CommitTransactionCommand(); @@ -3733,7 +4288,7 @@ ApplyWorkerMain(Datum main_arg) options.proto.logical.publication_names = MySubscription->publications; options.proto.logical.binary = MySubscription->binary; - options.proto.logical.streaming = MySubscription->stream; + options.proto.logical.streaming = (MySubscription->stream != SUBSTREAM_OFF); options.proto.logical.twophase = false; if (!am_tablesync_worker()) @@ -3891,7 +4446,8 @@ clear_subscription_skip_lsn(XLogRecPtr finish_lsn) XLogRecPtr myskiplsn = MySubscription->skiplsn; bool started_tx = false; - if (likely(XLogRecPtrIsInvalid(myskiplsn))) + if (likely(XLogRecPtrIsInvalid(myskiplsn)) || + am_apply_bgworker()) return; if (!IsTransactionState()) @@ -4027,3 +4583,533 @@ reset_apply_error_context_info(void) apply_error_callback_arg.remote_attnum = -1; set_apply_error_context_xact(InvalidTransactionId, InvalidXLogRecPtr); } + +/* Apply Background Worker main loop */ +static void +LogicalApplyBgwLoop(shm_mq_handle *mqh, volatile ApplyBgworkerShared *pst) +{ + shm_mq_result shmq_res; + PGPROC *registrant; + ErrorContextCallback errcallback; + XLogRecPtr last_received = InvalidXLogRecPtr; + + registrant = BackendPidGetProc(MyBgworkerEntry->bgw_notify_pid); + SetLatch(®istrant->procLatch); + + /* + * Push apply error context callback. Fields will be filled during + * applying a change. + */ + errcallback.callback = apply_error_callback; + errcallback.previous = error_context_stack; + error_context_stack = &errcallback; + + for (;;) + { + void *data; + Size len; + StringInfoData s; + MemoryContext oldctx; + XLogRecPtr start_lsn; + XLogRecPtr end_lsn; + TimestampTz send_time; + + CHECK_FOR_INTERRUPTS(); + + /* Ensure we are reading the data into our memory context. */ + oldctx = MemoryContextSwitchTo(ApplyMessageContext); + + shmq_res = shm_mq_receive(mqh, &len, &data, false); + + if (shmq_res != SHM_MQ_SUCCESS) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("lost connection to the main apply worker"))); + + if (len == 0) + { + elog(LOG, "[Apply BGW #%u] got zero-length message, stopping", pst->n); + break; + } + + s.cursor = 0; + s.maxlen = -1; + s.data = (char *) data; + s.len = len; + + /* + * We use first byte of message for additional communication between + * main Logical replication worker and apply bgworkers, so if it + * differs from 'w', then process it first. + */ + switch (pq_getmsgbyte(&s)) + { + /* End message of streaming chunk */ + case LOGICAL_REP_MSG_STREAM_STOP: + elog(LOG, "[Apply BGW #%u] ended processing streaming chunk," + "waiting on shm_mq_receive", pst->n); + + apply_bgworker_set_state(APPLY_BGWORKER_READY); + + SetLatch(®istrant->procLatch); + + in_streamed_transaction = false; + pgstat_report_activity(STATE_IDLE, NULL); + continue; + case 'w': + break; + default: + elog(ERROR, "unexpected message"); + break; + } + + start_lsn = pq_getmsgint64(&s); + end_lsn = pq_getmsgint64(&s); + send_time = pq_getmsgint64(&s); + + if (last_received < start_lsn) + last_received = start_lsn; + + if (last_received < end_lsn) + last_received = end_lsn; + + /* + * TO IMPROVE: Do we need to display the bgworker's information in + * pg_stat_replication ? + */ + UpdateWorkerStats(last_received, send_time, false); + + apply_dispatch(&s); + + if (ConfigReloadPending) + { + ConfigReloadPending = false; + ProcessConfigFile(PGC_SIGHUP); + } + + MemoryContextSwitchTo(oldctx); + MemoryContextReset(ApplyMessageContext); + } + + MemoryContextSwitchTo(TopMemoryContext); + MemoryContextReset(ApplyContext); + + /* Pop the error context stack */ + error_context_stack = errcallback.previous; + + elog(LOG, "[Apply BGW #%u] exiting", pst->n); + + /* Signal main process that we are done. */ + SetLatch(®istrant->procLatch); +} + +/* + * Set the exit state so that the main apply worker can realize we have + * shutdown. + */ +static void +ApplyBgwShutdown(int code, Datum arg) +{ + SpinLockAcquire(&MyParallelState->mutex); + MyParallelState->state = APPLY_BGWORKER_EXIT; + SpinLockRelease(&MyParallelState->mutex); + + dsm_detach((dsm_segment *) DatumGetPointer(arg)); +} + +/* + * Apply Background Worker entry point + */ +void +ApplyBgworkerMain(Datum main_arg) +{ + volatile ApplyBgworkerShared *pst; + + dsm_handle handle; + dsm_segment *seg; + shm_toc *toc; + shm_mq *mq; + shm_mq_handle *mqh; + MemoryContext oldcontext; + RepOriginId originid; + int worker_slot = DatumGetInt32(main_arg); + char originname[NAMEDATALEN]; + + /* Attach to slot */ + logicalrep_worker_attach(worker_slot); + + MemoryContextSwitchTo(TopMemoryContext); + + /* Load the subscription into persistent memory context. */ + ApplyContext = AllocSetContextCreate(TopMemoryContext, + "ApplyContext", + ALLOCSET_DEFAULT_SIZES); + + /* + * Init the ApplyMessageContext which we clean up after each replication + * protocol message. + */ + ApplyMessageContext = AllocSetContextCreate(ApplyContext, + "ApplyMessageContext", + ALLOCSET_DEFAULT_SIZES); + + /* Setup signal handling */ + pqsignal(SIGHUP, SignalHandlerForConfigReload); + pqsignal(SIGTERM, die); + BackgroundWorkerUnblockSignals(); + + /* + * Connect to the dynamic shared memory segment. + * + * The backend that registered this worker passed us the ID of a shared + * memory segment to which we must attach for further instructions. In + * order to attach to dynamic shared memory, we need a resource owner. + * Once we've mapped the segment in our address space, attach to the table + * of contents so we can locate the various data structures we'll need to + * find within the segment. + */ + CurrentResourceOwner = ResourceOwnerCreate(NULL, "Logical apply worker"); + memcpy(&handle, MyBgworkerEntry->bgw_extra, sizeof(dsm_handle)); + seg = dsm_attach(handle); + if (seg == NULL) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("unable to map dynamic shared memory segment"))); + toc = shm_toc_attach(PG_LOGICAL_APPLY_SHM_MAGIC, dsm_segment_address(seg)); + if (toc == NULL) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("bad magic number in dynamic shared memory segment"))); + + before_shmem_exit(ApplyBgwShutdown, PointerGetDatum(seg)); + + /* Look up the parallel state. */ + pst = shm_toc_lookup(toc, 0, false); + MyParallelState = pst; + + /* + * Attach to the message queue. + */ + mq = shm_toc_lookup(toc, 1, false); + shm_mq_set_receiver(mq, MyProc); + mqh = shm_mq_attach(mq, seg, NULL); + + /* Connect to our database. */ + BackgroundWorkerInitializeConnectionByOid(MyLogicalRepWorker->dbid, + MyLogicalRepWorker->userid, + 0); + + /* + * Set the client encoding to the database encoding, since that is what + * the leader will expect. + */ + SetClientEncoding(GetDatabaseEncoding()); + + stream_xid = pst->stream_xid; + + StartTransactionCommand(); + oldcontext = MemoryContextSwitchTo(ApplyContext); + + MySubscription = GetSubscription(MyLogicalRepWorker->subid, true); + if (!MySubscription) + { + ereport(LOG, + (errmsg("logical replication apply worker for subscription %u will not " + "start because the subscription was removed during startup", + MyLogicalRepWorker->subid))); + proc_exit(0); + } + + MySubscriptionValid = true; + MemoryContextSwitchTo(oldcontext); + + /* Setup synchronous commit according to the user's wishes */ + SetConfigOption("synchronous_commit", MySubscription->synccommit, + PGC_BACKEND, PGC_S_OVERRIDE); + + /* Keep us informed about subscription changes. */ + CacheRegisterSyscacheCallback(SUBSCRIPTIONOID, + subscription_change_cb, + (Datum) 0); + + CommitTransactionCommand(); + + /* Setup replication origin tracking. */ + StartTransactionCommand(); + snprintf(originname, sizeof(originname), "pg_%u", MySubscription->oid); + originid = replorigin_by_name(originname, true); + if (!OidIsValid(originid)) + originid = replorigin_create(originname); + + /* + * The apply bgworker don't need to monopolize this replication origin + * which was already acquired by its leader process. + */ + replorigin_session_setup(originid, false); + replorigin_session_origin = originid; + CommitTransactionCommand(); + + /* + * Allocate the origin name in long-lived context for error context + * message. + */ + apply_error_callback_arg.origin_name = MemoryContextStrdup(ApplyContext, + originname); + + /* + * Indicate that we're fully initialized and ready to begin the main part + * of the apply operation. + */ + apply_bgworker_set_state(APPLY_BGWORKER_ATTACHED); + + elog(LOG, "[Apply BGW #%u] started", pst->n); + + PG_TRY(); + { + LogicalApplyBgwLoop(mqh, pst); + } + PG_CATCH(); + { + /* + * Report the worker failed while applying streaming transaction + * changes. Abort the current transaction so that the stats message is + * sent in an idle state. + */ + AbortOutOfAnyTransaction(); + pgstat_report_subscription_error(MySubscription->oid, false); + + PG_RE_THROW(); + } + PG_END_TRY(); + + /* + * We're done. Explicitly detach the shared memory segment so that we + * don't get a resource leak warning at commit time. This will fire any + * on_dsm_detach callbacks we've registered, as well. Once that's done, + * we can go ahead and exit. + */ + dsm_detach(seg); + proc_exit(0); +} + +/* + * Set up a dynamic shared memory segment. + * + * We set up a control region that contains a ApplyBgworkerShared, + * plus one region per message queue. There are as many message queues as + * the number of workers. + */ +static void +apply_bgworker_setup_dsm(WorkerState *wstate) +{ + shm_toc_estimator e; + int toc_key = 0; + Size segsize; + dsm_segment *seg; + shm_toc *toc; + ApplyBgworkerShared *pst; + shm_mq *mq; + int64 queue_size = 160000000; /* 16 MB for now */ + + /* + * Estimate how much shared memory we need. + * + * Because the TOC machinery may choose to insert padding of oddly-sized + * requests, we must estimate each chunk separately. + * + * We need one key to register the location of the header, and we need + * another key to track the location of the message queue. + */ + shm_toc_initialize_estimator(&e); + shm_toc_estimate_chunk(&e, sizeof(ApplyBgworkerShared)); + shm_toc_estimate_chunk(&e, (Size) queue_size); + + shm_toc_estimate_keys(&e, 1 + 1); + segsize = shm_toc_estimate(&e); + + /* Create the shared memory segment and establish a table of contents. */ + seg = dsm_create(shm_toc_estimate(&e), 0); + toc = shm_toc_create(PG_LOGICAL_APPLY_SHM_MAGIC, dsm_segment_address(seg), + segsize); + + /* Set up the header region. */ + pst = shm_toc_allocate(toc, sizeof(ApplyBgworkerShared)); + SpinLockInit(&pst->mutex); + pst->state = APPLY_BGWORKER_BUSY; + pst->stream_xid = stream_xid; + pst->n = list_length(ApplyWorkersList) + 1; + + shm_toc_insert(toc, toc_key++, pst); + + /* Set up one message queue per worker, plus one. */ + mq = shm_mq_create(shm_toc_allocate(toc, (Size) queue_size), + (Size) queue_size); + shm_toc_insert(toc, toc_key++, mq); + shm_mq_set_sender(mq, MyProc); + + /* Attach the queues. */ + wstate->mq_handle = shm_mq_attach(mq, seg, NULL); + + /* Return results to caller. */ + wstate->dsm_seg = seg; + wstate->pstate = pst; +} + +/* + * Start apply worker background worker process and allocate shared memory for + * it. + */ +static WorkerState * +apply_bgworker_setup(void) +{ + MemoryContext oldcontext; + bool launched; + WorkerState *wstate; + + elog(LOG, "setting up apply worker #%u", list_length(ApplyWorkersList) + 1); + + oldcontext = MemoryContextSwitchTo(ApplyContext); + + wstate = (WorkerState *) palloc0(sizeof(WorkerState)); + + /* Setup shared memory */ + apply_bgworker_setup_dsm(wstate); + + launched = logicalrep_worker_launch(MyLogicalRepWorker->dbid, + MySubscription->oid, + MySubscription->name, + MyLogicalRepWorker->userid, + InvalidOid, + dsm_segment_handle(wstate->dsm_seg)); + + if (launched) + { + /* Wait for worker to attach. */ + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_ATTACHED); + + ApplyWorkersList = lappend(ApplyWorkersList, wstate); + } + else + { + pfree(wstate); + wstate = NULL; + } + + MemoryContextSwitchTo(oldcontext); + + return wstate; +} + +/* + * Send the data to the specified apply bgworker via shared-memory queue. + */ +static void +apply_bgworker_send_data(WorkerState *wstate, Size nbytes, const void *data) +{ + shm_mq_result result; + + result = shm_mq_send(wstate->mq_handle, nbytes, data, false, true); + + if (result != SHM_MQ_SUCCESS) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("could not send tuples to shared-memory queue"))); +} + +/* + * Wait until the state of apply background worker reaches the 'wait_for_state' + */ +static void +apply_bgworker_wait_for(WorkerState *wstate, char wait_for_state) +{ + for (;;) + { + char status; + + SpinLockAcquire(&wstate->pstate->mutex); + status = wstate->pstate->state; + SpinLockRelease(&wstate->pstate->mutex); + + /* Done if already in correct state. */ + if (status == wait_for_state) + break; + + /* If any workers (or the postmaster) have died, we have failed. */ + if (status == APPLY_BGWORKER_EXIT) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("Background worker %u failed to apply transaction %u", + wstate->pstate->n, wstate->pstate->stream_xid))); + + /* Wait to be signalled. */ + WaitLatch(MyLatch, WL_LATCH_SET | WL_EXIT_ON_PM_DEATH, 0, + WAIT_EVENT_LOGICAL_APPLY_WORKER_STATE_CHANGE); + + /* Reset the latch so we don't spin. */ + ResetLatch(MyLatch); + + /* An interrupt may have occurred while we were waiting. */ + CHECK_FOR_INTERRUPTS(); + } +} + +/* + * Check the status of workers and report an error if any apply background + * worker has exited unexpectedly. + * + * Exit if any relation is not in the READY state and if any worker is handling + * the streaming transaction at the same time. Because for streaming + * transactions that is being applied in apply bgworker, we cannot decide whether to + * apply the change for a relation that is not in the READY state (see + * should_apply_changes_for_rel) as we won't know remote_final_lsn by that + * time. + */ +static void +apply_bgworker_check_status(void) +{ + ListCell *lc; + + if (am_apply_bgworker() || MySubscription->stream != SUBSTREAM_APPLY) + return; + + foreach(lc, ApplyWorkersList) + { + WorkerState *wstate = (WorkerState *) lfirst(lc); + + /* + * We don't lock here as in the worst case we will just detect the + * failure of worker a bit later. + */ + if (wstate->pstate->state == APPLY_BGWORKER_EXIT) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("Background worker %u exited unexpectedly", + wstate->pstate->n))); + } + + if (list_length(ApplyWorkersIdleList) != list_length(ApplyWorkersList) && + !AllTablesyncsReady()) + { + ereport(LOG, + (errmsg("logical replication apply workers for subscription \"%s\" will restart", + MySubscription->name), + errdetail("Cannot handle streamed replication transaction by apply " + "bgworkers until all tables are synchronized"))); + + proc_exit(0); + } +} + +/* Set the state of apply background worker */ +static void +apply_bgworker_set_state(char state) +{ + if (!am_apply_bgworker()) + return; + + elog(LOG, "[Apply BGW #%u] set state to %c", MyParallelState->n, state); + + SpinLockAcquire(&MyParallelState->mutex); + MyParallelState->state = state; + SpinLockRelease(&MyParallelState->mutex); +} diff --git a/src/backend/utils/activity/wait_event.c b/src/backend/utils/activity/wait_event.c index 87c15b9..ba781e6 100644 --- a/src/backend/utils/activity/wait_event.c +++ b/src/backend/utils/activity/wait_event.c @@ -388,6 +388,9 @@ pgstat_get_wait_ipc(WaitEventIPC w) case WAIT_EVENT_HASH_GROW_BUCKETS_REINSERT: event_name = "HashGrowBucketsReinsert"; break; + case WAIT_EVENT_LOGICAL_APPLY_WORKER_STATE_CHANGE: + event_name = "LogicalApplyWorkerStateChange"; + break; case WAIT_EVENT_LOGICAL_SYNC_DATA: event_name = "LogicalSyncData"; break; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 7cc9c72..6f8b30a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -4450,7 +4450,7 @@ getSubscriptions(Archive *fout) if (fout->remoteVersion >= 140000) appendPQExpBufferStr(query, " s.substream,\n"); else - appendPQExpBufferStr(query, " false AS substream,\n"); + appendPQExpBufferStr(query, " 'f' AS substream,\n"); if (fout->remoteVersion >= 150000) appendPQExpBufferStr(query, @@ -4580,8 +4580,10 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo) if (strcmp(subinfo->subbinary, "t") == 0) appendPQExpBufferStr(query, ", binary = true"); - if (strcmp(subinfo->substream, "f") != 0) + if (strcmp(subinfo->substream, "t") == 0) appendPQExpBufferStr(query, ", streaming = on"); + else if (strcmp(subinfo->substream, "a") == 0) + appendPQExpBufferStr(query, ", streaming = apply"); if (strcmp(subinfo->subtwophasestate, two_phase_disabled) != 0) appendPQExpBufferStr(query, ", two_phase = on"); diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h index d1260f5..9b394a4 100644 --- a/src/include/catalog/pg_subscription.h +++ b/src/include/catalog/pg_subscription.h @@ -68,7 +68,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW bool subbinary; /* True if the subscription wants the * publisher to send data in binary */ - bool substream; /* Stream in-progress transactions. */ + char substream; /* Stream in-progress transactions. + * See SUBSTREAM_xxx constants. */ char subtwophasestate; /* Stream two-phase transactions */ @@ -109,7 +110,7 @@ typedef struct Subscription bool enabled; /* Indicates if the subscription is enabled */ bool binary; /* Indicates if the subscription wants data in * binary format */ - bool stream; /* Allow streaming in-progress transactions. */ + char stream; /* Allow streaming in-progress transactions. */ char twophasestate; /* Allow streaming two-phase transactions */ bool disableonerr; /* Indicates if the subscription should be * automatically disabled if a worker error @@ -120,6 +121,18 @@ typedef struct Subscription List *publications; /* List of publication names to subscribe to */ } Subscription; +/* Disallow streaming in-progress transactions */ +#define SUBSTREAM_OFF 'f' + +/* + * Streaming transactions are written to a temporary file and applied only + * after the transaction is committed on upstream. + */ +#define SUBSTREAM_ON 't' + +/* Streaming transactions are applied immediately via a background worker */ +#define SUBSTREAM_APPLY 'a' + extern Subscription *GetSubscription(Oid subid, bool missing_ok); extern void FreeSubscription(Subscription *sub); extern void DisableSubscription(Oid subid); diff --git a/src/include/replication/logicalworker.h b/src/include/replication/logicalworker.h index cd1b6e8..6a1af7f 100644 --- a/src/include/replication/logicalworker.h +++ b/src/include/replication/logicalworker.h @@ -13,6 +13,7 @@ #define LOGICALWORKER_H extern void ApplyWorkerMain(Datum main_arg); +extern void ApplyBgworkerMain(Datum main_arg); extern bool IsLogicalWorker(void); diff --git a/src/include/replication/origin.h b/src/include/replication/origin.h index 2d1b5e5..b270434 100644 --- a/src/include/replication/origin.h +++ b/src/include/replication/origin.h @@ -53,7 +53,7 @@ extern XLogRecPtr replorigin_get_progress(RepOriginId node, bool flush); extern void replorigin_session_advance(XLogRecPtr remote_commit, XLogRecPtr local_commit); -extern void replorigin_session_setup(RepOriginId node); +extern void replorigin_session_setup(RepOriginId node, bool acquire); extern void replorigin_session_reset(void); extern XLogRecPtr replorigin_session_get_progress(bool flush); diff --git a/src/include/replication/worker_internal.h b/src/include/replication/worker_internal.h index 901845a..6613e32 100644 --- a/src/include/replication/worker_internal.h +++ b/src/include/replication/worker_internal.h @@ -60,6 +60,8 @@ typedef struct LogicalRepWorker */ FileSet *stream_fileset; + bool subworker; + /* Stats. */ XLogRecPtr last_lsn; TimestampTz last_send_time; @@ -84,8 +86,9 @@ extern void logicalrep_worker_attach(int slot); extern LogicalRepWorker *logicalrep_worker_find(Oid subid, Oid relid, bool only_running); extern List *logicalrep_workers_find(Oid subid, bool only_running); -extern void logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, - Oid userid, Oid relid); +extern bool logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, + Oid userid, Oid relid, + dsm_handle subworker_dsm); extern void logicalrep_worker_stop(Oid subid, Oid relid); extern void logicalrep_worker_wakeup(Oid subid, Oid relid); extern void logicalrep_worker_wakeup_ptr(LogicalRepWorker *worker); @@ -109,4 +112,10 @@ am_tablesync_worker(void) return OidIsValid(MyLogicalRepWorker->relid); } +static inline bool +am_apply_bgworker(void) +{ + return MyLogicalRepWorker->subworker; +} + #endif /* WORKER_INTERNAL_H */ diff --git a/src/include/utils/wait_event.h b/src/include/utils/wait_event.h index b578e2e..c2d2a11 100644 --- a/src/include/utils/wait_event.h +++ b/src/include/utils/wait_event.h @@ -105,6 +105,7 @@ typedef enum WAIT_EVENT_HASH_GROW_BUCKETS_ALLOCATE, WAIT_EVENT_HASH_GROW_BUCKETS_ELECT, WAIT_EVENT_HASH_GROW_BUCKETS_REINSERT, + WAIT_EVENT_LOGICAL_APPLY_WORKER_STATE_CHANGE, WAIT_EVENT_LOGICAL_SYNC_DATA, WAIT_EVENT_LOGICAL_SYNC_STATE_CHANGE, WAIT_EVENT_MQ_INTERNAL, diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out index 7fcfad1..f769835 100644 --- a/src/test/regress/expected/subscription.out +++ b/src/test/regress/expected/subscription.out @@ -197,7 +197,7 @@ ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); DROP SUBSCRIPTION regress_testsub; -- fail - streaming must be boolean CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = foo); -ERROR: streaming requires a Boolean value +ERROR: streaming requires a Boolean value or "apply" -- now it works CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = true); WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index dd12149..9d89912 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -116,6 +116,8 @@ Append AppendPath AppendRelInfo AppendState +ApplyBgworkerShared +ApplyBgworkerState ApplyErrorCallbackArg ApplyExecutionData ApplySubXactData @@ -2994,11 +2996,13 @@ WordEntryPosVector WordEntryPosVector1 WorkTableScan WorkTableScanState +WorkerEntry WorkerInfo WorkerInfoData WorkerInstrumentation WorkerJobDumpPtrType WorkerJobRestorePtrType +WorkerState Working_State WriteBufPtrType WriteBytePtrType -- 2.7.2.windows.1 ^ permalink raw reply [nested|flat] 66+ messages in thread
* RE: Perform streaming logical transactions by background workers and parallel apply @ 2022-05-13 09:57 [email protected] <[email protected]> parent: Peter Smith <[email protected]> 0 siblings, 2 replies; 66+ messages in thread From: [email protected] @ 2022-05-13 09:57 UTC (permalink / raw) To: Peter Smith <[email protected]>; +Cc: [email protected] <[email protected]>; Amit Kapila <[email protected]>; PostgreSQL Hackers <[email protected]> On Fri, May 6, 2022 4:56 PM Peter Smith <[email protected]> wrote: > > Here are my review comments for v5-0002 (TAP tests) > > Your changes followed a similar pattern of refactoring so most of my > comments below is repeated for all the files. > Thanks for your comments. > ====== > > 1. Commit message > > For the tap tests about streaming option in logical replication, test both > 'on' and 'apply' option. > > SUGGESTION > Change all TAP tests using the PUBLICATION "streaming" option, so they > now test both 'on' and 'apply' values. > OK. But "streaming" is a subscription option, so I modified it to: Change all TAP tests using the SUBSCRIPTION "streaming" option, so they now test both 'on' and 'apply' values. > ~~~ > > 4. src/test/subscription/t/015_stream.pl > > +# Test streaming mode apply > +$node_publisher->safe_psql('postgres', "DELETE FROM test_tab WHERE (a > 2)"); > $node_publisher->wait_for_catchup($appname); > > I think those 2 lines do not really belong after the "# Test streaming > mode apply" comment. IIUC they are really just doing cleanup from the > prior test part so I think they should > > a) be *above* this comment (and say "# cleanup the test data") or > b) maybe it is best to put all the cleanup lines actually inside the > 'test_streaming' function so that the last thing the function does is > clean up after itself. > > option b seems tidier to me. > I also think option b seems better, so I put them inside test_streaming(). The rest of the comments are fixed as suggested. Besides, I noticed that we didn't free the background worker after preparing a transaction in the main patch, so made some small changes to fix it. Attach the updated patches. Regards, Shi yu Attachments: [application/octet-stream] v6-0001-Perform-streaming-logical-transactions-by-backgro.patch (74.3K, ../../OSZPR01MB63106EADF50E0E710D36CE5DFDCA9@OSZPR01MB6310.jpnprd01.prod.outlook.com/2-v6-0001-Perform-streaming-logical-transactions-by-backgro.patch) download | inline diff: From f3114faedf12b7aa343b6627b79115dc955dad4d Mon Sep 17 00:00:00 2001 From: "houzj.fnst" <[email protected]> Date: Wed, 20 Apr 2022 16:45:07 +0800 Subject: [PATCH v6 1/2] Perform streaming logical transactions by background workers Currently, for large transactions, the publisher sends the data in multiple streams (changes divided into chunks depending upon logical_decoding_work_mem), and then on the subscriber-side, the apply worker writes the changes into temporary files and once it receives the commit, it reads from the file and applies the entire transaction. To improve the performance of such transactions, we can instead allow them to be applied via background workers. In this approach, we assign a new apply background worker (if available) as soon as the xact's first stream is received and the main apply worker will send changes to this new worker via shared memory. The apply background worker will directly apply the change instead of writing it to temporary files. We keep this worker assigned till the transaction commit is received and also wait for the worker to finish at commit. This preserves commit ordering and avoids writing to and reading from file in most cases. We still need to spill if there is no worker available. We also need to allow stream_stop to complete by the apply background worker to finish it to avoid deadlocks because T-1's current stream of changes can update rows in conflicting order with T-2's next stream of changes. This patch also extends the subscription streaming option so that the user can control whether to apply the streaming transaction in an apply background worker or spill the change to disk. User can set the streaming option to 'on/off', 'apply'. For now, 'apply' means the streaming will be applied via a apply background worker if available. 'on' means the streaming transaction will be spilled to disk. --- doc/src/sgml/catalogs.sgml | 10 +- doc/src/sgml/ref/create_subscription.sgml | 24 +- src/backend/commands/subscriptioncmds.c | 66 +- src/backend/postmaster/bgworker.c | 3 + src/backend/replication/logical/launcher.c | 87 +- src/backend/replication/logical/origin.c | 6 +- src/backend/replication/logical/tablesync.c | 10 +- src/backend/replication/logical/worker.c | 1397 +++++++++++++++++-- src/backend/utils/activity/wait_event.c | 3 + src/bin/pg_dump/pg_dump.c | 6 +- src/include/catalog/pg_subscription.h | 17 +- src/include/replication/logicalworker.h | 1 + src/include/replication/origin.h | 2 +- src/include/replication/worker_internal.h | 13 +- src/include/utils/wait_event.h | 1 + src/test/regress/expected/subscription.out | 2 +- src/tools/pgindent/typedefs.list | 4 + 17 files changed, 1456 insertions(+), 196 deletions(-) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index a533a2153e..5e46c4bc20 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -7863,11 +7863,15 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l <row> <entry role="catalog_table_entry"><para role="column_definition"> - <structfield>substream</structfield> <type>bool</type> + <structfield>substream</structfield> <type>char</type> </para> <para> - If true, the subscription will allow streaming of in-progress - transactions + Controls how to handle the streaming of in-progress transactions: + <literal>f</literal> = disallow streaming of in-progress transactions, + <literal>t</literal> = spill the changes of in-progress transactions to + disk and apply at once after the transaction is committed on the + publisher, + <literal>a</literal> = apply changes directly using a background worker </para></entry> </row> diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml index 203bb41844..22bd81800d 100644 --- a/doc/src/sgml/ref/create_subscription.sgml +++ b/doc/src/sgml/ref/create_subscription.sgml @@ -217,13 +217,29 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl </varlistentry> <varlistentry> - <term><literal>streaming</literal> (<type>boolean</type>)</term> + <term><literal>streaming</literal> (<type>enum</type>)</term> <listitem> <para> Specifies whether to enable streaming of in-progress transactions - for this subscription. By default, all transactions - are fully decoded on the publisher and only then sent to the - subscriber as a whole. + for this subscription. The default value is <literal>off</literal>, + all transactions are fully decoded on the publisher and only then + sent to the subscriber as a whole. + </para> + + <para> + If set to <literal>on</literal>, the changes of transaction are + written to temporary files and then applied at once after the + transaction is committed on the publisher. + </para> + + <para> + If set to <literal>apply</literal> incoming + changes are directly applied via one of the background workers, if + available. If no background worker is free to handle streaming + transaction then the changes are written to a file and applied after + the transaction is committed. Note that if an error happens when + applying changes in a background worker, it might not report the + finish LSN of the remote transaction in the server log. </para> </listitem> </varlistentry> diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c index 690cdaa426..f154762d0d 100644 --- a/src/backend/commands/subscriptioncmds.c +++ b/src/backend/commands/subscriptioncmds.c @@ -83,7 +83,7 @@ typedef struct SubOpts bool copy_data; bool refresh; bool binary; - bool streaming; + char streaming; bool twophase; bool disableonerr; XLogRecPtr lsn; @@ -95,6 +95,62 @@ static List *merge_publications(List *oldpublist, List *newpublist, bool addpub, static void ReportSlotConnectionError(List *rstates, Oid subid, char *slotname, char *err); +/* + * Extract the streaming mode value from a DefElem. This is like + * defGetBoolean() but also accepts the special value and "apply". + */ +static char +defGetStreamingMode(DefElem *def) +{ + /* + * If no parameter given, assume "true" is meant. + */ + if (def->arg == NULL) + return SUBSTREAM_ON; + + /* + * Allow 0, 1, "true", "false", "on", "off" or "apply". + */ + switch (nodeTag(def->arg)) + { + case T_Integer: + switch (intVal(def->arg)) + { + case 0: + return SUBSTREAM_OFF; + case 1: + return SUBSTREAM_ON; + default: + /* otherwise, error out below */ + break; + } + break; + default: + { + char *sval = defGetString(def); + + /* + * The set of strings accepted here should match up with the + * grammar's opt_boolean_or_string production. + */ + if (pg_strcasecmp(sval, "false") == 0 || + pg_strcasecmp(sval, "off") == 0) + return SUBSTREAM_OFF; + if (pg_strcasecmp(sval, "true") == 0 || + pg_strcasecmp(sval, "on") == 0) + return SUBSTREAM_ON; + if (pg_strcasecmp(sval, "apply") == 0) + return SUBSTREAM_APPLY; + } + break; + } + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s requires a Boolean value or \"apply\"", + def->defname))); + return SUBSTREAM_OFF; /* keep compiler quiet */ +} + /* * Common option parsing function for CREATE and ALTER SUBSCRIPTION commands. * @@ -132,7 +188,7 @@ parse_subscription_options(ParseState *pstate, List *stmt_options, if (IsSet(supported_opts, SUBOPT_BINARY)) opts->binary = false; if (IsSet(supported_opts, SUBOPT_STREAMING)) - opts->streaming = false; + opts->streaming = SUBSTREAM_OFF; if (IsSet(supported_opts, SUBOPT_TWOPHASE_COMMIT)) opts->twophase = false; if (IsSet(supported_opts, SUBOPT_DISABLE_ON_ERR)) @@ -233,7 +289,7 @@ parse_subscription_options(ParseState *pstate, List *stmt_options, errorConflictingDefElem(defel, pstate); opts->specified_opts |= SUBOPT_STREAMING; - opts->streaming = defGetBoolean(defel); + opts->streaming = defGetStreamingMode(defel); } else if (strcmp(defel->defname, "two_phase") == 0) { @@ -601,7 +657,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt, values[Anum_pg_subscription_subowner - 1] = ObjectIdGetDatum(owner); values[Anum_pg_subscription_subenabled - 1] = BoolGetDatum(opts.enabled); values[Anum_pg_subscription_subbinary - 1] = BoolGetDatum(opts.binary); - values[Anum_pg_subscription_substream - 1] = BoolGetDatum(opts.streaming); + values[Anum_pg_subscription_substream - 1] = CharGetDatum(opts.streaming); values[Anum_pg_subscription_subtwophasestate - 1] = CharGetDatum(opts.twophase ? LOGICALREP_TWOPHASE_STATE_PENDING : @@ -1060,7 +1116,7 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt, if (IsSet(opts.specified_opts, SUBOPT_STREAMING)) { values[Anum_pg_subscription_substream - 1] = - BoolGetDatum(opts.streaming); + CharGetDatum(opts.streaming); replaces[Anum_pg_subscription_substream - 1] = true; } diff --git a/src/backend/postmaster/bgworker.c b/src/backend/postmaster/bgworker.c index 40601aefd9..40ccb8993c 100644 --- a/src/backend/postmaster/bgworker.c +++ b/src/backend/postmaster/bgworker.c @@ -128,6 +128,9 @@ static const struct }, { "ApplyWorkerMain", ApplyWorkerMain + }, + { + "ApplyBgworkerMain", ApplyBgworkerMain } }; diff --git a/src/backend/replication/logical/launcher.c b/src/backend/replication/logical/launcher.c index bd5f78cf9a..f8f049062c 100644 --- a/src/backend/replication/logical/launcher.c +++ b/src/backend/replication/logical/launcher.c @@ -73,6 +73,7 @@ static void logicalrep_launcher_onexit(int code, Datum arg); static void logicalrep_worker_onexit(int code, Datum arg); static void logicalrep_worker_detach(void); static void logicalrep_worker_cleanup(LogicalRepWorker *worker); +static void logicalrep_worker_stop_internal(LogicalRepWorker *worker); static bool on_commit_launcher_wakeup = false; @@ -223,6 +224,10 @@ logicalrep_worker_find(Oid subid, Oid relid, bool only_running) { LogicalRepWorker *w = &LogicalRepCtx->workers[i]; + /* We only need main apply worker or table sync worker here */ + if (w->subworker) + continue; + if (w->in_use && w->subid == subid && w->relid == relid && (!only_running || w->proc)) { @@ -259,11 +264,11 @@ logicalrep_workers_find(Oid subid, bool only_running) } /* - * Start new apply background worker, if possible. + * Start new background worker, if possible. */ -void +bool logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, Oid userid, - Oid relid) + Oid relid, dsm_handle subworker_dsm) { BackgroundWorker bgw; BackgroundWorkerHandle *bgw_handle; @@ -273,6 +278,10 @@ logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, Oid userid, LogicalRepWorker *worker = NULL; int nsyncworkers; TimestampTz now; + bool is_subworker = (subworker_dsm != DSM_HANDLE_INVALID); + + /* We don't support table sync in subworker */ + Assert(!(is_subworker && OidIsValid(relid))); ereport(DEBUG1, (errmsg_internal("starting logical replication worker for subscription \"%s\"", @@ -350,7 +359,7 @@ retry: if (OidIsValid(relid) && nsyncworkers >= max_sync_workers_per_subscription) { LWLockRelease(LogicalRepWorkerLock); - return; + return false; } /* @@ -364,7 +373,7 @@ retry: (errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED), errmsg("out of logical replication worker slots"), errhint("You might need to increase max_logical_replication_workers."))); - return; + return false; } /* Prepare the worker slot. */ @@ -379,6 +388,7 @@ retry: worker->relstate = SUBREL_STATE_UNKNOWN; worker->relstate_lsn = InvalidXLogRecPtr; worker->stream_fileset = NULL; + worker->subworker = is_subworker; worker->last_lsn = InvalidXLogRecPtr; TIMESTAMP_NOBEGIN(worker->last_send_time); TIMESTAMP_NOBEGIN(worker->last_recv_time); @@ -396,19 +406,31 @@ retry: BGWORKER_BACKEND_DATABASE_CONNECTION; bgw.bgw_start_time = BgWorkerStart_RecoveryFinished; snprintf(bgw.bgw_library_name, BGW_MAXLEN, "postgres"); - snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyWorkerMain"); + + if (subworker_dsm == DSM_HANDLE_INVALID) + snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyWorkerMain"); + else + snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyBgworkerMain"); + + if (OidIsValid(relid)) snprintf(bgw.bgw_name, BGW_MAXLEN, "logical replication worker for subscription %u sync %u", subid, relid); - else + else if (subworker_dsm == DSM_HANDLE_INVALID) snprintf(bgw.bgw_name, BGW_MAXLEN, "logical replication worker for subscription %u", subid); + else + snprintf(bgw.bgw_name, BGW_MAXLEN, + "logical replication apply worker for subscription %u", subid); snprintf(bgw.bgw_type, BGW_MAXLEN, "logical replication worker"); bgw.bgw_restart_time = BGW_NEVER_RESTART; bgw.bgw_notify_pid = MyProcPid; bgw.bgw_main_arg = Int32GetDatum(slot); + if (is_subworker) + memcpy(bgw.bgw_extra, &subworker_dsm, sizeof(dsm_handle)); + if (!RegisterDynamicBackgroundWorker(&bgw, &bgw_handle)) { /* Failed to start worker, so clean up the worker slot. */ @@ -421,11 +443,13 @@ retry: (errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED), errmsg("out of background worker slots"), errhint("You might need to increase max_worker_processes."))); - return; + return false; } /* Now wait until it attaches. */ WaitForReplicationWorkerAttach(worker, generation, bgw_handle); + + return true; } /* @@ -436,7 +460,6 @@ void logicalrep_worker_stop(Oid subid, Oid relid) { LogicalRepWorker *worker; - uint16 generation; LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); @@ -449,6 +472,22 @@ logicalrep_worker_stop(Oid subid, Oid relid) return; } + logicalrep_worker_stop_internal(worker); + + LWLockRelease(LogicalRepWorkerLock); +} + +/* + * Workhorse for logicalrep_worker_stop() and logicalrep_worker_detach(). Stop + * the worker and wait for it to die. + */ +static void +logicalrep_worker_stop_internal(LogicalRepWorker *worker) +{ + uint16 generation; + + Assert(LWLockHeldByMe(LogicalRepWorkerLock)); + /* * Remember which generation was our worker so we can check if what we see * is still the same one. @@ -485,10 +524,7 @@ logicalrep_worker_stop(Oid subid, Oid relid) * different, meaning that a different worker has taken the slot. */ if (!worker->in_use || worker->generation != generation) - { - LWLockRelease(LogicalRepWorkerLock); return; - } /* Worker has assigned proc, so it has started. */ if (worker->proc) @@ -522,8 +558,6 @@ logicalrep_worker_stop(Oid subid, Oid relid) LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); } - - LWLockRelease(LogicalRepWorkerLock); } /* @@ -599,6 +633,28 @@ logicalrep_worker_attach(int slot) static void logicalrep_worker_detach(void) { + /* + * If we are the main apply worker, stop all the apply background workers + * we started before. + */ + if (!MyLogicalRepWorker->subworker) + { + List *workers; + ListCell *lc; + + LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); + + workers = logicalrep_workers_find(MyLogicalRepWorker->subid, true); + foreach(lc, workers) + { + LogicalRepWorker *w = (LogicalRepWorker *) lfirst(lc); + if (w->subworker) + logicalrep_worker_stop_internal(w); + } + + LWLockRelease(LogicalRepWorkerLock); + } + /* Block concurrent access. */ LWLockAcquire(LogicalRepWorkerLock, LW_EXCLUSIVE); @@ -621,6 +677,7 @@ logicalrep_worker_cleanup(LogicalRepWorker *worker) worker->userid = InvalidOid; worker->subid = InvalidOid; worker->relid = InvalidOid; + worker->subworker = false; } /* @@ -868,7 +925,7 @@ ApplyLauncherMain(Datum main_arg) wait_time = wal_retrieve_retry_interval; logicalrep_worker_launch(sub->dbid, sub->oid, sub->name, - sub->owner, InvalidOid); + sub->owner, InvalidOid, DSM_HANDLE_INVALID); } } diff --git a/src/backend/replication/logical/origin.c b/src/backend/replication/logical/origin.c index 21937ab2d3..aaa66947e0 100644 --- a/src/backend/replication/logical/origin.c +++ b/src/backend/replication/logical/origin.c @@ -1068,7 +1068,7 @@ ReplicationOriginExitCleanup(int code, Datum arg) * with replorigin_session_reset(). */ void -replorigin_session_setup(RepOriginId node) +replorigin_session_setup(RepOriginId node, bool acquire) { static bool registered_cleanup; int i; @@ -1110,7 +1110,7 @@ replorigin_session_setup(RepOriginId node) if (curstate->roident != node) continue; - else if (curstate->acquired_by != 0) + else if (curstate->acquired_by != 0 && acquire) { ereport(ERROR, (errcode(ERRCODE_OBJECT_IN_USE), @@ -1321,7 +1321,7 @@ pg_replication_origin_session_setup(PG_FUNCTION_ARGS) name = text_to_cstring((text *) DatumGetPointer(PG_GETARG_DATUM(0))); origin = replorigin_by_name(name, false); - replorigin_session_setup(origin); + replorigin_session_setup(origin, true); replorigin_session_origin = origin; diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c index b03e0f5aac..b9be9d0590 100644 --- a/src/backend/replication/logical/tablesync.c +++ b/src/backend/replication/logical/tablesync.c @@ -568,7 +568,8 @@ process_syncing_tables_for_apply(XLogRecPtr current_lsn) MySubscription->oid, MySubscription->name, MyLogicalRepWorker->userid, - rstate->relid); + rstate->relid, + DSM_HANDLE_INVALID); hentry->last_start_time = now; } } @@ -589,6 +590,9 @@ process_syncing_tables_for_apply(XLogRecPtr current_lsn) void process_syncing_tables(XLogRecPtr current_lsn) { + if (MyLogicalRepWorker->subworker) + return; + if (am_tablesync_worker()) process_syncing_tables_for_sync(current_lsn); else @@ -1275,7 +1279,7 @@ LogicalRepSyncTableStart(XLogRecPtr *origin_startpos) * time this tablesync was launched. */ originid = replorigin_by_name(originname, false); - replorigin_session_setup(originid); + replorigin_session_setup(originid, true); replorigin_session_origin = originid; *origin_startpos = replorigin_session_get_progress(false); @@ -1386,7 +1390,7 @@ LogicalRepSyncTableStart(XLogRecPtr *origin_startpos) true /* go backward */ , true /* WAL log */ ); UnlockRelationOid(ReplicationOriginRelationId, RowExclusiveLock); - replorigin_session_setup(originid); + replorigin_session_setup(originid, true); replorigin_session_origin = originid; } else diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c index fc210a9e7b..fd47c48524 100644 --- a/src/backend/replication/logical/worker.c +++ b/src/backend/replication/logical/worker.c @@ -22,8 +22,25 @@ * STREAMED TRANSACTIONS * --------------------- * Streamed transactions (large transactions exceeding a memory limit on the - * upstream) are not applied immediately, but instead, the data is written - * to temporary files and then applied at once when the final commit arrives. + * upstream) are applied via one of two approaches. + * + * 1) Separate background workers + * + * Assign a new bgworker (if available) as soon as the xact's first stream is + * received and the main apply worker will send changes to this new worker via + * shared memory. We keep this worker assigned till the transaction commit is + * received and also wait for the worker to finish at commit. This preserves + * commit ordering and avoids writing to and reading from file in most cases. + * We still need to spill if there is no worker available. We also need to + * allow stream_stop to complete by the background worker to finish it to avoid + * deadlocks because T-1's current stream of changes can update rows in + * conflicting order with T-2's next stream of changes. + * + * 2) Write to temporary files and apply when the final commit arrives + * + * If no worker is available to handle streamed transaction, the data is + * written to temporary files and then applied at once when the final commit + * arrives. * * Unlike the regular (non-streamed) case, handling streamed transactions has * to handle aborts of both the toplevel transaction and subtransactions. This @@ -174,11 +191,15 @@ #include "rewrite/rewriteHandler.h" #include "storage/buffile.h" #include "storage/bufmgr.h" +#include "storage/dsm.h" #include "storage/fd.h" #include "storage/ipc.h" #include "storage/lmgr.h" #include "storage/proc.h" #include "storage/procarray.h" +#include "storage/shm_mq.h" +#include "storage/shm_toc.h" +#include "storage/spin.h" #include "tcop/tcopprot.h" #include "utils/acl.h" #include "utils/builtins.h" @@ -198,6 +219,75 @@ #define NAPTIME_PER_CYCLE 1000 /* max sleep time between cycles (1s) */ +#define PG_LOGICAL_APPLY_SHM_MAGIC 0x79fb2447 // TODO Consider change + +/* + * States for apply background worker. + */ +typedef enum ApplyBgworkerState +{ + APPLY_BGWORKER_ATTACHED = 0, + APPLY_BGWORKER_READY, + APPLY_BGWORKER_BUSY, + APPLY_BGWORKER_FINISHED, + APPLY_BGWORKER_EXIT +} ApplyBgworkerState; + +/* + * Shared information among apply workers. + */ +typedef struct ApplyBgworkerShared +{ + slock_t mutex; + + /* State for apply background worker. */ + ApplyBgworkerState state; + + TransactionId stream_xid; + uint32 n; /* id of apply background worker */ +} ApplyBgworkerShared; + +/* + * Struct for maintaining an apply background worker. + */ +typedef struct WorkerState +{ + shm_mq_handle *mq_handle; + dsm_segment *dsm_seg; + ApplyBgworkerShared volatile *pstate; +} WorkerState; + +typedef struct WorkerEntry +{ + TransactionId xid; + WorkerState *wstate; +} WorkerEntry; + +/* Apply background workers hash table (initialized on first use) */ +static HTAB *ApplyWorkersHash = NULL; +static List *ApplyWorkersIdleList = NIL; +static List *ApplyWorkersList = NIL; + +/* Fields valid only for apply background workers */ +volatile ApplyBgworkerShared *MyParallelState = NULL; +static List *subxactlist = NIL; + +/* The number of changes during one streaming block */ +static uint32 nchanges = 0; + +/* Worker setup and interactions */ +static WorkerState *apply_bgworker_setup(void); +static WorkerState *apply_bgworker_find_or_start(TransactionId xid, + bool start); +static void apply_bgworker_setup_dsm(WorkerState *wstate); +static void apply_bgworker_wait_for(WorkerState *wstate, + char wait_for_state); +static void apply_bgworker_send_data(WorkerState *wstate, Size nbytes, + const void *data); +static void apply_bgworker_free(WorkerState *wstate); +static void apply_bgworker_check_status(void); +static void apply_bgworker_set_state(char state); + typedef struct FlushPosition { dlist_node node; @@ -260,18 +350,23 @@ static XLogRecPtr remote_final_lsn = InvalidXLogRecPtr; static bool in_streamed_transaction = false; static TransactionId stream_xid = InvalidTransactionId; +static WorkerState *stream_apply_worker = NULL; + +/* check if we apply transaction in apply bgworker */ +#define apply_bgworker_active() (in_streamed_transaction && stream_apply_worker != NULL) /* * We enable skipping all data modification changes (INSERT, UPDATE, etc.) for * the subscription if the remote transaction's finish LSN matches the subskiplsn. * Once we start skipping changes, we don't stop it until we skip all changes of * the transaction even if pg_subscription is updated and MySubscription->skiplsn - * gets changed or reset during that. Also, in streaming transaction cases, we - * don't skip receiving and spooling the changes since we decide whether or not + * gets changed or reset during that. Also, in streaming transaction cases (streaming = on), + * we don't skip receiving and spooling the changes since we decide whether or not * to skip applying the changes when starting to apply changes. The subskiplsn is * cleared after successfully skipping the transaction or applying non-empty * transaction. The latter prevents the mistakenly specified subskiplsn from - * being left. + * being left. Note that we cannot skip the streaming transaction in apply mode, + * because we cannot get the finish LSN before applying the changes. */ static XLogRecPtr skip_xact_finish_lsn = InvalidXLogRecPtr; #define is_skipping_changes() (unlikely(!XLogRecPtrIsInvalid(skip_xact_finish_lsn))) @@ -426,43 +521,103 @@ end_replication_step(void) } /* - * Handle streamed transactions. + * Handle streamed transactions for both main apply worker and apply background + * worker. * - * If in streaming mode (receiving a block of streamed transaction), we - * simply redirect it to a file for the proper toplevel transaction. + * For the main apply worker, if in streaming mode (receiving a block of + * streamed transaction), we send the data to the apply background worker. + * + * For the apply background worker, define a savepoint if new subtransaction + * was started. * * Returns true for streamed transactions, false otherwise (regular mode). */ static bool handle_streamed_transaction(LogicalRepMsgType action, StringInfo s) { - TransactionId xid; + TransactionId current_xid = InvalidTransactionId; - /* not in streaming mode */ - if (!in_streamed_transaction) + /* + * Return if we are not in streaming mode and are not in an apply + * background worker. + */ + if (!in_streamed_transaction && !am_apply_bgworker()) return false; - Assert(stream_fd != NULL); Assert(TransactionIdIsValid(stream_xid)); /* * We should have received XID of the subxact as the first part of the * message, so extract it. */ - xid = pq_getmsgint(s, 4); + current_xid = pq_getmsgint(s, 4); - if (!TransactionIdIsValid(xid)) + if (!TransactionIdIsValid(current_xid)) ereport(ERROR, (errcode(ERRCODE_PROTOCOL_VIOLATION), errmsg_internal("invalid transaction ID in streamed replication transaction"))); - /* Add the new subxact to the array (unless already there). */ - subxact_info_add(xid); + if (am_apply_bgworker()) + { + /* + * Inside apply background worker we can figure out that new + * subtransaction was started if new change arrived with different xid. + * In that case we can define named savepoint, so that we were able to + * commit/rollback it separately later. + * + * Special case is if the first change comes from subtransaction, then + * we check that current_xid differs from stream_xid. + */ + if (current_xid != stream_xid && + !list_member_int(subxactlist, (int) current_xid)) + { + MemoryContext oldctx; + char spname[MAXPGPATH]; + + snprintf(spname, MAXPGPATH, "savepoint_for_xid_%u", current_xid); + + elog(LOG, "[Apply BGW #%u] defining savepoint %s", + MyParallelState->n, spname); + + DefineSavepoint(spname); + CommitTransactionCommand(); + + oldctx = MemoryContextSwitchTo(ApplyContext); + subxactlist = lappend_int(subxactlist, (int) current_xid); + MemoryContextSwitchTo(oldctx); + } + } + else if (apply_bgworker_active()) + { + /* + * If we decided to apply the changes of this transaction in an apply + * background worker, pass the data to the worker. + */ + apply_bgworker_send_data(stream_apply_worker, s->len, s->data); + nchanges += 1; + + /* + * XXX The publisher side doesn't always send relation update messages + * after the streaming transaction, so update the relation in main + * apply worker here. + */ + if (action == LOGICAL_REP_MSG_RELATION) + { + LogicalRepRelation *rel = logicalrep_read_rel(s); + logicalrep_relmap_update(rel); + } + + } + else + { + /* Add the new subxact to the array (unless already there). */ + subxact_info_add(current_xid); - /* write the change to the current file */ - stream_write_change(action, s); + /* write the change to the current file */ + stream_write_change(action, s); + } - return true; + return !am_apply_bgworker(); } /* @@ -844,6 +999,9 @@ apply_handle_commit(StringInfo s) apply_handle_commit_internal(&commit_data); + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); + /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(commit_data.end_lsn); @@ -898,7 +1056,9 @@ apply_handle_prepare_internal(LogicalRepPreparedTxnData *prepare_data) * BeginTransactionBlock is necessary to balance the EndTransactionBlock * called within the PrepareTransactionBlock below. */ - BeginTransactionBlock(); + if (!IsTransactionBlock()) + BeginTransactionBlock(); + CommitTransactionCommand(); /* Completes the preceding Begin command. */ /* @@ -950,6 +1110,9 @@ apply_handle_prepare(StringInfo s) in_remote_transaction = false; + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); + /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(prepare_data.end_lsn); @@ -1063,11 +1226,130 @@ apply_handle_rollback_prepared(StringInfo s) } /* - * Handle STREAM PREPARE. + * Look up worker inside ApplyWorkersHash for requested xid. * - * Logic is in two parts: - * 1. Replay all the spooled operations - * 2. Mark the transaction as prepared + * If start flag is true, try to start a new worker if not found in hash table. + */ +static WorkerState * +apply_bgworker_find_or_start(TransactionId xid, bool start) +{ + bool found; + WorkerState *wstate; + WorkerEntry *entry = NULL; + + if (!TransactionIdIsValid(xid)) + return NULL; + + /* + * We don't start new background worker if we are not in streaming apply + * mode. + */ + if (MySubscription->stream != SUBSTREAM_APPLY) + return NULL; + + /* + * We don't start new background worker if user has set skiplsn as it's + * possible that user want to skip the streaming transaction. For streaming + * transaction, we need to spill the transaction to disk so that we can get + * the last LSN of the transaction to judge whether to skip before starting + * to apply the change. + */ + if (start && !XLogRecPtrIsInvalid(MySubscription->skiplsn)) + return NULL; + + /* + * For streaming transactions that are being applied in bgworker, we cannot + * decide whether to apply the change for a relation that is not in the + * READY state (see should_apply_changes_for_rel) as we won't know + * remote_final_lsn by that time. So, we don't start new bgworker in this + * case. + */ + if (start && !AllTablesyncsReady()) + return NULL; + + /* First time through, initialize apply workers hashtable */ + if (ApplyWorkersHash == NULL) + { + HASHCTL ctl; + + MemSet(&ctl, 0, sizeof(ctl)); + ctl.keysize = sizeof(TransactionId); + ctl.entrysize = sizeof(WorkerEntry); + ctl.hcxt = ApplyContext; + + ApplyWorkersHash = hash_create("logical apply workers hash", 8, &ctl, + HASH_ELEM | HASH_BLOBS | HASH_CONTEXT); + } + + /* + * Find entry for requested transaction. + */ + entry = hash_search(ApplyWorkersHash, &xid, start ? HASH_ENTER : HASH_FIND, + &found); + if (found) + { + entry->wstate->pstate->state = APPLY_BGWORKER_BUSY; + return entry->wstate; + } + else if (!start) + return NULL; + + /* If there is at least one worker in the idle list, then take one. */ + if (list_length(ApplyWorkersIdleList) > 0) + { + wstate = (WorkerState *) llast(ApplyWorkersIdleList); + ApplyWorkersIdleList = list_delete_last(ApplyWorkersIdleList); + } + else + { + wstate = apply_bgworker_setup(); + + if (wstate == NULL) + { + /* + * If the bgworker cannot be launched, remove entry in hash table. + */ + hash_search(ApplyWorkersHash, &xid, HASH_REMOVE, &found); + return NULL; + } + } + + /* Fill up the hash entry */ + wstate->pstate->state = APPLY_BGWORKER_BUSY; + wstate->pstate->stream_xid = xid; + entry->wstate = wstate; + entry->xid = xid; + + return wstate; +} + +/* + * Add the worker to the freelist and remove the entry from hash table. + */ +static void +apply_bgworker_free(WorkerState *wstate) +{ + bool found; + MemoryContext oldctx; + TransactionId xid = wstate->pstate->stream_xid; + + Assert(wstate->pstate->state == APPLY_BGWORKER_FINISHED); + + oldctx = MemoryContextSwitchTo(ApplyContext); + + hash_search(ApplyWorkersHash, &xid, + HASH_REMOVE, &found); + + elog(LOG, "adding finished apply worker #%u for xid %u to the idle list", + wstate->pstate->n, wstate->pstate->stream_xid); + + ApplyWorkersIdleList = lappend(ApplyWorkersIdleList, wstate); + + MemoryContextSwitchTo(oldctx); +} + +/* + * Handle STREAM PREPARE. */ static void apply_handle_stream_prepare(StringInfo s) @@ -1088,24 +1370,69 @@ apply_handle_stream_prepare(StringInfo s) logicalrep_read_stream_prepare(s, &prepare_data); set_apply_error_context_xact(prepare_data.xid, prepare_data.prepare_lsn); - elog(DEBUG1, "received prepare for streamed transaction %u", prepare_data.xid); + if (am_apply_bgworker()) + { + elog(LOG, "received prepare for streamed transaction %u", prepare_data.xid); - /* Replay all the spooled operations. */ - apply_spooled_messages(prepare_data.xid, prepare_data.prepare_lsn); + /* Mark the transaction as prepared. */ + apply_handle_prepare_internal(&prepare_data); - /* Mark the transaction as prepared. */ - apply_handle_prepare_internal(&prepare_data); + CommitTransactionCommand(); - CommitTransactionCommand(); + pgstat_report_stat(false); - pgstat_report_stat(false); + list_free(subxactlist); + subxactlist = NIL; - store_flush_position(prepare_data.end_lsn); + apply_bgworker_set_state(APPLY_BGWORKER_FINISHED); + } + else + { + WorkerState *wstate = apply_bgworker_find_or_start(prepare_data.xid, false); + + /* + * If we are in main apply worker, check if we are processing this + * transaction in a bgworker. + */ + if (wstate) + { + apply_bgworker_send_data(wstate, s->len, s->data); + + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_FINISHED); + apply_bgworker_free(wstate); + + pgstat_report_stat(false); + store_flush_position(prepare_data.end_lsn); + } + else + { + /* + * If we are in main apply worker and the transaction has been + * serialized to file, replay all the spooled operations. + */ + apply_spooled_messages(prepare_data.xid, prepare_data.prepare_lsn); + + /* Mark the transaction as prepared. */ + apply_handle_prepare_internal(&prepare_data); + + CommitTransactionCommand(); + + pgstat_report_stat(false); + + store_flush_position(prepare_data.end_lsn); + + in_remote_transaction = false; + + /* unlink the files with serialized changes and subxact info. */ + stream_cleanup_files(MyLogicalRepWorker->subid, prepare_data.xid); + } + } in_remote_transaction = false; + stream_apply_worker = NULL; - /* unlink the files with serialized changes and subxact info. */ - stream_cleanup_files(MyLogicalRepWorker->subid, prepare_data.xid); + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(prepare_data.end_lsn); @@ -1155,15 +1482,6 @@ apply_handle_stream_start(StringInfo s) (errcode(ERRCODE_PROTOCOL_VIOLATION), errmsg_internal("duplicate STREAM START message"))); - /* - * Start a transaction on stream start, this transaction will be committed - * on the stream stop unless it is a tablesync worker in which case it - * will be committed after processing all the messages. We need the - * transaction for handling the buffile, used for serializing the - * streaming data and subxact info. - */ - begin_replication_step(); - /* notify handle methods we're processing a remote transaction */ in_streamed_transaction = true; @@ -1177,36 +1495,90 @@ apply_handle_stream_start(StringInfo s) set_apply_error_context_xact(stream_xid, InvalidXLogRecPtr); + if (am_apply_bgworker()) + { + /* + * Make sure the handle apply_dispatch methods are aware we're in a + * remote transaction. + */ + in_remote_transaction = true; + + /* If we are in a bgworker, begin the transaction */ + AcceptInvalidationMessages(); + maybe_reread_subscription(); + + StartTransactionCommand(); + BeginTransactionBlock(); + CommitTransactionCommand(); + + return; + } + /* - * Initialize the worker's stream_fileset if we haven't yet. This will be - * used for the entire duration of the worker so create it in a permanent - * context. We create this on the very first streaming message from any - * transaction and then use it for this and other streaming transactions. - * Now, we could create a fileset at the start of the worker as well but - * then we won't be sure that it will ever be used. + * If we are in main apply worker, check if there is any free bgworker + * we can use to process this transaction. */ - if (MyLogicalRepWorker->stream_fileset == NULL) + stream_apply_worker = apply_bgworker_find_or_start(stream_xid, first_segment); + + if (stream_apply_worker) { - MemoryContext oldctx; + /* + * If we have free worker or we already started to apply this + * transaction in bgworker, we pass the data to worker. + */ + if (first_segment) + apply_bgworker_send_data(stream_apply_worker, s->len, s->data); - oldctx = MemoryContextSwitchTo(ApplyContext); + nchanges = 0; + elog(LOG, "starting streaming of xid %u", stream_xid); + } - MyLogicalRepWorker->stream_fileset = palloc(sizeof(FileSet)); - FileSetInit(MyLogicalRepWorker->stream_fileset); + /* + * If no worker is available for the first stream start, we start to + * serialize all the changes of the transaction. + */ + else + { + /* + * Start a transaction on stream start, this transaction will be committed + * on the stream stop unless it is a tablesync worker in which case it + * will be committed after processing all the messages. We need the + * transaction for handling the buffile, used for serializing the + * streaming data and subxact info. + */ + begin_replication_step(); - MemoryContextSwitchTo(oldctx); - } + /* + * Initialize the worker's stream_fileset if we haven't yet. This will be + * used for the entire duration of the worker so create it in a permanent + * context. We create this on the very first streaming message from any + * transaction and then use it for this and other streaming transactions. + * Now, we could create a fileset at the start of the worker as well but + * then we won't be sure that it will ever be used. + */ + if (MyLogicalRepWorker->stream_fileset == NULL) + { + MemoryContext oldctx; - /* open the spool file for this transaction */ - stream_open_file(MyLogicalRepWorker->subid, stream_xid, first_segment); + oldctx = MemoryContextSwitchTo(ApplyContext); - /* if this is not the first segment, open existing subxact file */ - if (!first_segment) - subxact_info_read(MyLogicalRepWorker->subid, stream_xid); + MyLogicalRepWorker->stream_fileset = palloc(sizeof(FileSet)); + FileSetInit(MyLogicalRepWorker->stream_fileset); - pgstat_report_activity(STATE_RUNNING, NULL); + MemoryContextSwitchTo(oldctx); + } - end_replication_step(); + /* open the spool file for this transaction */ + stream_open_file(MyLogicalRepWorker->subid, stream_xid, first_segment); + + /* if this is not the first segment, open existing subxact file */ + if (!first_segment) + subxact_info_read(MyLogicalRepWorker->subid, stream_xid); + + end_replication_step(); + } + + pgstat_report_activity(STATE_RUNNING, NULL); } /* @@ -1220,44 +1592,47 @@ apply_handle_stream_stop(StringInfo s) (errcode(ERRCODE_PROTOCOL_VIOLATION), errmsg_internal("STREAM STOP message without STREAM START"))); - /* - * Close the file with serialized changes, and serialize information about - * subxacts for the toplevel transaction. - */ - subxact_info_write(MyLogicalRepWorker->subid, stream_xid); - stream_close_file(); + if (apply_bgworker_active()) + { + char action = LOGICAL_REP_MSG_STREAM_STOP; - /* We must be in a valid transaction state */ - Assert(IsTransactionState()); + apply_bgworker_send_data(stream_apply_worker, 1, &action); + apply_bgworker_wait_for(stream_apply_worker, APPLY_BGWORKER_READY); - /* Commit the per-stream transaction */ - CommitTransactionCommand(); + elog(LOG, "stopped streaming of xid %u, %u changes streamed", stream_xid, nchanges); + } + else + { + /* + * Close the file with serialized changes, and serialize information + * about subxacts for the toplevel transaction. + */ + subxact_info_write(MyLogicalRepWorker->subid, stream_xid); + stream_close_file(); - in_streamed_transaction = false; + /* We must be in a valid transaction state */ + Assert(IsTransactionState()); + + /* Commit the per-stream transaction */ + CommitTransactionCommand(); + + /* Reset per-stream context */ + MemoryContextReset(LogicalStreamingContext); + } - /* Reset per-stream context */ - MemoryContextReset(LogicalStreamingContext); + in_streamed_transaction = false; + stream_apply_worker = NULL; pgstat_report_activity(STATE_IDLE, NULL); reset_apply_error_context_info(); } /* - * Handle STREAM abort message. + * Handle STREAM ABORT message when the transaction was spilled to disk. */ static void -apply_handle_stream_abort(StringInfo s) +serialize_stream_abort(TransactionId xid, TransactionId subxid) { - TransactionId xid; - TransactionId subxid; - - if (in_streamed_transaction) - ereport(ERROR, - (errcode(ERRCODE_PROTOCOL_VIOLATION), - errmsg_internal("STREAM ABORT message without STREAM STOP"))); - - logicalrep_read_stream_abort(s, &xid, &subxid); - /* * If the two XIDs are the same, it's in fact abort of toplevel xact, so * just delete the files with serialized info. @@ -1339,51 +1714,163 @@ apply_handle_stream_abort(StringInfo s) end_replication_step(); CommitTransactionCommand(); } - - reset_apply_error_context_info(); } /* - * Common spoolfile processing. + * Handle STREAM ABORT message. */ static void -apply_spooled_messages(TransactionId xid, XLogRecPtr lsn) +apply_handle_stream_abort(StringInfo s) { - StringInfoData s2; - int nchanges; - char path[MAXPGPATH]; - char *buffer = NULL; - MemoryContext oldcxt; - BufFile *fd; + TransactionId xid; + TransactionId subxid; - maybe_start_skipping_changes(lsn); + if (in_streamed_transaction) + ereport(ERROR, + (errcode(ERRCODE_PROTOCOL_VIOLATION), + errmsg_internal("STREAM COMMIT message without STREAM STOP"))); - /* Make sure we have an open transaction */ - begin_replication_step(); + logicalrep_read_stream_abort(s, &xid, &subxid); - /* - * Allocate file handle and memory required to process all the messages in - * TopTransactionContext to avoid them getting reset after each message is - * processed. - */ - oldcxt = MemoryContextSwitchTo(TopTransactionContext); + if (am_apply_bgworker()) + { + ereport(LOG, + (errmsg("[Apply BGW #%u] aborting current transaction xid=%u, subxid=%u", + MyParallelState->n, GetCurrentTransactionIdIfAny(), GetCurrentSubTransactionId()))); - /* Open the spool file for the committed/prepared transaction */ - changes_filename(path, MyLogicalRepWorker->subid, xid); - elog(DEBUG1, "replaying changes from file \"%s\"", path); + /* + * If the two XIDs are the same, it's in fact abort of toplevel xact, + * so just free the subxactlist. + */ + if (subxid == xid) + { + set_apply_error_context_xact(subxid, InvalidXLogRecPtr); - fd = BufFileOpenFileSet(MyLogicalRepWorker->stream_fileset, path, O_RDONLY, - false); + AbortCurrentTransaction(); - buffer = palloc(BLCKSZ); - initStringInfo(&s2); + EndTransactionBlock(false); + CommitTransactionCommand(); - MemoryContextSwitchTo(oldcxt); + in_remote_transaction = false; - remote_final_lsn = lsn; + list_free(subxactlist); + subxactlist = NIL; - /* - * Make sure the handle apply_dispatch methods are aware we're in a remote + apply_bgworker_set_state(APPLY_BGWORKER_FINISHED); + } + else + { + /* + * OK, so it's a subxact. Rollback to the savepoint. + * + * We also need to read the subxactlist, determine the offset + * tracked for the subxact, and truncate the list. + */ + int i; + bool found = false; + char spname[MAXPGPATH]; + + set_apply_error_context_xact(subxid, InvalidXLogRecPtr); + snprintf(spname, MAXPGPATH, "savepoint_for_xid_%u", subxid); + + ereport(LOG, + (errmsg("[Apply BGW #%u] rolling back to savepoint %s", + MyParallelState->n, spname))); + + for(i = list_length(subxactlist) - 1; i >= 0; i--) + { + xid = (TransactionId) list_nth_int(subxactlist, i); + if (xid == subxid) + { + found = true; + break; + } + } + + if (found) + { + elog(LOG, "rolled back to savepoint %s", spname); + RollbackToSavepoint(spname); + CommitTransactionCommand(); + subxactlist = list_truncate(subxactlist, i + 1); + } + + apply_bgworker_set_state(APPLY_BGWORKER_READY); + } + + reset_apply_error_context_info(); + } + else + { + WorkerState *wstate = apply_bgworker_find_or_start(xid, false); + + /* + * If we are in main apply worker, check if we are processing this + * transaction in a bgworker. + */ + if (wstate) + { + apply_bgworker_send_data(wstate, s->len, s->data); + + if (subxid == xid) + { + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_FINISHED); + apply_bgworker_free(wstate); + } + else + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_READY); + } + + /* + * We are in main apply worker and the transaction has been serialized + * to file. + */ + else + serialize_stream_abort(xid, subxid); + } +} + +/* + * Common spoolfile processing. + */ +static void +apply_spooled_messages(TransactionId xid, XLogRecPtr lsn) +{ + StringInfoData s2; + int nchanges; + char path[MAXPGPATH]; + char *buffer = NULL; + MemoryContext oldcxt; + BufFile *fd; + + maybe_start_skipping_changes(lsn); + + /* Make sure we have an open transaction */ + begin_replication_step(); + + /* + * Allocate file handle and memory required to process all the messages in + * TopTransactionContext to avoid them getting reset after each message is + * processed. + */ + oldcxt = MemoryContextSwitchTo(TopTransactionContext); + + /* Open the spool file for the committed/prepared transaction */ + changes_filename(path, MyLogicalRepWorker->subid, xid); + elog(DEBUG1, "replaying changes from file \"%s\"", path); + + fd = BufFileOpenFileSet(MyLogicalRepWorker->stream_fileset, path, O_RDONLY, + false); + + buffer = palloc(BLCKSZ); + initStringInfo(&s2); + + MemoryContextSwitchTo(oldcxt); + + remote_final_lsn = lsn; + + /* + * Make sure the handle apply_dispatch methods are aware we're in a remote * transaction. */ in_remote_transaction = true; @@ -1462,40 +1949,6 @@ apply_spooled_messages(TransactionId xid, XLogRecPtr lsn) return; } -/* - * Handle STREAM COMMIT message. - */ -static void -apply_handle_stream_commit(StringInfo s) -{ - TransactionId xid; - LogicalRepCommitData commit_data; - - if (in_streamed_transaction) - ereport(ERROR, - (errcode(ERRCODE_PROTOCOL_VIOLATION), - errmsg_internal("STREAM COMMIT message without STREAM STOP"))); - - xid = logicalrep_read_stream_commit(s, &commit_data); - set_apply_error_context_xact(xid, commit_data.commit_lsn); - - elog(DEBUG1, "received commit for streamed transaction %u", xid); - - apply_spooled_messages(xid, commit_data.commit_lsn); - - apply_handle_commit_internal(&commit_data); - - /* unlink the files with serialized changes and subxact info */ - stream_cleanup_files(MyLogicalRepWorker->subid, xid); - - /* Process any tables that are being synchronized in parallel. */ - process_syncing_tables(commit_data.end_lsn); - - pgstat_report_activity(STATE_IDLE, NULL); - - reset_apply_error_context_info(); -} - /* * Helper function for apply_handle_commit and apply_handle_stream_commit. */ @@ -2445,6 +2898,100 @@ apply_handle_truncate(StringInfo s) end_replication_step(); } +/* + * Handle STREAM COMMIT message. + */ +static void +apply_handle_stream_commit(StringInfo s) +{ + LogicalRepCommitData commit_data; + TransactionId xid; + + if (in_streamed_transaction) + ereport(ERROR, + (errcode(ERRCODE_PROTOCOL_VIOLATION), + errmsg_internal("STREAM COMMIT message without STREAM STOP"))); + + xid = logicalrep_read_stream_commit(s, &commit_data); + set_apply_error_context_xact(xid, commit_data.commit_lsn); + + elog(DEBUG1, "received commit for streamed transaction %u", xid); + + if (am_apply_bgworker()) + { + /* + * Update origin state so we can restart streaming from correct + * position in case of crash. + */ + replorigin_session_origin_lsn = commit_data.end_lsn; + replorigin_session_origin_timestamp = commit_data.committime; + + CommitTransactionCommand(); + EndTransactionBlock(false); + CommitTransactionCommand(); + + in_remote_transaction = false; + + pgstat_report_stat(false); + + list_free(subxactlist); + subxactlist = NIL; + + apply_bgworker_set_state(APPLY_BGWORKER_FINISHED); + } + else + { + /* + * If we are in main apply worker, check if we are processing this + * transaction in an apply background worker. + */ + WorkerState *wstate = apply_bgworker_find_or_start(xid, false); + + if (wstate) + { + /* Send commit message */ + apply_bgworker_send_data(wstate, s->len, s->data); + + /* Wait for apply background worker to finish */ + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_FINISHED); + + pgstat_report_stat(false); + store_flush_position(commit_data.end_lsn); + stop_skipping_changes(); + + apply_bgworker_free(wstate); + + /* + * The transaction is either non-empty or skipped, so we clear the + * subskiplsn. + */ + clear_subscription_skip_lsn(commit_data.commit_lsn); + } + else + { + /* + * If we are in main apply worker and the transaction has been + * serialized to file, replay all the spooled operations. + */ + apply_spooled_messages(xid, commit_data.commit_lsn); + + apply_handle_commit_internal(&commit_data); + + /* unlink the files with serialized changes and subxact info */ + stream_cleanup_files(MyLogicalRepWorker->subid, xid); + } + } + + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); + + /* Process any tables that are being synchronized in parallel. */ + process_syncing_tables(commit_data.end_lsn); + + pgstat_report_activity(STATE_IDLE, NULL); + + reset_apply_error_context_info(); +} /* * Logical replication protocol message dispatcher. @@ -2511,6 +3058,8 @@ apply_dispatch(StringInfo s) break; case LOGICAL_REP_MSG_STREAM_START: + /* FIXME : log for debugging here. */ + elog(LOG, "LOGICAL_REP_MSG_STREAM_START"); apply_handle_stream_start(s); break; @@ -2618,6 +3167,10 @@ store_flush_position(XLogRecPtr remote_lsn) { FlushPosition *flushpos; + /* We only need to collect the LSN in main apply worker */ + if (am_apply_bgworker()) + return; + /* Need to do this in permanent context */ MemoryContextSwitchTo(ApplyContext); @@ -2794,6 +3347,9 @@ LogicalRepApplyLoop(XLogRecPtr last_received) AcceptInvalidationMessages(); maybe_reread_subscription(); + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); + /* Process any table synchronization changes. */ process_syncing_tables(last_received); } @@ -3691,7 +4247,7 @@ ApplyWorkerMain(Datum main_arg) originid = replorigin_by_name(originname, true); if (!OidIsValid(originid)) originid = replorigin_create(originname); - replorigin_session_setup(originid); + replorigin_session_setup(originid, true); replorigin_session_origin = originid; origin_startpos = replorigin_session_get_progress(false); CommitTransactionCommand(); @@ -3738,7 +4294,7 @@ ApplyWorkerMain(Datum main_arg) options.proto.logical.publication_names = MySubscription->publications; options.proto.logical.binary = MySubscription->binary; - options.proto.logical.streaming = MySubscription->stream; + options.proto.logical.streaming = (MySubscription->stream != SUBSTREAM_OFF); options.proto.logical.twophase = false; if (!am_tablesync_worker()) @@ -3896,7 +4452,8 @@ clear_subscription_skip_lsn(XLogRecPtr finish_lsn) XLogRecPtr myskiplsn = MySubscription->skiplsn; bool started_tx = false; - if (likely(XLogRecPtrIsInvalid(myskiplsn))) + if (likely(XLogRecPtrIsInvalid(myskiplsn)) || + am_apply_bgworker()) return; if (!IsTransactionState()) @@ -4032,3 +4589,533 @@ reset_apply_error_context_info(void) apply_error_callback_arg.remote_attnum = -1; set_apply_error_context_xact(InvalidTransactionId, InvalidXLogRecPtr); } + +/* Apply Background Worker main loop */ +static void +LogicalApplyBgwLoop(shm_mq_handle *mqh, volatile ApplyBgworkerShared *pst) +{ + shm_mq_result shmq_res; + PGPROC *registrant; + ErrorContextCallback errcallback; + XLogRecPtr last_received = InvalidXLogRecPtr; + + registrant = BackendPidGetProc(MyBgworkerEntry->bgw_notify_pid); + SetLatch(®istrant->procLatch); + + /* + * Push apply error context callback. Fields will be filled during + * applying a change. + */ + errcallback.callback = apply_error_callback; + errcallback.previous = error_context_stack; + error_context_stack = &errcallback; + + for (;;) + { + void *data; + Size len; + StringInfoData s; + MemoryContext oldctx; + XLogRecPtr start_lsn; + XLogRecPtr end_lsn; + TimestampTz send_time; + + CHECK_FOR_INTERRUPTS(); + + /* Ensure we are reading the data into our memory context. */ + oldctx = MemoryContextSwitchTo(ApplyMessageContext); + + shmq_res = shm_mq_receive(mqh, &len, &data, false); + + if (shmq_res != SHM_MQ_SUCCESS) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("lost connection to the main apply worker"))); + + if (len == 0) + { + elog(LOG, "[Apply BGW #%u] got zero-length message, stopping", pst->n); + break; + } + + s.cursor = 0; + s.maxlen = -1; + s.data = (char *) data; + s.len = len; + + /* + * We use first byte of message for additional communication between + * main Logical replication worker and apply bgworkers, so if it + * differs from 'w', then process it first. + */ + switch (pq_getmsgbyte(&s)) + { + /* End message of streaming chunk */ + case LOGICAL_REP_MSG_STREAM_STOP: + elog(LOG, "[Apply BGW #%u] ended processing streaming chunk," + "waiting on shm_mq_receive", pst->n); + + apply_bgworker_set_state(APPLY_BGWORKER_READY); + + SetLatch(®istrant->procLatch); + + in_streamed_transaction = false; + pgstat_report_activity(STATE_IDLE, NULL); + continue; + case 'w': + break; + default: + elog(ERROR, "unexpected message"); + break; + } + + start_lsn = pq_getmsgint64(&s); + end_lsn = pq_getmsgint64(&s); + send_time = pq_getmsgint64(&s); + + if (last_received < start_lsn) + last_received = start_lsn; + + if (last_received < end_lsn) + last_received = end_lsn; + + /* + * TO IMPROVE: Do we need to display the bgworker's information in + * pg_stat_replication ? + */ + UpdateWorkerStats(last_received, send_time, false); + + apply_dispatch(&s); + + if (ConfigReloadPending) + { + ConfigReloadPending = false; + ProcessConfigFile(PGC_SIGHUP); + } + + MemoryContextSwitchTo(oldctx); + MemoryContextReset(ApplyMessageContext); + } + + MemoryContextSwitchTo(TopMemoryContext); + MemoryContextReset(ApplyContext); + + /* Pop the error context stack */ + error_context_stack = errcallback.previous; + + elog(LOG, "[Apply BGW #%u] exiting", pst->n); + + /* Signal main process that we are done. */ + SetLatch(®istrant->procLatch); +} + +/* + * Set the exit state so that the main apply worker can realize we have + * shutdown. + */ +static void +ApplyBgwShutdown(int code, Datum arg) +{ + SpinLockAcquire(&MyParallelState->mutex); + MyParallelState->state = APPLY_BGWORKER_EXIT; + SpinLockRelease(&MyParallelState->mutex); + + dsm_detach((dsm_segment *) DatumGetPointer(arg)); +} + +/* + * Apply Background Worker entry point + */ +void +ApplyBgworkerMain(Datum main_arg) +{ + volatile ApplyBgworkerShared *pst; + + dsm_handle handle; + dsm_segment *seg; + shm_toc *toc; + shm_mq *mq; + shm_mq_handle *mqh; + MemoryContext oldcontext; + RepOriginId originid; + int worker_slot = DatumGetInt32(main_arg); + char originname[NAMEDATALEN]; + + /* Attach to slot */ + logicalrep_worker_attach(worker_slot); + + MemoryContextSwitchTo(TopMemoryContext); + + /* Load the subscription into persistent memory context. */ + ApplyContext = AllocSetContextCreate(TopMemoryContext, + "ApplyContext", + ALLOCSET_DEFAULT_SIZES); + + /* + * Init the ApplyMessageContext which we clean up after each replication + * protocol message. + */ + ApplyMessageContext = AllocSetContextCreate(ApplyContext, + "ApplyMessageContext", + ALLOCSET_DEFAULT_SIZES); + + /* Setup signal handling */ + pqsignal(SIGHUP, SignalHandlerForConfigReload); + pqsignal(SIGTERM, die); + BackgroundWorkerUnblockSignals(); + + /* + * Connect to the dynamic shared memory segment. + * + * The backend that registered this worker passed us the ID of a shared + * memory segment to which we must attach for further instructions. In + * order to attach to dynamic shared memory, we need a resource owner. + * Once we've mapped the segment in our address space, attach to the table + * of contents so we can locate the various data structures we'll need to + * find within the segment. + */ + CurrentResourceOwner = ResourceOwnerCreate(NULL, "Logical apply worker"); + memcpy(&handle, MyBgworkerEntry->bgw_extra, sizeof(dsm_handle)); + seg = dsm_attach(handle); + if (seg == NULL) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("unable to map dynamic shared memory segment"))); + toc = shm_toc_attach(PG_LOGICAL_APPLY_SHM_MAGIC, dsm_segment_address(seg)); + if (toc == NULL) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("bad magic number in dynamic shared memory segment"))); + + before_shmem_exit(ApplyBgwShutdown, PointerGetDatum(seg)); + + /* Look up the parallel state. */ + pst = shm_toc_lookup(toc, 0, false); + MyParallelState = pst; + + /* + * Attach to the message queue. + */ + mq = shm_toc_lookup(toc, 1, false); + shm_mq_set_receiver(mq, MyProc); + mqh = shm_mq_attach(mq, seg, NULL); + + /* Connect to our database. */ + BackgroundWorkerInitializeConnectionByOid(MyLogicalRepWorker->dbid, + MyLogicalRepWorker->userid, + 0); + + /* + * Set the client encoding to the database encoding, since that is what + * the leader will expect. + */ + SetClientEncoding(GetDatabaseEncoding()); + + stream_xid = pst->stream_xid; + + StartTransactionCommand(); + oldcontext = MemoryContextSwitchTo(ApplyContext); + + MySubscription = GetSubscription(MyLogicalRepWorker->subid, true); + if (!MySubscription) + { + ereport(LOG, + (errmsg("logical replication apply worker for subscription %u will not " + "start because the subscription was removed during startup", + MyLogicalRepWorker->subid))); + proc_exit(0); + } + + MySubscriptionValid = true; + MemoryContextSwitchTo(oldcontext); + + /* Setup synchronous commit according to the user's wishes */ + SetConfigOption("synchronous_commit", MySubscription->synccommit, + PGC_BACKEND, PGC_S_OVERRIDE); + + /* Keep us informed about subscription changes. */ + CacheRegisterSyscacheCallback(SUBSCRIPTIONOID, + subscription_change_cb, + (Datum) 0); + + CommitTransactionCommand(); + + /* Setup replication origin tracking. */ + StartTransactionCommand(); + snprintf(originname, sizeof(originname), "pg_%u", MySubscription->oid); + originid = replorigin_by_name(originname, true); + if (!OidIsValid(originid)) + originid = replorigin_create(originname); + + /* + * The apply bgworker don't need to monopolize this replication origin + * which was already acquired by its leader process. + */ + replorigin_session_setup(originid, false); + replorigin_session_origin = originid; + CommitTransactionCommand(); + + /* + * Allocate the origin name in long-lived context for error context + * message. + */ + apply_error_callback_arg.origin_name = MemoryContextStrdup(ApplyContext, + originname); + + /* + * Indicate that we're fully initialized and ready to begin the main part + * of the apply operation. + */ + apply_bgworker_set_state(APPLY_BGWORKER_ATTACHED); + + elog(LOG, "[Apply BGW #%u] started", pst->n); + + PG_TRY(); + { + LogicalApplyBgwLoop(mqh, pst); + } + PG_CATCH(); + { + /* + * Report the worker failed while applying streaming transaction + * changes. Abort the current transaction so that the stats message is + * sent in an idle state. + */ + AbortOutOfAnyTransaction(); + pgstat_report_subscription_error(MySubscription->oid, false); + + PG_RE_THROW(); + } + PG_END_TRY(); + + /* + * We're done. Explicitly detach the shared memory segment so that we + * don't get a resource leak warning at commit time. This will fire any + * on_dsm_detach callbacks we've registered, as well. Once that's done, + * we can go ahead and exit. + */ + dsm_detach(seg); + proc_exit(0); +} + +/* + * Set up a dynamic shared memory segment. + * + * We set up a control region that contains a ApplyBgworkerShared, + * plus one region per message queue. There are as many message queues as + * the number of workers. + */ +static void +apply_bgworker_setup_dsm(WorkerState *wstate) +{ + shm_toc_estimator e; + int toc_key = 0; + Size segsize; + dsm_segment *seg; + shm_toc *toc; + ApplyBgworkerShared *pst; + shm_mq *mq; + int64 queue_size = 160000000; /* 16 MB for now */ + + /* + * Estimate how much shared memory we need. + * + * Because the TOC machinery may choose to insert padding of oddly-sized + * requests, we must estimate each chunk separately. + * + * We need one key to register the location of the header, and we need + * another key to track the location of the message queue. + */ + shm_toc_initialize_estimator(&e); + shm_toc_estimate_chunk(&e, sizeof(ApplyBgworkerShared)); + shm_toc_estimate_chunk(&e, (Size) queue_size); + + shm_toc_estimate_keys(&e, 1 + 1); + segsize = shm_toc_estimate(&e); + + /* Create the shared memory segment and establish a table of contents. */ + seg = dsm_create(shm_toc_estimate(&e), 0); + toc = shm_toc_create(PG_LOGICAL_APPLY_SHM_MAGIC, dsm_segment_address(seg), + segsize); + + /* Set up the header region. */ + pst = shm_toc_allocate(toc, sizeof(ApplyBgworkerShared)); + SpinLockInit(&pst->mutex); + pst->state = APPLY_BGWORKER_BUSY; + pst->stream_xid = stream_xid; + pst->n = list_length(ApplyWorkersList) + 1; + + shm_toc_insert(toc, toc_key++, pst); + + /* Set up one message queue per worker, plus one. */ + mq = shm_mq_create(shm_toc_allocate(toc, (Size) queue_size), + (Size) queue_size); + shm_toc_insert(toc, toc_key++, mq); + shm_mq_set_sender(mq, MyProc); + + /* Attach the queues. */ + wstate->mq_handle = shm_mq_attach(mq, seg, NULL); + + /* Return results to caller. */ + wstate->dsm_seg = seg; + wstate->pstate = pst; +} + +/* + * Start apply worker background worker process and allocate shared memory for + * it. + */ +static WorkerState * +apply_bgworker_setup(void) +{ + MemoryContext oldcontext; + bool launched; + WorkerState *wstate; + + elog(LOG, "setting up apply worker #%u", list_length(ApplyWorkersList) + 1); + + oldcontext = MemoryContextSwitchTo(ApplyContext); + + wstate = (WorkerState *) palloc0(sizeof(WorkerState)); + + /* Setup shared memory */ + apply_bgworker_setup_dsm(wstate); + + launched = logicalrep_worker_launch(MyLogicalRepWorker->dbid, + MySubscription->oid, + MySubscription->name, + MyLogicalRepWorker->userid, + InvalidOid, + dsm_segment_handle(wstate->dsm_seg)); + + if (launched) + { + /* Wait for worker to attach. */ + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_ATTACHED); + + ApplyWorkersList = lappend(ApplyWorkersList, wstate); + } + else + { + pfree(wstate); + wstate = NULL; + } + + MemoryContextSwitchTo(oldcontext); + + return wstate; +} + +/* + * Send the data to the specified apply bgworker via shared-memory queue. + */ +static void +apply_bgworker_send_data(WorkerState *wstate, Size nbytes, const void *data) +{ + shm_mq_result result; + + result = shm_mq_send(wstate->mq_handle, nbytes, data, false, true); + + if (result != SHM_MQ_SUCCESS) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("could not send tuples to shared-memory queue"))); +} + +/* + * Wait until the state of apply background worker reaches the 'wait_for_state' + */ +static void +apply_bgworker_wait_for(WorkerState *wstate, char wait_for_state) +{ + for (;;) + { + char status; + + SpinLockAcquire(&wstate->pstate->mutex); + status = wstate->pstate->state; + SpinLockRelease(&wstate->pstate->mutex); + + /* Done if already in correct state. */ + if (status == wait_for_state) + break; + + /* If any workers (or the postmaster) have died, we have failed. */ + if (status == APPLY_BGWORKER_EXIT) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("Background worker %u failed to apply transaction %u", + wstate->pstate->n, wstate->pstate->stream_xid))); + + /* Wait to be signalled. */ + WaitLatch(MyLatch, WL_LATCH_SET | WL_EXIT_ON_PM_DEATH, 0, + WAIT_EVENT_LOGICAL_APPLY_WORKER_STATE_CHANGE); + + /* Reset the latch so we don't spin. */ + ResetLatch(MyLatch); + + /* An interrupt may have occurred while we were waiting. */ + CHECK_FOR_INTERRUPTS(); + } +} + +/* + * Check the status of workers and report an error if any apply background + * worker has exited unexpectedly. + * + * Exit if any relation is not in the READY state and if any worker is handling + * the streaming transaction at the same time. Because for streaming + * transactions that is being applied in apply bgworker, we cannot decide whether to + * apply the change for a relation that is not in the READY state (see + * should_apply_changes_for_rel) as we won't know remote_final_lsn by that + * time. + */ +static void +apply_bgworker_check_status(void) +{ + ListCell *lc; + + if (am_apply_bgworker() || MySubscription->stream != SUBSTREAM_APPLY) + return; + + foreach(lc, ApplyWorkersList) + { + WorkerState *wstate = (WorkerState *) lfirst(lc); + + /* + * We don't lock here as in the worst case we will just detect the + * failure of worker a bit later. + */ + if (wstate->pstate->state == APPLY_BGWORKER_EXIT) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("Background worker %u exited unexpectedly", + wstate->pstate->n))); + } + + if (list_length(ApplyWorkersIdleList) != list_length(ApplyWorkersList) && + !AllTablesyncsReady()) + { + ereport(LOG, + (errmsg("logical replication apply workers for subscription \"%s\" will restart", + MySubscription->name), + errdetail("Cannot handle streamed replication transaction by apply " + "bgworkers until all tables are synchronized"))); + + proc_exit(0); + } +} + +/* Set the state of apply background worker */ +static void +apply_bgworker_set_state(char state) +{ + if (!am_apply_bgworker()) + return; + + elog(LOG, "[Apply BGW #%u] set state to %d", MyParallelState->n, state); + + SpinLockAcquire(&MyParallelState->mutex); + MyParallelState->state = state; + SpinLockRelease(&MyParallelState->mutex); +} diff --git a/src/backend/utils/activity/wait_event.c b/src/backend/utils/activity/wait_event.c index 87c15b9c6f..ba781e6f08 100644 --- a/src/backend/utils/activity/wait_event.c +++ b/src/backend/utils/activity/wait_event.c @@ -388,6 +388,9 @@ pgstat_get_wait_ipc(WaitEventIPC w) case WAIT_EVENT_HASH_GROW_BUCKETS_REINSERT: event_name = "HashGrowBucketsReinsert"; break; + case WAIT_EVENT_LOGICAL_APPLY_WORKER_STATE_CHANGE: + event_name = "LogicalApplyWorkerStateChange"; + break; case WAIT_EVENT_LOGICAL_SYNC_DATA: event_name = "LogicalSyncData"; break; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 7cc9c72e49..6f8b30abb0 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -4450,7 +4450,7 @@ getSubscriptions(Archive *fout) if (fout->remoteVersion >= 140000) appendPQExpBufferStr(query, " s.substream,\n"); else - appendPQExpBufferStr(query, " false AS substream,\n"); + appendPQExpBufferStr(query, " 'f' AS substream,\n"); if (fout->remoteVersion >= 150000) appendPQExpBufferStr(query, @@ -4580,8 +4580,10 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo) if (strcmp(subinfo->subbinary, "t") == 0) appendPQExpBufferStr(query, ", binary = true"); - if (strcmp(subinfo->substream, "f") != 0) + if (strcmp(subinfo->substream, "t") == 0) appendPQExpBufferStr(query, ", streaming = on"); + else if (strcmp(subinfo->substream, "a") == 0) + appendPQExpBufferStr(query, ", streaming = apply"); if (strcmp(subinfo->subtwophasestate, two_phase_disabled) != 0) appendPQExpBufferStr(query, ", two_phase = on"); diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h index d1260f590c..9b394a45fe 100644 --- a/src/include/catalog/pg_subscription.h +++ b/src/include/catalog/pg_subscription.h @@ -68,7 +68,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW bool subbinary; /* True if the subscription wants the * publisher to send data in binary */ - bool substream; /* Stream in-progress transactions. */ + char substream; /* Stream in-progress transactions. + * See SUBSTREAM_xxx constants. */ char subtwophasestate; /* Stream two-phase transactions */ @@ -109,7 +110,7 @@ typedef struct Subscription bool enabled; /* Indicates if the subscription is enabled */ bool binary; /* Indicates if the subscription wants data in * binary format */ - bool stream; /* Allow streaming in-progress transactions. */ + char stream; /* Allow streaming in-progress transactions. */ char twophasestate; /* Allow streaming two-phase transactions */ bool disableonerr; /* Indicates if the subscription should be * automatically disabled if a worker error @@ -120,6 +121,18 @@ typedef struct Subscription List *publications; /* List of publication names to subscribe to */ } Subscription; +/* Disallow streaming in-progress transactions */ +#define SUBSTREAM_OFF 'f' + +/* + * Streaming transactions are written to a temporary file and applied only + * after the transaction is committed on upstream. + */ +#define SUBSTREAM_ON 't' + +/* Streaming transactions are applied immediately via a background worker */ +#define SUBSTREAM_APPLY 'a' + extern Subscription *GetSubscription(Oid subid, bool missing_ok); extern void FreeSubscription(Subscription *sub); extern void DisableSubscription(Oid subid); diff --git a/src/include/replication/logicalworker.h b/src/include/replication/logicalworker.h index cd1b6e8afc..6a1af7f13c 100644 --- a/src/include/replication/logicalworker.h +++ b/src/include/replication/logicalworker.h @@ -13,6 +13,7 @@ #define LOGICALWORKER_H extern void ApplyWorkerMain(Datum main_arg); +extern void ApplyBgworkerMain(Datum main_arg); extern bool IsLogicalWorker(void); diff --git a/src/include/replication/origin.h b/src/include/replication/origin.h index 2d1b5e5c28..b270434853 100644 --- a/src/include/replication/origin.h +++ b/src/include/replication/origin.h @@ -53,7 +53,7 @@ extern XLogRecPtr replorigin_get_progress(RepOriginId node, bool flush); extern void replorigin_session_advance(XLogRecPtr remote_commit, XLogRecPtr local_commit); -extern void replorigin_session_setup(RepOriginId node); +extern void replorigin_session_setup(RepOriginId node, bool acquire); extern void replorigin_session_reset(void); extern XLogRecPtr replorigin_session_get_progress(bool flush); diff --git a/src/include/replication/worker_internal.h b/src/include/replication/worker_internal.h index 901845abc2..6613e32077 100644 --- a/src/include/replication/worker_internal.h +++ b/src/include/replication/worker_internal.h @@ -60,6 +60,8 @@ typedef struct LogicalRepWorker */ FileSet *stream_fileset; + bool subworker; + /* Stats. */ XLogRecPtr last_lsn; TimestampTz last_send_time; @@ -84,8 +86,9 @@ extern void logicalrep_worker_attach(int slot); extern LogicalRepWorker *logicalrep_worker_find(Oid subid, Oid relid, bool only_running); extern List *logicalrep_workers_find(Oid subid, bool only_running); -extern void logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, - Oid userid, Oid relid); +extern bool logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, + Oid userid, Oid relid, + dsm_handle subworker_dsm); extern void logicalrep_worker_stop(Oid subid, Oid relid); extern void logicalrep_worker_wakeup(Oid subid, Oid relid); extern void logicalrep_worker_wakeup_ptr(LogicalRepWorker *worker); @@ -109,4 +112,10 @@ am_tablesync_worker(void) return OidIsValid(MyLogicalRepWorker->relid); } +static inline bool +am_apply_bgworker(void) +{ + return MyLogicalRepWorker->subworker; +} + #endif /* WORKER_INTERNAL_H */ diff --git a/src/include/utils/wait_event.h b/src/include/utils/wait_event.h index b578e2ec75..c2d2a114d7 100644 --- a/src/include/utils/wait_event.h +++ b/src/include/utils/wait_event.h @@ -105,6 +105,7 @@ typedef enum WAIT_EVENT_HASH_GROW_BUCKETS_ALLOCATE, WAIT_EVENT_HASH_GROW_BUCKETS_ELECT, WAIT_EVENT_HASH_GROW_BUCKETS_REINSERT, + WAIT_EVENT_LOGICAL_APPLY_WORKER_STATE_CHANGE, WAIT_EVENT_LOGICAL_SYNC_DATA, WAIT_EVENT_LOGICAL_SYNC_STATE_CHANGE, WAIT_EVENT_MQ_INTERNAL, diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out index 7fcfad1591..f769835af0 100644 --- a/src/test/regress/expected/subscription.out +++ b/src/test/regress/expected/subscription.out @@ -197,7 +197,7 @@ ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); DROP SUBSCRIPTION regress_testsub; -- fail - streaming must be boolean CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = foo); -ERROR: streaming requires a Boolean value +ERROR: streaming requires a Boolean value or "apply" -- now it works CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = true); WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index dd1214977a..9d89912eeb 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -116,6 +116,8 @@ Append AppendPath AppendRelInfo AppendState +ApplyBgworkerShared +ApplyBgworkerState ApplyErrorCallbackArg ApplyExecutionData ApplySubXactData @@ -2994,11 +2996,13 @@ WordEntryPosVector WordEntryPosVector1 WorkTableScan WorkTableScanState +WorkerEntry WorkerInfo WorkerInfoData WorkerInstrumentation WorkerJobDumpPtrType WorkerJobRestorePtrType +WorkerState Working_State WriteBufPtrType WriteBytePtrType -- 2.18.4 [application/octet-stream] v6-0002-Test-streaming-apply-option-in-tap-test.patch (64.7K, ../../OSZPR01MB63106EADF50E0E710D36CE5DFDCA9@OSZPR01MB6310.jpnprd01.prod.outlook.com/3-v6-0002-Test-streaming-apply-option-in-tap-test.patch) download | inline diff: From 5ef6130e2a4cb34adeb66cddd4bd6b3485a98dcd Mon Sep 17 00:00:00 2001 From: "shiy.fnst" <[email protected]> Date: Fri, 13 May 2022 14:50:30 +0800 Subject: [PATCH v6 2/2] Test streaming apply option in tap test Change all TAP tests using the SUBSCRIPTION "streaming" option, so they now test both 'on' and 'apply' values. --- src/test/subscription/t/015_stream.pl | 181 ++++--- src/test/subscription/t/016_stream_subxact.pl | 98 ++-- src/test/subscription/t/017_stream_ddl.pl | 171 ++++--- .../t/018_stream_subxact_abort.pl | 178 ++++--- .../t/019_stream_subxact_ddl_abort.pl | 89 +++- .../subscription/t/022_twophase_cascade.pl | 330 ++++++------ .../subscription/t/023_twophase_stream.pl | 472 ++++++++++-------- 7 files changed, 875 insertions(+), 644 deletions(-) diff --git a/src/test/subscription/t/015_stream.pl b/src/test/subscription/t/015_stream.pl index 6561b189de..0ab35e2f70 100644 --- a/src/test/subscription/t/015_stream.pl +++ b/src/test/subscription/t/015_stream.pl @@ -8,6 +8,96 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" option +# so the same code can be run both for the streaming=on and streaming=apply +# cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname) = @_; + + # Interleave a pair of transactions, each exceeding the 64kB limit. + my $in = ''; + my $out = ''; + + my $timer = IPC::Run::timeout($PostgreSQL::Test::Utils::timeout_default); + + my $h = $node_publisher->background_psql('postgres', \$in, \$out, $timer, + on_error_stop => 0); + + $in .= q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + }; + $h->pump_nb; + + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001, 9999) s(i); + DELETE FROM test_tab WHERE a > 5000; + COMMIT; + }); + + $in .= q{ + COMMIT; + \q + }; + $h->finish; # errors make the next test fail, so ignore them here + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), 'check extra columns contain local defaults'); + + # Test the streaming in binary mode + $node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET (binary = on)"); + + # Insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001, 10000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + COMMIT; + }); + + $node_publisher->wait_for_catchup($appname); + + $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(6667|6667|6667), 'check extra columns contain local defaults'); + + # Change the local values of the extra columns on the subscriber, + # update publisher, and check that subscriber retains the expected + # values. This is to ensure that non-streaming transactions behave + # properly after a streaming transaction. + $node_subscriber->safe_psql('postgres', + "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'" + ); + $node_publisher->safe_psql('postgres', + "UPDATE test_tab SET b = md5(a::text)"); + + $node_publisher->wait_for_catchup($appname); + + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab" + ); + is($result, qq(6667|6667|6667), + 'check extra columns contain locally changed data'); + + # Cleanup the test data + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE (a > 2)"); + $node_publisher->wait_for_catchup($appname); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -37,99 +127,40 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); - $node_publisher->wait_for_catchup($appname); - # Also wait for initial table sync to finish my $synced_query = "SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');"; $node_subscriber->poll_query_until('postgres', $synced_query) or die "Timed out while waiting for subscriber to synchronize data"; - my $result = $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab"); is($result, qq(2|2|2), 'check initial data was copied to subscriber'); -# Interleave a pair of transactions, each exceeding the 64kB limit. -my $in = ''; -my $out = ''; - -my $timer = IPC::Run::timeout($PostgreSQL::Test::Utils::timeout_default); - -my $h = $node_publisher->background_psql('postgres', \$in, \$out, $timer, - on_error_stop => 0); - -$in .= q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -}; -$h->pump_nb; - -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001, 9999) s(i); -DELETE FROM test_tab WHERE a > 5000; -COMMIT; -}); - -$in .= q{ -COMMIT; -\q -}; -$h->finish; # errors make the next test fail, so ignore them here - -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), 'check extra columns contain local defaults'); - -# Test the streaming in binary mode -$node_subscriber->safe_psql('postgres', - "ALTER SUBSCRIPTION tap_sub SET (binary = on)"); - -# Insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001, 10000) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -COMMIT; -}); - -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(6667|6667|6667), 'check extra columns contain local defaults'); +test_streaming($node_publisher, $node_subscriber, $appname); -# Change the local values of the extra columns on the subscriber, -# update publisher, and check that subscriber retains the expected -# values. This is to ensure that non-streaming transactions behave -# properly after a streaming transaction. +################################### +# Test using streaming mode 'apply' +################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" +); $node_subscriber->safe_psql('postgres', - "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'" + "ALTER SUBSCRIPTION tap_sub SET(streaming = apply, binary = off)" ); -$node_publisher->safe_psql('postgres', - "UPDATE test_tab SET b = md5(a::text)"); +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" +) or die "Timed out while waiting for apply to restart after changing PUBLICATION"; -$node_publisher->wait_for_catchup($appname); - -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab" -); -is($result, qq(6667|6667|6667), - 'check extra columns contain locally changed data'); +test_streaming($node_publisher, $node_subscriber, $appname); $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/016_stream_subxact.pl b/src/test/subscription/t/016_stream_subxact.pl index f27f1694f2..d40c7c7ffd 100644 --- a/src/test/subscription/t/016_stream_subxact.pl +++ b/src/test/subscription/t/016_stream_subxact.pl @@ -8,6 +8,54 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" option +# so the same code can be run both for the streaming=on and streaming=apply +# cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname) = @_; + + # Insert, update and delete enough rows to exceed 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series( 3, 500) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501, 1000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001, 1500) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + SAVEPOINT s3; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501, 2000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + SAVEPOINT s4; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001, 2500) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + COMMIT; + }); + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(1667|1667|1667), + 'check data was copied to subscriber in streaming mode and extra columns contain local defaults' + ); + + # Cleanup the test data + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE (a > 2)"); + $node_publisher->wait_for_catchup($appname); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -37,6 +85,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -54,40 +106,22 @@ my $result = "SELECT count(*), count(c), count(d = 999) FROM test_tab"); is($result, qq(2|2|2), 'check initial data was copied to subscriber'); -# Insert, update and delete enough rows to exceed 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series( 3, 500) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501, 1000) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001, 1500) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -SAVEPOINT s3; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501, 2000) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -SAVEPOINT s4; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001, 2500) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -COMMIT; -}); +test_streaming($node_publisher, $node_subscriber, $appname); -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(1667|1667|1667), - 'check data was copied to subscriber in streaming mode and extra columns contain local defaults' +################################### +# Test using streaming mode 'apply' +################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" +); +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = apply)" ); +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" +) or die "Timed out while waiting for apply to restart after changing PUBLICATION"; + +test_streaming($node_publisher, $node_subscriber, $appname); $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/017_stream_ddl.pl b/src/test/subscription/t/017_stream_ddl.pl index 0bce63b716..1bc0c4dbc5 100644 --- a/src/test/subscription/t/017_stream_ddl.pl +++ b/src/test/subscription/t/017_stream_ddl.pl @@ -8,6 +8,93 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" option +# so the same code can be run both for the streaming=on and streaming=apply +# cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname) = @_; + + # a small (non-streamed) transaction with DDL and DML + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab VALUES (3, md5(3::text)); + ALTER TABLE test_tab ADD COLUMN c INT; + SAVEPOINT s1; + INSERT INTO test_tab VALUES (4, md5(4::text), -4); + COMMIT; + }); + + # large (streamed) transaction with DDL and DML + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(5, 1000) s(i); + ALTER TABLE test_tab ADD COLUMN d INT; + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001, 2000) s(i); + COMMIT; + }); + + # a small (non-streamed) transaction with DDL and DML + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab VALUES (2001, md5(2001::text), -2001, 2*2001); + ALTER TABLE test_tab ADD COLUMN e INT; + SAVEPOINT s1; + INSERT INTO test_tab VALUES (2002, md5(2002::text), -2002, 2*2002, -3*2002); + COMMIT; + }); + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d), count(e) FROM test_tab"); + is($result, qq(2002|1999|1002|1), + 'check data was copied to subscriber in streaming mode and extra columns contain local defaults' + ); + + # A large (streamed) transaction with DDL and DML. One of the DDL is performed + # after DML to ensure that we invalidate the schema sent for test_tab so that + # the next transaction has to send the schema again. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(2003,5000) s(i); + ALTER TABLE test_tab ADD COLUMN f INT; + COMMIT; + }); + + # A small transaction that won't get streamed. This is just to ensure that we + # send the schema again to reflect the last column added in the previous test. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i, 4*i FROM generate_series(5001,5005) s(i); + COMMIT; + }); + + $node_publisher->wait_for_catchup($appname); + + $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d), count(e), count(f) FROM test_tab"); + is($result, qq(5005|5002|4005|3004|5), + 'check data was copied to subscriber for both streaming and non-streaming transactions' + ); + + # Cleanup the test data + $node_publisher->safe_psql( + 'postgres', q{ + DELETE FROM test_tab WHERE (a > 2); + ALTER TABLE test_tab DROP COLUMN c, DROP COLUMN d, DROP COLUMN e, DROP COLUMN f; + }); + $node_publisher->wait_for_catchup($appname); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -37,6 +124,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -54,76 +145,22 @@ my $result = "SELECT count(*), count(c), count(d = 999) FROM test_tab"); is($result, qq(2|0|0), 'check initial data was copied to subscriber'); -# a small (non-streamed) transaction with DDL and DML -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab VALUES (3, md5(3::text)); -ALTER TABLE test_tab ADD COLUMN c INT; -SAVEPOINT s1; -INSERT INTO test_tab VALUES (4, md5(4::text), -4); -COMMIT; -}); - -# large (streamed) transaction with DDL and DML -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(5, 1000) s(i); -ALTER TABLE test_tab ADD COLUMN d INT; -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001, 2000) s(i); -COMMIT; -}); - -# a small (non-streamed) transaction with DDL and DML -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab VALUES (2001, md5(2001::text), -2001, 2*2001); -ALTER TABLE test_tab ADD COLUMN e INT; -SAVEPOINT s1; -INSERT INTO test_tab VALUES (2002, md5(2002::text), -2002, 2*2002, -3*2002); -COMMIT; -}); - -$node_publisher->wait_for_catchup($appname); +test_streaming($node_publisher, $node_subscriber, $appname); -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d), count(e) FROM test_tab"); -is($result, qq(2002|1999|1002|1), - 'check data was copied to subscriber in streaming mode and extra columns contain local defaults' +################################### +# Test using streaming mode 'apply' +################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" ); - -# A large (streamed) transaction with DDL and DML. One of the DDL is performed -# after DML to ensure that we invalidate the schema sent for test_tab so that -# the next transaction has to send the schema again. -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(2003,5000) s(i); -ALTER TABLE test_tab ADD COLUMN f INT; -COMMIT; -}); - -# A small transaction that won't get streamed. This is just to ensure that we -# send the schema again to reflect the last column added in the previous test. -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i, 4*i FROM generate_series(5001,5005) s(i); -COMMIT; -}); - -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d), count(e), count(f) FROM test_tab"); -is($result, qq(5005|5002|4005|3004|5), - 'check data was copied to subscriber for both streaming and non-streaming transactions' +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = apply)" ); +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" +) or die "Timed out while waiting for apply to restart after changing PUBLICATION"; + +test_streaming($node_publisher, $node_subscriber, $appname); $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/018_stream_subxact_abort.pl b/src/test/subscription/t/018_stream_subxact_abort.pl index 7155442e76..411bb47e96 100644 --- a/src/test/subscription/t/018_stream_subxact_abort.pl +++ b/src/test/subscription/t/018_stream_subxact_abort.pl @@ -8,6 +8,95 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" option +# so the same code can be run both for the streaming=on and streaming=apply +# cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname) = @_; + + # large (streamed) transaction with DDL, DML and ROLLBACKs + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i); + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501,1000) s(i); + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001,1500) s(i); + SAVEPOINT s3; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501,2000) s(i); + ROLLBACK TO s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001,2500) s(i); + ROLLBACK TO s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2501,3000) s(i); + SAVEPOINT s4; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3001,3500) s(i); + SAVEPOINT s5; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3501,4000) s(i); + COMMIT; + }); + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c) FROM test_tab"); + is($result, qq(2000|0), + 'check rollback to savepoint was reflected on subscriber and extra columns contain local defaults' + ); + + # large (streamed) transaction with subscriber receiving out of order + # subtransaction ROLLBACKs + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(4001,4500) s(i); + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001,5500) s(i); + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(6001,6500) s(i); + SAVEPOINT s3; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(7001,7500) s(i); + RELEASE s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(8001,8500) s(i); + ROLLBACK TO s1; + COMMIT; + }); + + $node_publisher->wait_for_catchup($appname); + + $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c) FROM test_tab"); + is($result, qq(2500|0), + 'check rollback to savepoint was reflected on subscriber'); + + # large (streamed) transaction with subscriber receiving rollback + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(8501,9000) s(i); + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(9001,9500) s(i); + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(9501,10000) s(i); + ROLLBACK; + }); + + $node_publisher->wait_for_catchup($appname); + + $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c) FROM test_tab"); + is($result, qq(2500|0), 'check rollback was reflected on subscriber'); + + # Cleanup the test data + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE (a > 2)"); + $node_publisher->wait_for_catchup($appname); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -36,6 +125,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -53,81 +146,22 @@ my $result = "SELECT count(*), count(c) FROM test_tab"); is($result, qq(2|0), 'check initial data was copied to subscriber'); -# large (streamed) transaction with DDL, DML and ROLLBACKs -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i); -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501,1000) s(i); -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001,1500) s(i); -SAVEPOINT s3; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501,2000) s(i); -ROLLBACK TO s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001,2500) s(i); -ROLLBACK TO s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2501,3000) s(i); -SAVEPOINT s4; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3001,3500) s(i); -SAVEPOINT s5; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3501,4000) s(i); -COMMIT; -}); +test_streaming($node_publisher, $node_subscriber, $appname); -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c) FROM test_tab"); -is($result, qq(2000|0), - 'check rollback to savepoint was reflected on subscriber and extra columns contain local defaults' +################################### +# Test using streaming mode 'apply' +################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" ); +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = apply)" +); +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" +) or die "Timed out while waiting for apply to restart after changing PUBLICATION"; -# large (streamed) transaction with subscriber receiving out of order -# subtransaction ROLLBACKs -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(4001,4500) s(i); -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001,5500) s(i); -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(6001,6500) s(i); -SAVEPOINT s3; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(7001,7500) s(i); -RELEASE s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(8001,8500) s(i); -ROLLBACK TO s1; -COMMIT; -}); - -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c) FROM test_tab"); -is($result, qq(2500|0), - 'check rollback to savepoint was reflected on subscriber'); - -# large (streamed) transaction with subscriber receiving rollback -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(8501,9000) s(i); -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(9001,9500) s(i); -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(9501,10000) s(i); -ROLLBACK; -}); - -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c) FROM test_tab"); -is($result, qq(2500|0), 'check rollback was reflected on subscriber'); +test_streaming($node_publisher, $node_subscriber, $appname); $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/019_stream_subxact_ddl_abort.pl b/src/test/subscription/t/019_stream_subxact_ddl_abort.pl index dbd0fca4d1..31151676f0 100644 --- a/src/test/subscription/t/019_stream_subxact_ddl_abort.pl +++ b/src/test/subscription/t/019_stream_subxact_ddl_abort.pl @@ -9,6 +9,51 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" option +# so the same code can be run both for the streaming=on and streaming=apply +# cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname) = @_; + + # large (streamed) transaction with DDL, DML and ROLLBACKs + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i); + ALTER TABLE test_tab ADD COLUMN c INT; + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(501,1000) s(i); + ALTER TABLE test_tab ADD COLUMN d INT; + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001,1500) s(i); + ALTER TABLE test_tab ADD COLUMN e INT; + SAVEPOINT s3; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(1501,2000) s(i); + ALTER TABLE test_tab DROP COLUMN c; + ROLLBACK TO s1; + INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(501,1000) s(i); + COMMIT; + }); + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c) FROM test_tab"); + is($result, qq(1000|500), + 'check rollback to savepoint was reflected on subscriber and extra columns contain local defaults' + ); + + # Cleanup the test data + $node_publisher->safe_psql( + 'postgres', q{ + DELETE FROM test_tab WHERE (a > 2); + ALTER TABLE test_tab DROP COLUMN c; + }); + $node_publisher->wait_for_catchup($appname); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -37,6 +82,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -54,34 +103,22 @@ my $result = "SELECT count(*), count(c) FROM test_tab"); is($result, qq(2|0), 'check initial data was copied to subscriber'); -# large (streamed) transaction with DDL, DML and ROLLBACKs -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i); -ALTER TABLE test_tab ADD COLUMN c INT; -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(501,1000) s(i); -ALTER TABLE test_tab ADD COLUMN d INT; -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001,1500) s(i); -ALTER TABLE test_tab ADD COLUMN e INT; -SAVEPOINT s3; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(1501,2000) s(i); -ALTER TABLE test_tab DROP COLUMN c; -ROLLBACK TO s1; -INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(501,1000) s(i); -COMMIT; -}); +test_streaming($node_publisher, $node_subscriber, $appname); -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c) FROM test_tab"); -is($result, qq(1000|500), - 'check rollback to savepoint was reflected on subscriber and extra columns contain local defaults' +################################### +# Test using streaming mode 'apply' +################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" +); +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = apply)" ); +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" +) or die "Timed out while waiting for apply to restart after changing PUBLICATION"; + +test_streaming($node_publisher, $node_subscriber, $appname); $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/022_twophase_cascade.pl b/src/test/subscription/t/022_twophase_cascade.pl index 7a797f37ba..663808acb9 100644 --- a/src/test/subscription/t/022_twophase_cascade.pl +++ b/src/test/subscription/t/022_twophase_cascade.pl @@ -11,6 +11,175 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" option +# so the same code can be run both for the streaming=on and streaming=apply +# cases. +sub test_streaming +{ + my ($node_A, $node_B, $node_C, $appname_B, $appname_C, $streaming_mode) = @_; + + my $oldpid_B = $node_A->safe_psql( + 'postgres', " + SELECT pid FROM pg_stat_replication + WHERE application_name = '$appname_B' AND state = 'streaming';"); + my $oldpid_C = $node_B->safe_psql( + 'postgres', " + SELECT pid FROM pg_stat_replication + WHERE application_name = '$appname_C' AND state = 'streaming';"); + + # Setup logical replication streaming mode + + $node_B->safe_psql( + 'postgres', " + ALTER SUBSCRIPTION tap_sub_B + SET (streaming = $streaming_mode);"); + $node_C->safe_psql( + 'postgres', " + ALTER SUBSCRIPTION tap_sub_C + SET (streaming = $streaming_mode)"); + + # Wait for subscribers to finish initialization + + $node_A->poll_query_until( + 'postgres', " + SELECT pid != $oldpid_B FROM pg_stat_replication + WHERE application_name = '$appname_B' AND state = 'streaming';" + ) or die "Timed out while waiting for apply to restart"; + $node_B->poll_query_until( + 'postgres', " + SELECT pid != $oldpid_C FROM pg_stat_replication + WHERE application_name = '$appname_C' AND state = 'streaming';" + ) or die "Timed out while waiting for apply to restart"; + + ############################### + # Test 2PC PREPARE / COMMIT PREPARED. + # 1. Data is streamed as a 2PC transaction. + # 2. Then do commit prepared. + # + # Expect all data is replicated on subscriber(s) after the commit. + ############################### + + # Insert, update and delete enough rows to exceed the 64kB limit. + # Then 2PC PREPARE + $node_A->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); + + # check the transaction state is prepared on subscriber(s) + my $result = + $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber B'); + $result = + $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber C'); + + # 2PC COMMIT + $node_A->safe_psql('postgres', "COMMIT PREPARED 'test_prepared_tab';"); + + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); + + # check that transaction was committed on subscriber(s) + $result = $node_B->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), + 'Rows inserted by 2PC have committed on subscriber B, and extra columns have local defaults' + ); + $result = $node_C->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), + 'Rows inserted by 2PC have committed on subscriber C, and extra columns have local defaults' + ); + + # check the transaction state is ended on subscriber(s) + $result = + $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is committed on subscriber B'); + $result = + $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is committed on subscriber C'); + + ############################### + # Test 2PC PREPARE with a nested ROLLBACK TO SAVEPOINT. + # 0. Cleanup from previous test leaving only 2 rows. + # 1. Insert one more row. + # 2. Record a SAVEPOINT. + # 3. Data is streamed using 2PC. + # 4. Do rollback to SAVEPOINT prior to the streamed inserts. + # 5. Then COMMIT PREPARED. + # + # Expect data after the SAVEPOINT is aborted leaving only 3 rows (= 2 original + 1 from step 1). + ############################### + + # First, delete the data except for 2 rows (delete will be replicated) + $node_A->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); + + # 2PC PREPARE with a nested ROLLBACK TO SAVEPOINT + $node_A->safe_psql( + 'postgres', " + BEGIN; + INSERT INTO test_tab VALUES (9999, 'foobar'); + SAVEPOINT sp_inner; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + ROLLBACK TO SAVEPOINT sp_inner; + PREPARE TRANSACTION 'outer'; + "); + + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); + + # check the transaction state prepared on subscriber(s) + $result = + $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber B'); + $result = + $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber C'); + + # 2PC COMMIT + $node_A->safe_psql('postgres', "COMMIT PREPARED 'outer';"); + + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); + + # check the transaction state is ended on subscriber + $result = + $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is ended on subscriber B'); + $result = + $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is ended on subscriber C'); + + # check inserts are visible at subscriber(s). + # All the streamed data (prior to the SAVEPOINT) should be rolled back. + # (9999, 'foobar') should be committed. + $result = $node_B->safe_psql('postgres', + "SELECT count(*) FROM test_tab where b = 'foobar';"); + is($result, qq(1), 'Rows committed are present on subscriber B'); + $result = $node_B->safe_psql('postgres', "SELECT count(*) FROM test_tab;"); + is($result, qq(3), 'Rows committed are present on subscriber B'); + $result = $node_C->safe_psql('postgres', + "SELECT count(*) FROM test_tab where b = 'foobar';"); + is($result, qq(1), 'Rows committed are present on subscriber C'); + $result = $node_C->safe_psql('postgres', "SELECT count(*) FROM test_tab;"); + is($result, qq(3), 'Rows committed are present on subscriber C'); + + # Cleanup the test data + $node_A->safe_psql('postgres', + "DELETE FROM test_tab WHERE a > 2;"); + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); +} + ############################### # Setup a cascade of pub/sub nodes. # node_A -> node_B -> node_C @@ -260,160 +429,15 @@ is($result, qq(21), 'Rows committed are present on subscriber C'); # 2PC + STREAMING TESTS # --------------------- -my $oldpid_B = $node_A->safe_psql( - 'postgres', " - SELECT pid FROM pg_stat_replication - WHERE application_name = '$appname_B' AND state = 'streaming';"); -my $oldpid_C = $node_B->safe_psql( - 'postgres', " - SELECT pid FROM pg_stat_replication - WHERE application_name = '$appname_C' AND state = 'streaming';"); - -# Setup logical replication (streaming = on) - -$node_B->safe_psql( - 'postgres', " - ALTER SUBSCRIPTION tap_sub_B - SET (streaming = on);"); -$node_C->safe_psql( - 'postgres', " - ALTER SUBSCRIPTION tap_sub_C - SET (streaming = on)"); - -# Wait for subscribers to finish initialization - -$node_A->poll_query_until( - 'postgres', " - SELECT pid != $oldpid_B FROM pg_stat_replication - WHERE application_name = '$appname_B' AND state = 'streaming';" -) or die "Timed out while waiting for apply to restart"; -$node_B->poll_query_until( - 'postgres', " - SELECT pid != $oldpid_C FROM pg_stat_replication - WHERE application_name = '$appname_C' AND state = 'streaming';" -) or die "Timed out while waiting for apply to restart"; - -############################### -# Test 2PC PREPARE / COMMIT PREPARED. -# 1. Data is streamed as a 2PC transaction. -# 2. Then do commit prepared. -# -# Expect all data is replicated on subscriber(s) after the commit. -############################### - -# Insert, update and delete enough rows to exceed the 64kB limit. -# Then 2PC PREPARE -$node_A->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_A->wait_for_catchup($appname_B); -$node_B->wait_for_catchup($appname_C); +################################ +# Test using streaming mode 'on' +################################ +test_streaming($node_A, $node_B, $node_C, $appname_B, $appname_C, 'on'); -# check the transaction state is prepared on subscriber(s) -$result = - $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber B'); -$result = - $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber C'); - -# 2PC COMMIT -$node_A->safe_psql('postgres', "COMMIT PREPARED 'test_prepared_tab';"); - -$node_A->wait_for_catchup($appname_B); -$node_B->wait_for_catchup($appname_C); - -# check that transaction was committed on subscriber(s) -$result = $node_B->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), - 'Rows inserted by 2PC have committed on subscriber B, and extra columns have local defaults' -); -$result = $node_C->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), - 'Rows inserted by 2PC have committed on subscriber C, and extra columns have local defaults' -); - -# check the transaction state is ended on subscriber(s) -$result = - $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is committed on subscriber B'); -$result = - $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is committed on subscriber C'); - -############################### -# Test 2PC PREPARE with a nested ROLLBACK TO SAVEPOINT. -# 0. Cleanup from previous test leaving only 2 rows. -# 1. Insert one more row. -# 2. Record a SAVEPOINT. -# 3. Data is streamed using 2PC. -# 4. Do rollback to SAVEPOINT prior to the streamed inserts. -# 5. Then COMMIT PREPARED. -# -# Expect data after the SAVEPOINT is aborted leaving only 3 rows (= 2 original + 1 from step 1). -############################### - -# First, delete the data except for 2 rows (delete will be replicated) -$node_A->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); - -# 2PC PREPARE with a nested ROLLBACK TO SAVEPOINT -$node_A->safe_psql( - 'postgres', " - BEGIN; - INSERT INTO test_tab VALUES (9999, 'foobar'); - SAVEPOINT sp_inner; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - ROLLBACK TO SAVEPOINT sp_inner; - PREPARE TRANSACTION 'outer'; - "); - -$node_A->wait_for_catchup($appname_B); -$node_B->wait_for_catchup($appname_C); - -# check the transaction state prepared on subscriber(s) -$result = - $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber B'); -$result = - $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber C'); - -# 2PC COMMIT -$node_A->safe_psql('postgres', "COMMIT PREPARED 'outer';"); - -$node_A->wait_for_catchup($appname_B); -$node_B->wait_for_catchup($appname_C); - -# check the transaction state is ended on subscriber -$result = - $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is ended on subscriber B'); -$result = - $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is ended on subscriber C'); - -# check inserts are visible at subscriber(s). -# All the streamed data (prior to the SAVEPOINT) should be rolled back. -# (9999, 'foobar') should be committed. -$result = $node_B->safe_psql('postgres', - "SELECT count(*) FROM test_tab where b = 'foobar';"); -is($result, qq(1), 'Rows committed are present on subscriber B'); -$result = $node_B->safe_psql('postgres', "SELECT count(*) FROM test_tab;"); -is($result, qq(3), 'Rows committed are present on subscriber B'); -$result = $node_C->safe_psql('postgres', - "SELECT count(*) FROM test_tab where b = 'foobar';"); -is($result, qq(1), 'Rows committed are present on subscriber C'); -$result = $node_C->safe_psql('postgres', "SELECT count(*) FROM test_tab;"); -is($result, qq(3), 'Rows committed are present on subscriber C'); +################################### +# Test using streaming mode 'apply' +################################### +test_streaming($node_A, $node_B, $node_C, $appname_B, $appname_C, 'apply'); ############################### # check all the cleanup diff --git a/src/test/subscription/t/023_twophase_stream.pl b/src/test/subscription/t/023_twophase_stream.pl index d8475d25a4..1282cfadb7 100644 --- a/src/test/subscription/t/023_twophase_stream.pl +++ b/src/test/subscription/t/023_twophase_stream.pl @@ -8,6 +8,243 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" option +# so the same code can be run both for the streaming=on and streaming=apply +# cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname) = @_; + + ############################### + # Test 2PC PREPARE / COMMIT PREPARED. + # 1. Data is streamed as a 2PC transaction. + # 2. Then do commit prepared. + # + # Expect all data is replicated on subscriber side after the commit. + ############################### + + # check that 2PC gets replicated to subscriber + # Insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is in prepared state on subscriber + my $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber'); + + # 2PC transaction gets committed + $node_publisher->safe_psql('postgres', + "COMMIT PREPARED 'test_prepared_tab';"); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is committed on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), + 'Rows inserted by 2PC have committed on subscriber, and extra columns contain local defaults' + ); + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is committed on subscriber'); + + ############################### + # Test 2PC PREPARE / ROLLBACK PREPARED. + # 1. Table is deleted back to 2 rows which are replicated on subscriber. + # 2. Data is streamed using 2PC. + # 3. Do rollback prepared. + # + # Expect data rolls back leaving only the original 2 rows. + ############################### + + # First, delete the data except for 2 rows (will be replicated) + $node_publisher->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); + + # Then insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is in prepared state on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber'); + + # 2PC transaction gets aborted + $node_publisher->safe_psql('postgres', + "ROLLBACK PREPARED 'test_prepared_tab';"); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is aborted on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(2|2|2), + 'Rows inserted by 2PC are rolled back, leaving only the original 2 rows'); + + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is aborted on subscriber'); + + ############################### + # Check that 2PC COMMIT PREPARED is decoded properly on crash restart. + # 1. insert, update and delete enough rows to exceed the 64kB limit. + # 2. Then server crashes before the 2PC transaction is committed. + # 3. After servers are restarted the pending transaction is committed. + # + # Expect all data is replicated on subscriber side after the commit. + # Note: both publisher and subscriber do crash/restart. + ############################### + + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + $node_subscriber->stop('immediate'); + $node_publisher->stop('immediate'); + + $node_publisher->start; + $node_subscriber->start; + + # commit post the restart + $node_publisher->safe_psql('postgres', + "COMMIT PREPARED 'test_prepared_tab';"); + $node_publisher->wait_for_catchup($appname); + + # check inserts are visible + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), + 'Rows inserted by 2PC have committed on subscriber, and extra columns contain local defaults' + ); + + ############################### + # Do INSERT after the PREPARE but before ROLLBACK PREPARED. + # 1. Table is deleted back to 2 rows which are replicated on subscriber. + # 2. Data is streamed using 2PC. + # 3. A single row INSERT is done which is after the PREPARE. + # 4. Then do a ROLLBACK PREPARED. + # + # Expect the 2PC data rolls back leaving only 3 rows on the subscriber + # (the original 2 + inserted 1). + ############################### + + # First, delete the data except for 2 rows (will be replicated) + $node_publisher->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); + + # Then insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is in prepared state on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber'); + + # Insert a different record (now we are outside of the 2PC transaction) + # Note: the 2PC transaction still holds row locks so make sure this insert is for a separate primary key + $node_publisher->safe_psql('postgres', + "INSERT INTO test_tab VALUES (99999, 'foobar')"); + + # 2PC transaction gets aborted + $node_publisher->safe_psql('postgres', + "ROLLBACK PREPARED 'test_prepared_tab';"); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is aborted on subscriber, + # but the extra INSERT outside of the 2PC still was replicated + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3|3|3), 'check the outside insert was copied to subscriber'); + + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is aborted on subscriber'); + + ############################### + # Do INSERT after the PREPARE but before COMMIT PREPARED. + # 1. Table is deleted back to 2 rows which are replicated on subscriber. + # 2. Data is streamed using 2PC. + # 3. A single row INSERT is done which is after the PREPARE. + # 4. Then do a COMMIT PREPARED. + # + # Expect 2PC data + the extra row are on the subscriber + # (the 3334 + inserted 1 = 3335). + ############################### + + # First, delete the data except for 2 rows (will be replicated) + $node_publisher->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); + + # Then insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is in prepared state on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber'); + + # Insert a different record (now we are outside of the 2PC transaction) + # Note: the 2PC transaction still holds row locks so make sure this insert is for a separare primary key + $node_publisher->safe_psql('postgres', + "INSERT INTO test_tab VALUES (99999, 'foobar')"); + + # 2PC transaction gets committed + $node_publisher->safe_psql('postgres', + "COMMIT PREPARED 'test_prepared_tab';"); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is committed on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3335|3335|3335), + 'Rows inserted by 2PC (as well as outside insert) have committed on subscriber, and extra columns contain local defaults' + ); + + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is committed on subscriber'); + + # Cleanup the test data + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE a > 2;"); + $node_publisher->wait_for_catchup($appname); +} + ############################### # Setup ############################### @@ -48,6 +285,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql( 'postgres', " CREATE SUBSCRIPTION tap_sub @@ -77,229 +318,22 @@ my $result = $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab"); is($result, qq(2|2|2), 'check initial data was copied to subscriber'); -############################### -# Test 2PC PREPARE / COMMIT PREPARED. -# 1. Data is streamed as a 2PC transaction. -# 2. Then do commit prepared. -# -# Expect all data is replicated on subscriber side after the commit. -############################### - -# check that 2PC gets replicated to subscriber -# Insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is in prepared state on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber'); +test_streaming($node_publisher, $node_subscriber, $appname); -# 2PC transaction gets committed -$node_publisher->safe_psql('postgres', - "COMMIT PREPARED 'test_prepared_tab';"); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is committed on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), - 'Rows inserted by 2PC have committed on subscriber, and extra columns contain local defaults' +################################### +# Test using streaming mode 'apply' +################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" ); -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is committed on subscriber'); - -############################### -# Test 2PC PREPARE / ROLLBACK PREPARED. -# 1. Table is deleted back to 2 rows which are replicated on subscriber. -# 2. Data is streamed using 2PC. -# 3. Do rollback prepared. -# -# Expect data rolls back leaving only the original 2 rows. -############################### - -# First, delete the data except for 2 rows (will be replicated) -$node_publisher->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); - -# Then insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is in prepared state on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber'); - -# 2PC transaction gets aborted -$node_publisher->safe_psql('postgres', - "ROLLBACK PREPARED 'test_prepared_tab';"); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is aborted on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(2|2|2), - 'Rows inserted by 2PC are rolled back, leaving only the original 2 rows'); - -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is aborted on subscriber'); - -############################### -# Check that 2PC COMMIT PREPARED is decoded properly on crash restart. -# 1. insert, update and delete enough rows to exceed the 64kB limit. -# 2. Then server crashes before the 2PC transaction is committed. -# 3. After servers are restarted the pending transaction is committed. -# -# Expect all data is replicated on subscriber side after the commit. -# Note: both publisher and subscriber do crash/restart. -############################### - -$node_publisher->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_subscriber->stop('immediate'); -$node_publisher->stop('immediate'); - -$node_publisher->start; -$node_subscriber->start; - -# commit post the restart -$node_publisher->safe_psql('postgres', - "COMMIT PREPARED 'test_prepared_tab';"); -$node_publisher->wait_for_catchup($appname); - -# check inserts are visible -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), - 'Rows inserted by 2PC have committed on subscriber, and extra columns contain local defaults' -); - -############################### -# Do INSERT after the PREPARE but before ROLLBACK PREPARED. -# 1. Table is deleted back to 2 rows which are replicated on subscriber. -# 2. Data is streamed using 2PC. -# 3. A single row INSERT is done which is after the PREPARE. -# 4. Then do a ROLLBACK PREPARED. -# -# Expect the 2PC data rolls back leaving only 3 rows on the subscriber -# (the original 2 + inserted 1). -############################### - -# First, delete the data except for 2 rows (will be replicated) -$node_publisher->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); - -# Then insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is in prepared state on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber'); - -# Insert a different record (now we are outside of the 2PC transaction) -# Note: the 2PC transaction still holds row locks so make sure this insert is for a separate primary key -$node_publisher->safe_psql('postgres', - "INSERT INTO test_tab VALUES (99999, 'foobar')"); - -# 2PC transaction gets aborted -$node_publisher->safe_psql('postgres', - "ROLLBACK PREPARED 'test_prepared_tab';"); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is aborted on subscriber, -# but the extra INSERT outside of the 2PC still was replicated -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3|3|3), 'check the outside insert was copied to subscriber'); - -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is aborted on subscriber'); - -############################### -# Do INSERT after the PREPARE but before COMMIT PREPARED. -# 1. Table is deleted back to 2 rows which are replicated on subscriber. -# 2. Data is streamed using 2PC. -# 3. A single row INSERT is done which is after the PREPARE. -# 4. Then do a COMMIT PREPARED. -# -# Expect 2PC data + the extra row are on the subscriber -# (the 3334 + inserted 1 = 3335). -############################### - -# First, delete the data except for 2 rows (will be replicated) -$node_publisher->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); - -# Then insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is in prepared state on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber'); - -# Insert a different record (now we are outside of the 2PC transaction) -# Note: the 2PC transaction still holds row locks so make sure this insert is for a separare primary key -$node_publisher->safe_psql('postgres', - "INSERT INTO test_tab VALUES (99999, 'foobar')"); - -# 2PC transaction gets committed -$node_publisher->safe_psql('postgres', - "COMMIT PREPARED 'test_prepared_tab';"); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is committed on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3335|3335|3335), - 'Rows inserted by 2PC (as well as outside insert) have committed on subscriber, and extra columns contain local defaults' +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = apply)" ); +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" +) or die "Timed out while waiting for apply to restart after changing PUBLICATION"; -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is committed on subscriber'); +test_streaming($node_publisher, $node_subscriber, $appname); ############################### # check all the cleanup -- 2.18.4 ^ permalink raw reply [nested|flat] 66+ messages in thread
* Re: Perform streaming logical transactions by background workers and parallel apply @ 2022-05-18 07:11 Peter Smith <[email protected]> parent: [email protected] <[email protected]> 1 sibling, 1 reply; 66+ messages in thread From: Peter Smith @ 2022-05-18 07:11 UTC (permalink / raw) To: [email protected] <[email protected]>; +Cc: [email protected] <[email protected]>; Amit Kapila <[email protected]>; PostgreSQL Hackers <[email protected]> "Here are my review comments for v6-0001. ====== 1. General I saw that now in most places you are referring to the new kind of worker as the "apply background worker". But there are a few comments remaining that still refer to "bgworker". Please search the entire patch for "bgworker" in the comments and replace them with "apply background worker". ====== 2. Commit message We also need to allow stream_stop to complete by the apply background worker to finish it to avoid deadlocks because T-1's current stream of changes can update rows in conflicting order with T-2's next stream of changes. Something is not right with this wording: "to complete by the apply background worker to finish it...". Maybe just omit the words "to finish it" (??). ~~~ 3. Commit message This patch also extends the subscription streaming option so that... SUGGESTION This patch also extends the SUBSCRIPTION 'streaming' option so that... ====== 4. src/backend/commands/subscriptioncmds.c - defGetStreamingMode +/* + * Extract the streaming mode value from a DefElem. This is like + * defGetBoolean() but also accepts the special value and "apply". + */ +static char +defGetStreamingMode(DefElem *def) Typo: "special value and..." -> "special value of..." ====== 5. src/backend/replication/logical/launcher.c - logicalrep_worker_launch + + if (subworker_dsm == DSM_HANDLE_INVALID) + snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyWorkerMain"); + else + snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyBgworkerMain"); + + 5a. This condition should be using the new 'is_subworker' bool 5b. Double blank lines? ~~~ 6. src/backend/replication/logical/launcher.c - logicalrep_worker_launch - else + else if (subworker_dsm == DSM_HANDLE_INVALID) snprintf(bgw.bgw_name, BGW_MAXLEN, "logical replication worker for subscription %u", subid); + else + snprintf(bgw.bgw_name, BGW_MAXLEN, + "logical replication apply worker for subscription %u", subid); snprintf(bgw.bgw_type, BGW_MAXLEN, "logical replication worker"); This condition also should be using the new 'is_subworker' bool ~~~ 7. src/backend/replication/logical/launcher.c - logicalrep_worker_stop_internal + + Assert(LWLockHeldByMe(LogicalRepWorkerLock)); + I think there should be a comment here to say that this lock is required/expected to be released by the caller of this function. ====== 8. src/backend/replication/logical/origin.c - replorigin_session_setup @@ -1068,7 +1068,7 @@ ReplicationOriginExitCleanup(int code, Datum arg) * with replorigin_session_reset(). */ void -replorigin_session_setup(RepOriginId node) +replorigin_session_setup(RepOriginId node, bool acquire) { This function has been problematic for several reviews. I saw that you removed the previously confusing comment but I still feel some kind of explanation is needed for the vague 'acquire' parameter. OTOH perhaps if you just change the param name to 'must_acquire' then I think it would be self-explanatory. ====== 9. src/backend/replication/logical/worker.c - General Some of the logs have a prefix "[Apply BGW #%u]" and some do not; I did not really understand how you decided to prefix or not so I did not comment about them individually. Are they all OK? Perhaps if you can explain the reason for the choices I can review it better next time. ~~~ 10. src/backend/replication/logical/worker.c - General There are multiple places in the code where there is code checking if/else for bgworker or normal apply worker. And in those places, there is often a comment like: "If we are in main apply worker..." But it is redundant to say "If we are" because we know we are. Instead, those cases should say a comment at the top of the else like: /* This is the main apply worker. */ And then the "If we are in main apply worker" text can be removed from the comment. There are many examples in the patch like this. Please search and modify all of them. ~~~ 11. src/backend/replication/logical/worker.c - file header comment The whole comment is similar to the commit message so any changes made there (for #2, #3) should be made here also. ~~~ 12. src/backend/replication/logical/worker.c +typedef struct WorkerEntry +{ + TransactionId xid; + WorkerState *wstate; +} WorkerEntry; Missing comment for this structure ~~~ 13. src/backend/replication/logical/worker.c WorkerState WorkerEntry I felt that these struct names seem too generic - shouldn't they be something more like ApplyBgworkerState, ApplyBgworkerEntry ~~~ 14. src/backend/replication/logical/worker.c +static List *ApplyWorkersIdleList = NIL; IMO maybe ApplyWorkersFreeList is a better name than IdleList for this. "Idle" sounds just like it is paused rather than available for someone else to use. If you change this then please search the rest of the patch for mentions in log messages etc ~~~ 15. src/backend/replication/logical/worker.c +static WorkerState *stream_apply_worker = NULL; + +/* check if we apply transaction in apply bgworker */ +#define apply_bgworker_active() (in_streamed_transaction && stream_apply_worker != NULL) Wording: "if we apply transaction" -> "if we are applying the transaction" ~~~ 16. src/backend/replication/logical/worker.c - handle_streamed_transaction + * For the main apply worker, if in streaming mode (receiving a block of + * streamed transaction), we send the data to the apply background worker. + * + * For the apply background worker, define a savepoint if new subtransaction + * was started. * * Returns true for streamed transactions, false otherwise (regular mode). */ static bool handle_streamed_transaction(LogicalRepMsgType action, StringInfo s) 16a. Typo: "if new subtransaction" -> "if a new subtransaction" 16b. That "regular mode" comment seems not quite right because IIUC it also returns false also for a bgworker (which hardly seems like a "regular mode") ~~~ 17. src/backend/replication/logical/worker.c - handle_streamed_transaction - /* not in streaming mode */ - if (!in_streamed_transaction) + /* + * Return if we are not in streaming mode and are not in an apply + * background worker. + */ + if (!in_streamed_transaction && !am_apply_bgworker()) return false; Somehow I found this condition confusing, the comment is not helpful either because it just says exactly what the code says. Can you give a better explanatory comment? e.g. Maybe the comment should be: "Return if not in streaming mode (unless this is an apply background worker)" e.g. Maybe condition is easier to understand if written as: if (!(in_streamed_transaction || am_apply_bgworker())) ~~~ 18. src/backend/replication/logical/worker.c - handle_streamed_transaction + if (action == LOGICAL_REP_MSG_RELATION) + { + LogicalRepRelation *rel = logicalrep_read_rel(s); + logicalrep_relmap_update(rel); + } + + } + else + { + /* Add the new subxact to the array (unless already there). */ + subxact_info_add(current_xid); Unnecessary blank line. ~~~ 19. src/backend/replication/logical/worker.c - find_or_start_apply_bgworker + if (found) + { + entry->wstate->pstate->state = APPLY_BGWORKER_BUSY; + return entry->wstate; + } + else if (!start) + return NULL; + + /* If there is at least one worker in the idle list, then take one. */ + if (list_length(ApplyWorkersIdleList) > 0) I felt that there should be a comment (after the return NULL) that says: /* * Start a new apply background worker */ ~~~ 20. src/backend/replication/logical/worker.c - apply_bgworker_free +/* + * Add the worker to the freelist and remove the entry from hash table. + */ +static void +apply_bgworker_free(WorkerState *wstate) 20a. Typo: "freelist" -> "free list" 20b. Elsewhere (and in the log message) this is called the idle list (but actually I prefer "free list" like in this comment). See also comment #14. ~~~ 21. src/backend/replication/logical/worker.c - apply_bgworker_free + hash_search(ApplyWorkersHash, &xid, + HASH_REMOVE, &found); 21a. If you are not going to check the value of ‘found’ then why bother to pass this param at all; can’t you just pass NULL? (I think I asked the same question in a previous review) 21b. The wrapping over 2 lines seems unnecessary here. ~~~ 22. src/backend/replication/logical/worker.c - apply_handle_stream_start /* - * Initialize the worker's stream_fileset if we haven't yet. This will be - * used for the entire duration of the worker so create it in a permanent - * context. We create this on the very first streaming message from any - * transaction and then use it for this and other streaming transactions. - * Now, we could create a fileset at the start of the worker as well but - * then we won't be sure that it will ever be used. + * If we are in main apply worker, check if there is any free bgworker + * we can use to process this transaction. */ - if (MyLogicalRepWorker->stream_fileset == NULL) + stream_apply_worker = apply_bgworker_find_or_start(stream_xid, first_segment); 22a. Typo: "in main apply worker" -> "in the main apply worker" 22b. Since this is not if/else code, it might be better to put Assert(!am_apply_bgworker()); above this just to make it more clear. ~~~ 23. src/backend/replication/logical/worker.c - apply_handle_stream_start + /* + * If we have free worker or we already started to apply this + * transaction in bgworker, we pass the data to worker. + */ SUGGESTION If we have found a free worker or if we are already applying this transaction in an apply background worker, then we pass the data to that worker. ~~~ 24. src/backend/replication/logical/worker.c - apply_handle_stream_abort +apply_handle_stream_abort(StringInfo s) { - StringInfoData s2; - int nchanges; - char path[MAXPGPATH]; - char *buffer = NULL; - MemoryContext oldcxt; - BufFile *fd; + TransactionId xid; + TransactionId subxid; - maybe_start_skipping_changes(lsn); + if (in_streamed_transaction) + ereport(ERROR, + (errcode(ERRCODE_PROTOCOL_VIOLATION), + errmsg_internal("STREAM COMMIT message without STREAM STOP"))); Typo? Shouldn't that errmsg say "STREAM ABORT message..." instead of "STREAM COMMIT message..." ~~~ 25. src/backend/replication/logical/worker.c - apply_handle_stream_abort + for(i = list_length(subxactlist) - 1; i >= 0; i--) + { Missing space after "for" ~~~ 26. src/backend/replication/logical/worker.c - apply_handle_stream_abort + if (found) + { + elog(LOG, "rolled back to savepoint %s", spname); + RollbackToSavepoint(spname); + CommitTransactionCommand(); + subxactlist = list_truncate(subxactlist, i + 1); + } Does this need to log anything if nothing was found? Or is it ok to leave as-is and silently ignore it? ~~~ 27. src/backend/replication/logical/worker.c - LogicalApplyBgwLoop + if (len == 0) + { + elog(LOG, "[Apply BGW #%u] got zero-length message, stopping", pst->n); + break; + } Maybe it is unnecessary to say "stopping" because it will say that in the next log anyway when it breaks out of the main loop. ~~~ 28. src/backend/replication/logical/worker.c - LogicalApplyBgwLoop + default: + elog(ERROR, "unexpected message"); + break; Perhaps the switch byte should be in a variable so then you can log what was the unexpected byte code received. e.g. Similar to apply_handle_tuple_routing function. ~~~ 29. src/backend/replication/logical/worker.c - LogicalApplyBgwMain + /* + * The apply bgworker don't need to monopolize this replication origin + * which was already acquired by its leader process. + */ + replorigin_session_setup(originid, false); + replorigin_session_origin = originid; + CommitTransactionCommand(); Typo: The apply bgworker don't need ..." -> "The apply background workers don't need ..." or -> "The apply background worker doesn't need ..." ~~~ 30. src/backend/replication/logical/worker.c - apply_bgworker_setup +/* + * Start apply worker background worker process and allocate shared memory for + * it. + */ +static WorkerState * +apply_bgworker_setup(void) Typo: "apply worker background worker process" -> "apply background worker process" ~~~ 31. src/backend/replication/logical/worker.c - apply_bgworker_wait_for + /* If any workers (or the postmaster) have died, we have failed. */ + if (status == APPLY_BGWORKER_EXIT) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("Background worker %u failed to apply transaction %u", + wstate->pstate->n, wstate->pstate->stream_xid))); The errmsg should start with a lowercase letter. ~~~ 32. src/backend/replication/logical/worker.c - check_workers_status + /* + * We don't lock here as in the worst case we will just detect the + * failure of worker a bit later. + */ + if (wstate->pstate->state == APPLY_BGWORKER_EXIT) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("Background worker %u exited unexpectedly", + wstate->pstate->n))); The errmsg should start with a lowercase letter. ~~~ 33. src/backend/replication/logical/worker.c - check_workers_status +/* Set the state of apply background worker */ +static void +apply_bgworker_set_state(char state) Maybe OK, or perhaps choose from one of: - "Set the state of an apply background worker" - "Set the apply background worker state" ====== 34. src/bin/pg_dump/pg_dump.c - getSubscriptions @@ -4450,7 +4450,7 @@ getSubscriptions(Archive *fout) if (fout->remoteVersion >= 140000) appendPQExpBufferStr(query, " s.substream,\n"); else - appendPQExpBufferStr(query, " false AS substream,\n"); + appendPQExpBufferStr(query, " 'f' AS substream,\n"); Is that logic right? Before this patch the attribute was bool; now it is char. So doesn't there need to be some conversion/mapping here for when you read from >= 140000 but it was still bool so you need to convert 'false' -> 'f' and 'true' -> 't'? ====== 35. src/include/replication/origin.h @@ -53,7 +53,7 @@ extern XLogRecPtr replorigin_get_progress(RepOriginId node, bool flush); extern void replorigin_session_advance(XLogRecPtr remote_commit, XLogRecPtr local_commit); -extern void replorigin_session_setup(RepOriginId node); +extern void replorigin_session_setup(RepOriginId node, bool acquire); As previously suggested in comment #8 maybe the 2nd parm should be 'must_acquire'. ====== 36. src/include/replication/worker_internal.h @@ -60,6 +60,8 @@ typedef struct LogicalRepWorker */ FileSet *stream_fileset; + bool subworker; + Probably this new member deserves a comment. ------ Kind Regards, Peter Smith. Fujitsu Australia ^ permalink raw reply [nested|flat] 66+ messages in thread
* Re: Perform streaming logical transactions by background workers and parallel apply @ 2022-05-19 06:22 Peter Smith <[email protected]> parent: [email protected] <[email protected]> 1 sibling, 0 replies; 66+ messages in thread From: Peter Smith @ 2022-05-19 06:22 UTC (permalink / raw) To: [email protected] <[email protected]>; +Cc: [email protected] <[email protected]>; Amit Kapila <[email protected]>; PostgreSQL Hackers <[email protected]> Here are my review comments for v6-0002. ====== 1. src/test/subscription/t/015_stream.pl +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); - $node_publisher->wait_for_catchup($appname); - # Also wait for initial table sync to finish my $synced_query = "SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');"; $node_subscriber->poll_query_until('postgres', $synced_query) or die "Timed out while waiting for subscriber to synchronize data"; - my $result = $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab"); is($result, qq(2|2|2), 'check initial data was copied to subscriber'); 1a. Several whitespace lines became removed by the patch. IMO it was better (e.g. less squishy) how it looked originally. 1b. Maybe some more blank lines should be added to the 'apply' test part too, to match 1a. ~~~ 2. src/test/subscription/t/015_stream.pl +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" +) or die "Timed out while waiting for apply to restart after changing PUBLICATION"; Should that say "... after changing SUBSCRIPTION"? ~~~ 3. src/test/subscription/t/016_stream_subxact.pl +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" +) or die "Timed out while waiting for apply to restart after changing PUBLICATION"; + Should that say "... after changing SUBSCRIPTION"? ~~~ 4. src/test/subscription/t/017_stream_ddl.pl +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" +) or die "Timed out while waiting for apply to restart after changing PUBLICATION"; + Should that say "... after changing SUBSCRIPTION"? ~~~ 5. .../t/018_stream_subxact_abort.pl +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" +) or die "Timed out while waiting for apply to restart after changing PUBLICATION"; Should that say "... after changing SUBSCRIPTION" ? ~~~ 6. .../t/019_stream_subxact_ddl_abort.pl +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" +) or die "Timed out while waiting for apply to restart after changing PUBLICATION"; + Should that say "... after changing SUBSCRIPTION"? ~~~ 7. .../subscription/t/023_twophase_stream.pl ############################### # Check initial data was copied to subscriber ############################### Perhaps the above comment now looks a bit out-of-place with the extra #####. Looks better now as just: # Check initial data was copied to the subscriber ~~~ 8. .../subscription/t/023_twophase_stream.pl +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" +) or die "Timed out while waiting for apply to restart after changing PUBLICATION"; Should that say "... after changing SUBSCRIPTION"? ------ Kind Regards, Peter Smith. Fujitsu Australia ^ permalink raw reply [nested|flat] 66+ messages in thread
* RE: Perform streaming logical transactions by background workers and parallel apply @ 2022-05-30 08:51 [email protected] <[email protected]> parent: Peter Smith <[email protected]> 0 siblings, 1 reply; 66+ messages in thread From: [email protected] @ 2022-05-30 08:51 UTC (permalink / raw) To: Peter Smith <[email protected]>; [email protected] <[email protected]>; +Cc: [email protected] <[email protected]>; Amit Kapila <[email protected]>; PostgreSQL Hackers <[email protected]> On Wed, May 18, 2022 3:11 PM Peter Smith <[email protected]> wrote: > "Here are my review comments for v6-0001. Thanks for your comments. > 7. src/backend/replication/logical/launcher.c - logicalrep_worker_stop_internal > > + > + Assert(LWLockHeldByMe(LogicalRepWorkerLock)); > + > > I think there should be a comment here to say that this lock is > required/expected to be released by the caller of this function. IMHO, it maybe not a problem to read code here. In addition, keep consistent with other places where invoke this function in the same file. So I did not change this. > 9. src/backend/replication/logical/worker.c - General > > Some of the logs have a prefix "[Apply BGW #%u]" and some do not; I > did not really understand how you decided to prefix or not so I did > not comment about them individually. Are they all OK? Perhaps if you > can explain the reason for the choices I can review it better next > time. I think most of these logs should be logged in debug mode. So I changed them to "DEBUG1" level. And I added the prefix to all messages logged by apply background worker and deleted some logs that I think maybe not very helpful. > 11. src/backend/replication/logical/worker.c - file header comment > > The whole comment is similar to the commit message so any changes made > there (for #2, #3) should be made here also. Improve the comments as suggested in #2. Sorry but I did not find same message as #2 here. > 13. src/backend/replication/logical/worker.c > > WorkerState > WorkerEntry > > I felt that these struct names seem too generic - shouldn't they be > something more like ApplyBgworkerState, ApplyBgworkerEntry > > ~~~ I think we have used "ApplyBgworkerState" in the patch. So I improved this with the following modifications: ``` ApplyBgworkerState -> ApplyBgworkerStatus WorkerState -> ApplyBgworkerState WorkerEntry -> ApplyBgworkerEntry ``` BTW, I also modified the relevant comments and variable names. > 16. src/backend/replication/logical/worker.c - handle_streamed_transaction > > + * For the main apply worker, if in streaming mode (receiving a block of > + * streamed transaction), we send the data to the apply background worker. > + * > + * For the apply background worker, define a savepoint if new subtransaction > + * was started. > * > * Returns true for streamed transactions, false otherwise (regular mode). > */ > static bool > handle_streamed_transaction(LogicalRepMsgType action, StringInfo s) > > 16a. > Typo: "if new subtransaction" -> "if a new subtransaction" > > 16b. > That "regular mode" comment seems not quite right because IIUC it also > returns false also for a bgworker (which hardly seems like a "regular > mode") 16a. Improved it as suggested. 16b. Changed the comment as follows: From: ``` * Returns true for streamed transactions, false otherwise (regular mode). ``` To: ``` * For non-streamed transactions, returns false; * For streamed transactions, returns true if in main apply worker, false * otherwise. ``` > 19. src/backend/replication/logical/worker.c - find_or_start_apply_bgworker > > + if (found) > + { > + entry->wstate->pstate->state = APPLY_BGWORKER_BUSY; > + return entry->wstate; > + } > + else if (!start) > + return NULL; > + > + /* If there is at least one worker in the idle list, then take one. */ > + if (list_length(ApplyWorkersIdleList) > 0) > > I felt that there should be a comment (after the return NULL) that says: > > /* > * Start a new apply background worker > */ > > ~~~ Improve this comment here. After the code that you mentioned, it will try to get a apply background worker (try to start one or take one from idle list). So I change the comment as follows: From: ``` /* If there is at least one worker in the idle list, then take one. */ ``` To: ``` /* * Now, we try to get a apply background worker. * If there is at least one worker in the idle list, then take one. * Otherwise, we try to start a new apply background worker. */ ``` > 22. src/backend/replication/logical/worker.c - apply_handle_stream_start > > /* > - * Initialize the worker's stream_fileset if we haven't yet. This will be > - * used for the entire duration of the worker so create it in a permanent > - * context. We create this on the very first streaming message from any > - * transaction and then use it for this and other streaming transactions. > - * Now, we could create a fileset at the start of the worker as well but > - * then we won't be sure that it will ever be used. > + * If we are in main apply worker, check if there is any free bgworker > + * we can use to process this transaction. > */ > - if (MyLogicalRepWorker->stream_fileset == NULL) > + stream_apply_worker = apply_bgworker_find_or_start(stream_xid, > first_segment); > > 22a. > Typo: "in main apply worker" -> "in the main apply worker" > > 22b. > Since this is not if/else code, it might be better to put > Assert(!am_apply_bgworker()); above this just to make it more clear. 22a. Improved it as suggested. 22b. IMHO, since we have `if (am_apply_bgworker())` above and it will return in this if-condition, so I just think Assert() might be a bit redundant here. So I did not change this. > 26. src/backend/replication/logical/worker.c - apply_handle_stream_abort > > + if (found) > + { > + elog(LOG, "rolled back to savepoint %s", spname); > + RollbackToSavepoint(spname); > + CommitTransactionCommand(); > + subxactlist = list_truncate(subxactlist, i + 1); > + } > > Does this need to log anything if nothing was found? Or is it ok to > leave as-is and silently ignore it? Yes, I think it is okay. > 33. src/backend/replication/logical/worker.c - check_workers_status > > +/* Set the state of apply background worker */ > +static void > +apply_bgworker_set_state(char state) > > Maybe OK, or perhaps choose from one of: > - "Set the state of an apply background worker" > - "Set the apply background worker state" Improve it by using the second one. > 34. src/bin/pg_dump/pg_dump.c - getSubscriptions > > @@ -4450,7 +4450,7 @@ getSubscriptions(Archive *fout) > if (fout->remoteVersion >= 140000) > appendPQExpBufferStr(query, " s.substream,\n"); > else > - appendPQExpBufferStr(query, " false AS substream,\n"); > + appendPQExpBufferStr(query, " 'f' AS substream,\n"); > > > Is that logic right? Before this patch the attribute was bool; now it > is char. So doesn't there need to be some conversion/mapping here for > when you read from >= 140000 but it was still bool so you need to > convert 'false' -> 'f' and 'true' -> 't'? Yes, I think it is right. We could handle the input of option "streaming" : on/true/off/false/apply. The rest of the comments are improved as suggested. And thanks for Shi Yu to improve the patch 0002 by addressing the comments in [1]. Attach the new patches(only changed 0001 and 0002) [1] - https://www.postgresql.org/message-id/CAHut%2BPv_0nfUxriwxBQnZTOF5dy5nfG5NtWMr8e00mPrt2Vjzw%40mail.g... Regards, Wang wei Attachments: [application/octet-stream] v7-0001-Perform-streaming-logical-transactions-by-backgro.patch (74.7K, ../../OS3PR01MB6275797F66EF0A47EEB2D8FC9EDD9@OS3PR01MB6275.jpnprd01.prod.outlook.com/2-v7-0001-Perform-streaming-logical-transactions-by-backgro.patch) download | inline diff: From 0322e37da7dca9b3fedae439551b641c101030c3 Mon Sep 17 00:00:00 2001 From: "houzj.fnst" <[email protected]> Date: Wed, 20 Apr 2022 16:45:07 +0800 Subject: [PATCH v7 1/4] Perform streaming logical transactions by background workers Currently, for large transactions, the publisher sends the data in multiple streams (changes divided into chunks depending upon logical_decoding_work_mem), and then on the subscriber-side, the apply worker writes the changes into temporary files and once it receives the commit, it reads from the file and applies the entire transaction. To improve the performance of such transactions, we can instead allow them to be applied via background workers. In this approach, we assign a new apply background worker (if available) as soon as the xact's first stream is received and the main apply worker will send changes to this new worker via shared memory. The apply background worker will directly apply the change instead of writing it to temporary files. We keep this worker assigned till the transaction commit is received and also wait for the worker to finish at commit. This preserves commit ordering and avoids writing to and reading from file in most cases. We still need to spill if there is no worker available. We also need to allow stream_stop to complete by the apply background worker to avoid deadlocks because T-1's current stream of changes can update rows in conflicting order with T-2's next stream of changes. This patch also extends the SUBSCRIPTION 'streaming' option so that the user can control whether to apply the streaming transaction in an apply background worker or spill the change to disk. User can set the streaming option to 'on/off', 'apply'. For now, 'apply' means the streaming will be applied via a apply background worker if available. 'on' means the streaming transaction will be spilled to disk. --- doc/src/sgml/catalogs.sgml | 10 +- doc/src/sgml/ref/create_subscription.sgml | 24 +- src/backend/commands/subscriptioncmds.c | 66 +- src/backend/postmaster/bgworker.c | 3 + src/backend/replication/logical/launcher.c | 86 +- src/backend/replication/logical/origin.c | 6 +- src/backend/replication/logical/tablesync.c | 10 +- src/backend/replication/logical/worker.c | 1402 +++++++++++++++++-- src/backend/utils/activity/wait_event.c | 3 + src/bin/pg_dump/pg_dump.c | 6 +- src/include/catalog/pg_subscription.h | 17 +- src/include/replication/logicalworker.h | 1 + src/include/replication/origin.h | 2 +- src/include/replication/worker_internal.h | 14 +- src/include/utils/wait_event.h | 1 + src/test/regress/expected/subscription.out | 2 +- src/tools/pgindent/typedefs.list | 4 + 17 files changed, 1465 insertions(+), 192 deletions(-) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index c00c93dd7b..5cb39b18bd 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -7873,11 +7873,15 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l <row> <entry role="catalog_table_entry"><para role="column_definition"> - <structfield>substream</structfield> <type>bool</type> + <structfield>substream</structfield> <type>char</type> </para> <para> - If true, the subscription will allow streaming of in-progress - transactions + Controls how to handle the streaming of in-progress transactions: + <literal>f</literal> = disallow streaming of in-progress transactions, + <literal>t</literal> = spill the changes of in-progress transactions to + disk and apply at once after the transaction is committed on the + publisher, + <literal>a</literal> = apply changes directly using a background worker </para></entry> </row> diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml index 203bb41844..22bd81800d 100644 --- a/doc/src/sgml/ref/create_subscription.sgml +++ b/doc/src/sgml/ref/create_subscription.sgml @@ -217,13 +217,29 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl </varlistentry> <varlistentry> - <term><literal>streaming</literal> (<type>boolean</type>)</term> + <term><literal>streaming</literal> (<type>enum</type>)</term> <listitem> <para> Specifies whether to enable streaming of in-progress transactions - for this subscription. By default, all transactions - are fully decoded on the publisher and only then sent to the - subscriber as a whole. + for this subscription. The default value is <literal>off</literal>, + all transactions are fully decoded on the publisher and only then + sent to the subscriber as a whole. + </para> + + <para> + If set to <literal>on</literal>, the changes of transaction are + written to temporary files and then applied at once after the + transaction is committed on the publisher. + </para> + + <para> + If set to <literal>apply</literal> incoming + changes are directly applied via one of the background workers, if + available. If no background worker is free to handle streaming + transaction then the changes are written to a file and applied after + the transaction is committed. Note that if an error happens when + applying changes in a background worker, it might not report the + finish LSN of the remote transaction in the server log. </para> </listitem> </varlistentry> diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c index 690cdaa426..79519cf695 100644 --- a/src/backend/commands/subscriptioncmds.c +++ b/src/backend/commands/subscriptioncmds.c @@ -83,7 +83,7 @@ typedef struct SubOpts bool copy_data; bool refresh; bool binary; - bool streaming; + char streaming; bool twophase; bool disableonerr; XLogRecPtr lsn; @@ -95,6 +95,62 @@ static List *merge_publications(List *oldpublist, List *newpublist, bool addpub, static void ReportSlotConnectionError(List *rstates, Oid subid, char *slotname, char *err); +/* + * Extract the streaming mode value from a DefElem. This is like + * defGetBoolean() but also accepts the special value of "apply". + */ +static char +defGetStreamingMode(DefElem *def) +{ + /* + * If no parameter given, assume "true" is meant. + */ + if (def->arg == NULL) + return SUBSTREAM_ON; + + /* + * Allow 0, 1, "true", "false", "on", "off" or "apply". + */ + switch (nodeTag(def->arg)) + { + case T_Integer: + switch (intVal(def->arg)) + { + case 0: + return SUBSTREAM_OFF; + case 1: + return SUBSTREAM_ON; + default: + /* otherwise, error out below */ + break; + } + break; + default: + { + char *sval = defGetString(def); + + /* + * The set of strings accepted here should match up with the + * grammar's opt_boolean_or_string production. + */ + if (pg_strcasecmp(sval, "false") == 0 || + pg_strcasecmp(sval, "off") == 0) + return SUBSTREAM_OFF; + if (pg_strcasecmp(sval, "true") == 0 || + pg_strcasecmp(sval, "on") == 0) + return SUBSTREAM_ON; + if (pg_strcasecmp(sval, "apply") == 0) + return SUBSTREAM_APPLY; + } + break; + } + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s requires a Boolean value or \"apply\"", + def->defname))); + return SUBSTREAM_OFF; /* keep compiler quiet */ +} + /* * Common option parsing function for CREATE and ALTER SUBSCRIPTION commands. * @@ -132,7 +188,7 @@ parse_subscription_options(ParseState *pstate, List *stmt_options, if (IsSet(supported_opts, SUBOPT_BINARY)) opts->binary = false; if (IsSet(supported_opts, SUBOPT_STREAMING)) - opts->streaming = false; + opts->streaming = SUBSTREAM_OFF; if (IsSet(supported_opts, SUBOPT_TWOPHASE_COMMIT)) opts->twophase = false; if (IsSet(supported_opts, SUBOPT_DISABLE_ON_ERR)) @@ -233,7 +289,7 @@ parse_subscription_options(ParseState *pstate, List *stmt_options, errorConflictingDefElem(defel, pstate); opts->specified_opts |= SUBOPT_STREAMING; - opts->streaming = defGetBoolean(defel); + opts->streaming = defGetStreamingMode(defel); } else if (strcmp(defel->defname, "two_phase") == 0) { @@ -601,7 +657,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt, values[Anum_pg_subscription_subowner - 1] = ObjectIdGetDatum(owner); values[Anum_pg_subscription_subenabled - 1] = BoolGetDatum(opts.enabled); values[Anum_pg_subscription_subbinary - 1] = BoolGetDatum(opts.binary); - values[Anum_pg_subscription_substream - 1] = BoolGetDatum(opts.streaming); + values[Anum_pg_subscription_substream - 1] = CharGetDatum(opts.streaming); values[Anum_pg_subscription_subtwophasestate - 1] = CharGetDatum(opts.twophase ? LOGICALREP_TWOPHASE_STATE_PENDING : @@ -1060,7 +1116,7 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt, if (IsSet(opts.specified_opts, SUBOPT_STREAMING)) { values[Anum_pg_subscription_substream - 1] = - BoolGetDatum(opts.streaming); + CharGetDatum(opts.streaming); replaces[Anum_pg_subscription_substream - 1] = true; } diff --git a/src/backend/postmaster/bgworker.c b/src/backend/postmaster/bgworker.c index 40601aefd9..40ccb8993c 100644 --- a/src/backend/postmaster/bgworker.c +++ b/src/backend/postmaster/bgworker.c @@ -128,6 +128,9 @@ static const struct }, { "ApplyWorkerMain", ApplyWorkerMain + }, + { + "ApplyBgworkerMain", ApplyBgworkerMain } }; diff --git a/src/backend/replication/logical/launcher.c b/src/backend/replication/logical/launcher.c index bd5f78cf9a..58337bef91 100644 --- a/src/backend/replication/logical/launcher.c +++ b/src/backend/replication/logical/launcher.c @@ -73,6 +73,7 @@ static void logicalrep_launcher_onexit(int code, Datum arg); static void logicalrep_worker_onexit(int code, Datum arg); static void logicalrep_worker_detach(void); static void logicalrep_worker_cleanup(LogicalRepWorker *worker); +static void logicalrep_worker_stop_internal(LogicalRepWorker *worker); static bool on_commit_launcher_wakeup = false; @@ -223,6 +224,10 @@ logicalrep_worker_find(Oid subid, Oid relid, bool only_running) { LogicalRepWorker *w = &LogicalRepCtx->workers[i]; + /* We only need main apply worker or table sync worker here */ + if (w->subworker) + continue; + if (w->in_use && w->subid == subid && w->relid == relid && (!only_running || w->proc)) { @@ -259,11 +264,11 @@ logicalrep_workers_find(Oid subid, bool only_running) } /* - * Start new apply background worker, if possible. + * Start new background worker, if possible. */ -void +bool logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, Oid userid, - Oid relid) + Oid relid, dsm_handle subworker_dsm) { BackgroundWorker bgw; BackgroundWorkerHandle *bgw_handle; @@ -273,6 +278,10 @@ logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, Oid userid, LogicalRepWorker *worker = NULL; int nsyncworkers; TimestampTz now; + bool is_subworker = (subworker_dsm != DSM_HANDLE_INVALID); + + /* We don't support table sync in subworker */ + Assert(!(is_subworker && OidIsValid(relid))); ereport(DEBUG1, (errmsg_internal("starting logical replication worker for subscription \"%s\"", @@ -350,7 +359,7 @@ retry: if (OidIsValid(relid) && nsyncworkers >= max_sync_workers_per_subscription) { LWLockRelease(LogicalRepWorkerLock); - return; + return false; } /* @@ -364,7 +373,7 @@ retry: (errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED), errmsg("out of logical replication worker slots"), errhint("You might need to increase max_logical_replication_workers."))); - return; + return false; } /* Prepare the worker slot. */ @@ -379,6 +388,7 @@ retry: worker->relstate = SUBREL_STATE_UNKNOWN; worker->relstate_lsn = InvalidXLogRecPtr; worker->stream_fileset = NULL; + worker->subworker = is_subworker; worker->last_lsn = InvalidXLogRecPtr; TIMESTAMP_NOBEGIN(worker->last_send_time); TIMESTAMP_NOBEGIN(worker->last_recv_time); @@ -396,19 +406,30 @@ retry: BGWORKER_BACKEND_DATABASE_CONNECTION; bgw.bgw_start_time = BgWorkerStart_RecoveryFinished; snprintf(bgw.bgw_library_name, BGW_MAXLEN, "postgres"); - snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyWorkerMain"); + + if (!is_subworker) + snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyWorkerMain"); + else + snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyBgworkerMain"); + if (OidIsValid(relid)) snprintf(bgw.bgw_name, BGW_MAXLEN, "logical replication worker for subscription %u sync %u", subid, relid); - else + else if (!is_subworker) snprintf(bgw.bgw_name, BGW_MAXLEN, "logical replication worker for subscription %u", subid); + else + snprintf(bgw.bgw_name, BGW_MAXLEN, + "logical replication apply worker for subscription %u", subid); snprintf(bgw.bgw_type, BGW_MAXLEN, "logical replication worker"); bgw.bgw_restart_time = BGW_NEVER_RESTART; bgw.bgw_notify_pid = MyProcPid; bgw.bgw_main_arg = Int32GetDatum(slot); + if (is_subworker) + memcpy(bgw.bgw_extra, &subworker_dsm, sizeof(dsm_handle)); + if (!RegisterDynamicBackgroundWorker(&bgw, &bgw_handle)) { /* Failed to start worker, so clean up the worker slot. */ @@ -421,11 +442,13 @@ retry: (errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED), errmsg("out of background worker slots"), errhint("You might need to increase max_worker_processes."))); - return; + return false; } /* Now wait until it attaches. */ WaitForReplicationWorkerAttach(worker, generation, bgw_handle); + + return true; } /* @@ -436,7 +459,6 @@ void logicalrep_worker_stop(Oid subid, Oid relid) { LogicalRepWorker *worker; - uint16 generation; LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); @@ -449,6 +471,22 @@ logicalrep_worker_stop(Oid subid, Oid relid) return; } + logicalrep_worker_stop_internal(worker); + + LWLockRelease(LogicalRepWorkerLock); +} + +/* + * Workhorse for logicalrep_worker_stop() and logicalrep_worker_detach(). Stop + * the worker and wait for it to die. + */ +static void +logicalrep_worker_stop_internal(LogicalRepWorker *worker) +{ + uint16 generation; + + Assert(LWLockHeldByMe(LogicalRepWorkerLock)); + /* * Remember which generation was our worker so we can check if what we see * is still the same one. @@ -485,10 +523,7 @@ logicalrep_worker_stop(Oid subid, Oid relid) * different, meaning that a different worker has taken the slot. */ if (!worker->in_use || worker->generation != generation) - { - LWLockRelease(LogicalRepWorkerLock); return; - } /* Worker has assigned proc, so it has started. */ if (worker->proc) @@ -522,8 +557,6 @@ logicalrep_worker_stop(Oid subid, Oid relid) LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); } - - LWLockRelease(LogicalRepWorkerLock); } /* @@ -599,6 +632,28 @@ logicalrep_worker_attach(int slot) static void logicalrep_worker_detach(void) { + /* + * If we are the main apply worker, stop all the apply background workers + * we started before. + */ + if (!MyLogicalRepWorker->subworker) + { + List *workers; + ListCell *lc; + + LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); + + workers = logicalrep_workers_find(MyLogicalRepWorker->subid, true); + foreach(lc, workers) + { + LogicalRepWorker *w = (LogicalRepWorker *) lfirst(lc); + if (w->subworker) + logicalrep_worker_stop_internal(w); + } + + LWLockRelease(LogicalRepWorkerLock); + } + /* Block concurrent access. */ LWLockAcquire(LogicalRepWorkerLock, LW_EXCLUSIVE); @@ -621,6 +676,7 @@ logicalrep_worker_cleanup(LogicalRepWorker *worker) worker->userid = InvalidOid; worker->subid = InvalidOid; worker->relid = InvalidOid; + worker->subworker = false; } /* @@ -868,7 +924,7 @@ ApplyLauncherMain(Datum main_arg) wait_time = wal_retrieve_retry_interval; logicalrep_worker_launch(sub->dbid, sub->oid, sub->name, - sub->owner, InvalidOid); + sub->owner, InvalidOid, DSM_HANDLE_INVALID); } } diff --git a/src/backend/replication/logical/origin.c b/src/backend/replication/logical/origin.c index 21937ab2d3..e1cfa495f8 100644 --- a/src/backend/replication/logical/origin.c +++ b/src/backend/replication/logical/origin.c @@ -1068,7 +1068,7 @@ ReplicationOriginExitCleanup(int code, Datum arg) * with replorigin_session_reset(). */ void -replorigin_session_setup(RepOriginId node) +replorigin_session_setup(RepOriginId node, bool must_acquire) { static bool registered_cleanup; int i; @@ -1110,7 +1110,7 @@ replorigin_session_setup(RepOriginId node) if (curstate->roident != node) continue; - else if (curstate->acquired_by != 0) + else if (curstate->acquired_by != 0 && must_acquire) { ereport(ERROR, (errcode(ERRCODE_OBJECT_IN_USE), @@ -1321,7 +1321,7 @@ pg_replication_origin_session_setup(PG_FUNCTION_ARGS) name = text_to_cstring((text *) DatumGetPointer(PG_GETARG_DATUM(0))); origin = replorigin_by_name(name, false); - replorigin_session_setup(origin); + replorigin_session_setup(origin, true); replorigin_session_origin = origin; diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c index 994c7a09d9..5eb60327b6 100644 --- a/src/backend/replication/logical/tablesync.c +++ b/src/backend/replication/logical/tablesync.c @@ -568,7 +568,8 @@ process_syncing_tables_for_apply(XLogRecPtr current_lsn) MySubscription->oid, MySubscription->name, MyLogicalRepWorker->userid, - rstate->relid); + rstate->relid, + DSM_HANDLE_INVALID); hentry->last_start_time = now; } } @@ -589,6 +590,9 @@ process_syncing_tables_for_apply(XLogRecPtr current_lsn) void process_syncing_tables(XLogRecPtr current_lsn) { + if (MyLogicalRepWorker->subworker) + return; + if (am_tablesync_worker()) process_syncing_tables_for_sync(current_lsn); else @@ -1269,7 +1273,7 @@ LogicalRepSyncTableStart(XLogRecPtr *origin_startpos) * time this tablesync was launched. */ originid = replorigin_by_name(originname, false); - replorigin_session_setup(originid); + replorigin_session_setup(originid, true); replorigin_session_origin = originid; *origin_startpos = replorigin_session_get_progress(false); @@ -1380,7 +1384,7 @@ LogicalRepSyncTableStart(XLogRecPtr *origin_startpos) true /* go backward */ , true /* WAL log */ ); UnlockRelationOid(ReplicationOriginRelationId, RowExclusiveLock); - replorigin_session_setup(originid); + replorigin_session_setup(originid, true); replorigin_session_origin = originid; } else diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c index fc210a9e7b..788c1c65c9 100644 --- a/src/backend/replication/logical/worker.c +++ b/src/backend/replication/logical/worker.c @@ -22,8 +22,25 @@ * STREAMED TRANSACTIONS * --------------------- * Streamed transactions (large transactions exceeding a memory limit on the - * upstream) are not applied immediately, but instead, the data is written - * to temporary files and then applied at once when the final commit arrives. + * upstream) are applied via one of two approaches. + * + * 1) Separate background workers + * + * Assign a new apply background worker (if available) as soon as the xact's + * first stream is received and the main apply worker will send changes to this + * new worker via shared memory. We keep this worker assigned till the + * transaction commit is received and also wait for the worker to finish at + * commit. This preserves commit ordering and avoids writing to and reading + * from file in most cases. We still need to spill if there is no worker + * available. We also need to allow stream_stop to complete by the background + * worker to avoid deadlocks because T-1's current stream of changes can update + * rows in conflicting order with T-2's next stream of changes. + * + * 2) Write to temporary files and apply when the final commit arrives + * + * If no worker is available to handle streamed transaction, the data is + * written to temporary files and then applied at once when the final commit + * arrives. * * Unlike the regular (non-streamed) case, handling streamed transactions has * to handle aborts of both the toplevel transaction and subtransactions. This @@ -174,11 +191,15 @@ #include "rewrite/rewriteHandler.h" #include "storage/buffile.h" #include "storage/bufmgr.h" +#include "storage/dsm.h" #include "storage/fd.h" #include "storage/ipc.h" #include "storage/lmgr.h" #include "storage/proc.h" #include "storage/procarray.h" +#include "storage/shm_mq.h" +#include "storage/shm_toc.h" +#include "storage/spin.h" #include "tcop/tcopprot.h" #include "utils/acl.h" #include "utils/builtins.h" @@ -198,6 +219,87 @@ #define NAPTIME_PER_CYCLE 1000 /* max sleep time between cycles (1s) */ +#define PG_LOGICAL_APPLY_SHM_MAGIC 0x79fb2447 // TODO Consider change + +/* + * DSM keys for apply background worker. Unlike other parallel execution code, + * since we don't need to worry about DSM keys conflicting with plan_node_id we + * can use small integers. + */ +#define APPLY_BGWORKER_KEY_SHARED 1 +#define APPLY_BGWORKER_KEY_MQ 2 + +/* + * Status for apply background worker. + */ +typedef enum ApplyBgworkerStatus +{ + APPLY_BGWORKER_ATTACHED = 0, + APPLY_BGWORKER_READY, + APPLY_BGWORKER_BUSY, + APPLY_BGWORKER_FINISHED, + APPLY_BGWORKER_EXIT +} ApplyBgworkerStatus; + +/* + * Shared information among apply workers. + */ +typedef struct ApplyBgworkerShared +{ + slock_t mutex; + + /* Status for apply background worker. */ + ApplyBgworkerStatus status; + + TransactionId stream_xid; + uint32 n; /* id of apply background worker */ +} ApplyBgworkerShared; + +/* + * Struct for maintaining an apply background worker. + */ +typedef struct ApplyBgworkerState +{ + shm_mq_handle *mq_handle; + dsm_segment *dsm_seg; + ApplyBgworkerShared volatile *pstate; +} ApplyBgworkerState; + +/* + * entry for a hash table we use to map from xid to our apply background worker + * state. + */ +typedef struct ApplyBgworkerEntry +{ + TransactionId xid; + ApplyBgworkerState *wstate; +} ApplyBgworkerEntry; + +/* Apply background workers hash table (initialized on first use) */ +static HTAB *ApplyWorkersHash = NULL; +static List *ApplyWorkersFreeList = NIL; +static List *ApplyWorkersList = NIL; + +/* Fields valid only for apply background workers */ +volatile ApplyBgworkerShared *MyParallelState = NULL; +static List *subxactlist = NIL; + +/* The number of changes during one streaming block */ +static uint32 nchanges = 0; + +/* Worker setup and interactions */ +static ApplyBgworkerState *apply_bgworker_setup(void); +static ApplyBgworkerState *apply_bgworker_find_or_start(TransactionId xid, + bool start); +static void apply_bgworker_setup_dsm(ApplyBgworkerState *wstate); +static void apply_bgworker_wait_for(ApplyBgworkerState *wstate, + ApplyBgworkerStatus wait_for_status); +static void apply_bgworker_send_data(ApplyBgworkerState *wstate, Size nbytes, + const void *data); +static void apply_bgworker_free(ApplyBgworkerState *wstate); +static void apply_bgworker_check_status(void); +static void apply_bgworker_set_status(ApplyBgworkerStatus status); + typedef struct FlushPosition { dlist_node node; @@ -260,18 +362,23 @@ static XLogRecPtr remote_final_lsn = InvalidXLogRecPtr; static bool in_streamed_transaction = false; static TransactionId stream_xid = InvalidTransactionId; +static ApplyBgworkerState *stream_apply_worker = NULL; + +/* check if we are applying the transaction in apply background worker */ +#define apply_bgworker_active() (in_streamed_transaction && stream_apply_worker != NULL) /* * We enable skipping all data modification changes (INSERT, UPDATE, etc.) for * the subscription if the remote transaction's finish LSN matches the subskiplsn. * Once we start skipping changes, we don't stop it until we skip all changes of * the transaction even if pg_subscription is updated and MySubscription->skiplsn - * gets changed or reset during that. Also, in streaming transaction cases, we - * don't skip receiving and spooling the changes since we decide whether or not + * gets changed or reset during that. Also, in streaming transaction cases (streaming = on), + * we don't skip receiving and spooling the changes since we decide whether or not * to skip applying the changes when starting to apply changes. The subskiplsn is * cleared after successfully skipping the transaction or applying non-empty * transaction. The latter prevents the mistakenly specified subskiplsn from - * being left. + * being left. Note that we cannot skip the streaming transaction in apply mode, + * because we cannot get the finish LSN before applying the changes. */ static XLogRecPtr skip_xact_finish_lsn = InvalidXLogRecPtr; #define is_skipping_changes() (unlikely(!XLogRecPtrIsInvalid(skip_xact_finish_lsn))) @@ -426,43 +533,104 @@ end_replication_step(void) } /* - * Handle streamed transactions. + * Handle streamed transactions for both main apply worker and apply background + * worker. + * + * For the main apply worker, if in streaming mode (receiving a block of + * streamed transaction), we send the data to the apply background worker. * - * If in streaming mode (receiving a block of streamed transaction), we - * simply redirect it to a file for the proper toplevel transaction. + * For the apply background worker, define a savepoint if a new subtransaction + * was started. * - * Returns true for streamed transactions, false otherwise (regular mode). + * For non-streamed transactions, returns false; + * For streamed transactions, returns true if in main apply worker, false + * otherwise. */ static bool handle_streamed_transaction(LogicalRepMsgType action, StringInfo s) { - TransactionId xid; + TransactionId current_xid = InvalidTransactionId; - /* not in streaming mode */ - if (!in_streamed_transaction) + /* + * Return if not in streaming mode (unless this is an apply background + * worker). + */ + if (!(in_streamed_transaction || am_apply_bgworker())) return false; - Assert(stream_fd != NULL); Assert(TransactionIdIsValid(stream_xid)); /* * We should have received XID of the subxact as the first part of the * message, so extract it. */ - xid = pq_getmsgint(s, 4); + current_xid = pq_getmsgint(s, 4); - if (!TransactionIdIsValid(xid)) + if (!TransactionIdIsValid(current_xid)) ereport(ERROR, (errcode(ERRCODE_PROTOCOL_VIOLATION), errmsg_internal("invalid transaction ID in streamed replication transaction"))); - /* Add the new subxact to the array (unless already there). */ - subxact_info_add(xid); + if (am_apply_bgworker()) + { + /* + * Inside apply background worker we can figure out that new + * subtransaction was started if new change arrived with different xid. + * In that case we can define named savepoint, so that we were able to + * commit/rollback it separately later. + * + * Special case is if the first change comes from subtransaction, then + * we check that current_xid differs from stream_xid. + */ + if (current_xid != stream_xid && + !list_member_int(subxactlist, (int) current_xid)) + { + MemoryContext oldctx; + char spname[MAXPGPATH]; + + snprintf(spname, MAXPGPATH, "savepoint_for_xid_%u", current_xid); + + elog(DEBUG1, "[Apply BGW #%u] defining savepoint %s", + MyParallelState->n, spname); + + DefineSavepoint(spname); + CommitTransactionCommand(); + + oldctx = MemoryContextSwitchTo(ApplyContext); + subxactlist = lappend_int(subxactlist, (int) current_xid); + MemoryContextSwitchTo(oldctx); + } + } + else if (apply_bgworker_active()) + { + /* + * If we decided to apply the changes of this transaction in an apply + * background worker, pass the data to the worker. + */ + apply_bgworker_send_data(stream_apply_worker, s->len, s->data); + nchanges += 1; + + /* + * XXX The publisher side doesn't always send relation update messages + * after the streaming transaction, so update the relation in main + * apply worker here. + */ + if (action == LOGICAL_REP_MSG_RELATION) + { + LogicalRepRelation *rel = logicalrep_read_rel(s); + logicalrep_relmap_update(rel); + } + } + else + { + /* Add the new subxact to the array (unless already there). */ + subxact_info_add(current_xid); - /* write the change to the current file */ - stream_write_change(action, s); + /* write the change to the current file */ + stream_write_change(action, s); + } - return true; + return !am_apply_bgworker(); } /* @@ -844,6 +1012,9 @@ apply_handle_commit(StringInfo s) apply_handle_commit_internal(&commit_data); + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); + /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(commit_data.end_lsn); @@ -898,7 +1069,9 @@ apply_handle_prepare_internal(LogicalRepPreparedTxnData *prepare_data) * BeginTransactionBlock is necessary to balance the EndTransactionBlock * called within the PrepareTransactionBlock below. */ - BeginTransactionBlock(); + if (!IsTransactionBlock()) + BeginTransactionBlock(); + CommitTransactionCommand(); /* Completes the preceding Begin command. */ /* @@ -950,6 +1123,9 @@ apply_handle_prepare(StringInfo s) in_remote_transaction = false; + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); + /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(prepare_data.end_lsn); @@ -1063,11 +1239,133 @@ apply_handle_rollback_prepared(StringInfo s) } /* - * Handle STREAM PREPARE. + * Look up worker inside ApplyWorkersHash for requested xid. * - * Logic is in two parts: - * 1. Replay all the spooled operations - * 2. Mark the transaction as prepared + * If start flag is true, try to start a new worker if not found in hash table. + */ +static ApplyBgworkerState * +apply_bgworker_find_or_start(TransactionId xid, bool start) +{ + bool found; + ApplyBgworkerState *wstate; + ApplyBgworkerEntry *entry = NULL; + + if (!TransactionIdIsValid(xid)) + return NULL; + + /* + * We don't start new background worker if we are not in streaming apply + * mode. + */ + if (MySubscription->stream != SUBSTREAM_APPLY) + return NULL; + + /* + * We don't start new background worker if user has set skiplsn as it's + * possible that user want to skip the streaming transaction. For streaming + * transaction, we need to spill the transaction to disk so that we can get + * the last LSN of the transaction to judge whether to skip before starting + * to apply the change. + */ + if (start && !XLogRecPtrIsInvalid(MySubscription->skiplsn)) + return NULL; + + /* + * For streaming transactions that are being applied in apply background + * worker, we cannot decide whether to apply the change for a relation that + * is not in the READY state (see should_apply_changes_for_rel) as we won't + * know remote_final_lsn by that time. So, we don't start new apply + * background worker in this case. + */ + if (start && !AllTablesyncsReady()) + return NULL; + + /* First time through, initialize apply workers hashtable */ + if (ApplyWorkersHash == NULL) + { + HASHCTL ctl; + + MemSet(&ctl, 0, sizeof(ctl)); + ctl.keysize = sizeof(TransactionId); + ctl.entrysize = sizeof(ApplyBgworkerEntry); + ctl.hcxt = ApplyContext; + + ApplyWorkersHash = hash_create("logical apply workers hash", 8, &ctl, + HASH_ELEM | HASH_BLOBS | HASH_CONTEXT); + } + + /* + * Find entry for requested transaction. + */ + entry = hash_search(ApplyWorkersHash, &xid, start ? HASH_ENTER : HASH_FIND, + &found); + if (found) + { + entry->wstate->pstate->status = APPLY_BGWORKER_BUSY; + return entry->wstate; + } + else if (!start) + return NULL; + + /* + * Now, we try to get a apply background worker. + * If there is at least one worker in the idle list, then take one. + * Otherwise, we try to start a new apply background worker. + */ + if (list_length(ApplyWorkersFreeList) > 0) + { + wstate = (ApplyBgworkerState *) llast(ApplyWorkersFreeList); + ApplyWorkersFreeList = list_delete_last(ApplyWorkersFreeList); + } + else + { + wstate = apply_bgworker_setup(); + + if (wstate == NULL) + { + /* + * If the apply background worker cannot be launched, remove entry + * in hash table. + */ + hash_search(ApplyWorkersHash, &xid, HASH_REMOVE, &found); + return NULL; + } + } + + /* Fill up the hash entry */ + wstate->pstate->status = APPLY_BGWORKER_BUSY; + wstate->pstate->stream_xid = xid; + entry->wstate = wstate; + entry->xid = xid; + + return wstate; +} + +/* + * Add the worker to the free list and remove the entry from hash table. + */ +static void +apply_bgworker_free(ApplyBgworkerState *wstate) +{ + MemoryContext oldctx; + TransactionId xid = wstate->pstate->stream_xid; + + Assert(wstate->pstate->status == APPLY_BGWORKER_FINISHED); + + oldctx = MemoryContextSwitchTo(ApplyContext); + + hash_search(ApplyWorkersHash, &xid, HASH_REMOVE, NULL); + + elog(DEBUG1, "adding finished apply worker #%u for xid %u to the idle list", + wstate->pstate->n, wstate->pstate->stream_xid); + + ApplyWorkersFreeList = lappend(ApplyWorkersFreeList, wstate); + + MemoryContextSwitchTo(oldctx); +} + +/* + * Handle STREAM PREPARE. */ static void apply_handle_stream_prepare(StringInfo s) @@ -1088,24 +1386,70 @@ apply_handle_stream_prepare(StringInfo s) logicalrep_read_stream_prepare(s, &prepare_data); set_apply_error_context_xact(prepare_data.xid, prepare_data.prepare_lsn); - elog(DEBUG1, "received prepare for streamed transaction %u", prepare_data.xid); + if (am_apply_bgworker()) + { + /* Mark the transaction as prepared. */ + apply_handle_prepare_internal(&prepare_data); - /* Replay all the spooled operations. */ - apply_spooled_messages(prepare_data.xid, prepare_data.prepare_lsn); + CommitTransactionCommand(); - /* Mark the transaction as prepared. */ - apply_handle_prepare_internal(&prepare_data); + pgstat_report_stat(false); - CommitTransactionCommand(); + list_free(subxactlist); + subxactlist = NIL; - pgstat_report_stat(false); + apply_bgworker_set_status(APPLY_BGWORKER_FINISHED); + } + else + { + ApplyBgworkerState *wstate = apply_bgworker_find_or_start(prepare_data.xid, false); - store_flush_position(prepare_data.end_lsn); + elog(DEBUG1, "received prepare for streamed transaction %u", + prepare_data.xid); + + /* + * This is the main apply worker. Check if we are processing this + * transaction in a apply background worker. + */ + if (wstate) + { + apply_bgworker_send_data(wstate, s->len, s->data); + + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_FINISHED); + apply_bgworker_free(wstate); + + pgstat_report_stat(false); + store_flush_position(prepare_data.end_lsn); + } + else + { + /* + * This is the main apply worker and the transaction has been + * serialized to file, replay all the spooled operations. + */ + apply_spooled_messages(prepare_data.xid, prepare_data.prepare_lsn); + + /* Mark the transaction as prepared. */ + apply_handle_prepare_internal(&prepare_data); + + CommitTransactionCommand(); + + pgstat_report_stat(false); + + store_flush_position(prepare_data.end_lsn); + + in_remote_transaction = false; + + /* unlink the files with serialized changes and subxact info. */ + stream_cleanup_files(MyLogicalRepWorker->subid, prepare_data.xid); + } + } in_remote_transaction = false; + stream_apply_worker = NULL; - /* unlink the files with serialized changes and subxact info. */ - stream_cleanup_files(MyLogicalRepWorker->subid, prepare_data.xid); + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(prepare_data.end_lsn); @@ -1155,15 +1499,6 @@ apply_handle_stream_start(StringInfo s) (errcode(ERRCODE_PROTOCOL_VIOLATION), errmsg_internal("duplicate STREAM START message"))); - /* - * Start a transaction on stream start, this transaction will be committed - * on the stream stop unless it is a tablesync worker in which case it - * will be committed after processing all the messages. We need the - * transaction for handling the buffile, used for serializing the - * streaming data and subxact info. - */ - begin_replication_step(); - /* notify handle methods we're processing a remote transaction */ in_streamed_transaction = true; @@ -1177,36 +1512,91 @@ apply_handle_stream_start(StringInfo s) set_apply_error_context_xact(stream_xid, InvalidXLogRecPtr); + if (am_apply_bgworker()) + { + /* + * Make sure the handle apply_dispatch methods are aware we're in a + * remote transaction. + */ + in_remote_transaction = true; + + /* If we are in a apply background worker, begin the transaction */ + AcceptInvalidationMessages(); + maybe_reread_subscription(); + + StartTransactionCommand(); + BeginTransactionBlock(); + CommitTransactionCommand(); + + return; + } + /* - * Initialize the worker's stream_fileset if we haven't yet. This will be - * used for the entire duration of the worker so create it in a permanent - * context. We create this on the very first streaming message from any - * transaction and then use it for this and other streaming transactions. - * Now, we could create a fileset at the start of the worker as well but - * then we won't be sure that it will ever be used. + * This is the main apply worker. Check if there is any free apply + * background worker we can use to process this transaction. */ - if (MyLogicalRepWorker->stream_fileset == NULL) + stream_apply_worker = apply_bgworker_find_or_start(stream_xid, first_segment); + + if (stream_apply_worker) { - MemoryContext oldctx; + /* + * If we have found a free worker or if we are already applying this + * transaction in an apply background worker, then we pass the data to + * that worker. + */ + if (first_segment) + apply_bgworker_send_data(stream_apply_worker, s->len, s->data); + + nchanges = 0; + elog(DEBUG1, "starting streaming of xid %u", stream_xid); + } + + /* + * If no worker is available for the first stream start, we start to + * serialize all the changes of the transaction. + */ + else + { + /* + * Start a transaction on stream start, this transaction will be committed + * on the stream stop unless it is a tablesync worker in which case it + * will be committed after processing all the messages. We need the + * transaction for handling the buffile, used for serializing the + * streaming data and subxact info. + */ + begin_replication_step(); + + /* + * Initialize the worker's stream_fileset if we haven't yet. This will be + * used for the entire duration of the worker so create it in a permanent + * context. We create this on the very first streaming message from any + * transaction and then use it for this and other streaming transactions. + * Now, we could create a fileset at the start of the worker as well but + * then we won't be sure that it will ever be used. + */ + if (MyLogicalRepWorker->stream_fileset == NULL) + { + MemoryContext oldctx; - oldctx = MemoryContextSwitchTo(ApplyContext); + oldctx = MemoryContextSwitchTo(ApplyContext); - MyLogicalRepWorker->stream_fileset = palloc(sizeof(FileSet)); - FileSetInit(MyLogicalRepWorker->stream_fileset); + MyLogicalRepWorker->stream_fileset = palloc(sizeof(FileSet)); + FileSetInit(MyLogicalRepWorker->stream_fileset); - MemoryContextSwitchTo(oldctx); - } + MemoryContextSwitchTo(oldctx); + } - /* open the spool file for this transaction */ - stream_open_file(MyLogicalRepWorker->subid, stream_xid, first_segment); + /* open the spool file for this transaction */ + stream_open_file(MyLogicalRepWorker->subid, stream_xid, first_segment); - /* if this is not the first segment, open existing subxact file */ - if (!first_segment) - subxact_info_read(MyLogicalRepWorker->subid, stream_xid); + /* if this is not the first segment, open existing subxact file */ + if (!first_segment) + subxact_info_read(MyLogicalRepWorker->subid, stream_xid); - pgstat_report_activity(STATE_RUNNING, NULL); + end_replication_step(); + } - end_replication_step(); + pgstat_report_activity(STATE_RUNNING, NULL); } /* @@ -1220,44 +1610,47 @@ apply_handle_stream_stop(StringInfo s) (errcode(ERRCODE_PROTOCOL_VIOLATION), errmsg_internal("STREAM STOP message without STREAM START"))); - /* - * Close the file with serialized changes, and serialize information about - * subxacts for the toplevel transaction. - */ - subxact_info_write(MyLogicalRepWorker->subid, stream_xid); - stream_close_file(); + if (apply_bgworker_active()) + { + char action = LOGICAL_REP_MSG_STREAM_STOP; - /* We must be in a valid transaction state */ - Assert(IsTransactionState()); + apply_bgworker_send_data(stream_apply_worker, 1, &action); + apply_bgworker_wait_for(stream_apply_worker, APPLY_BGWORKER_READY); - /* Commit the per-stream transaction */ - CommitTransactionCommand(); + elog(DEBUG1, "stopped streaming of xid %u, %u changes streamed", stream_xid, nchanges); + } + else + { + /* + * Close the file with serialized changes, and serialize information + * about subxacts for the toplevel transaction. + */ + subxact_info_write(MyLogicalRepWorker->subid, stream_xid); + stream_close_file(); - in_streamed_transaction = false; + /* We must be in a valid transaction state */ + Assert(IsTransactionState()); + + /* Commit the per-stream transaction */ + CommitTransactionCommand(); + + /* Reset per-stream context */ + MemoryContextReset(LogicalStreamingContext); + } - /* Reset per-stream context */ - MemoryContextReset(LogicalStreamingContext); + in_streamed_transaction = false; + stream_apply_worker = NULL; pgstat_report_activity(STATE_IDLE, NULL); reset_apply_error_context_info(); } /* - * Handle STREAM abort message. + * Handle STREAM ABORT message when the transaction was spilled to disk. */ static void -apply_handle_stream_abort(StringInfo s) +serialize_stream_abort(TransactionId xid, TransactionId subxid) { - TransactionId xid; - TransactionId subxid; - - if (in_streamed_transaction) - ereport(ERROR, - (errcode(ERRCODE_PROTOCOL_VIOLATION), - errmsg_internal("STREAM ABORT message without STREAM STOP"))); - - logicalrep_read_stream_abort(s, &xid, &subxid); - /* * If the two XIDs are the same, it's in fact abort of toplevel xact, so * just delete the files with serialized info. @@ -1339,43 +1732,152 @@ apply_handle_stream_abort(StringInfo s) end_replication_step(); CommitTransactionCommand(); } - - reset_apply_error_context_info(); } /* - * Common spoolfile processing. + * Handle STREAM ABORT message. */ static void -apply_spooled_messages(TransactionId xid, XLogRecPtr lsn) +apply_handle_stream_abort(StringInfo s) { - StringInfoData s2; - int nchanges; - char path[MAXPGPATH]; - char *buffer = NULL; - MemoryContext oldcxt; - BufFile *fd; + TransactionId xid; + TransactionId subxid; - maybe_start_skipping_changes(lsn); + if (in_streamed_transaction) + ereport(ERROR, + (errcode(ERRCODE_PROTOCOL_VIOLATION), + errmsg_internal("STREAM ABORT message without STREAM STOP"))); - /* Make sure we have an open transaction */ - begin_replication_step(); + logicalrep_read_stream_abort(s, &xid, &subxid); - /* - * Allocate file handle and memory required to process all the messages in - * TopTransactionContext to avoid them getting reset after each message is - * processed. - */ - oldcxt = MemoryContextSwitchTo(TopTransactionContext); + if (am_apply_bgworker()) + { + elog(DEBUG1, "[Apply BGW #%u] aborting current transaction xid=%u, subxid=%u", + MyParallelState->n, GetCurrentTransactionIdIfAny(), GetCurrentSubTransactionId()); - /* Open the spool file for the committed/prepared transaction */ - changes_filename(path, MyLogicalRepWorker->subid, xid); - elog(DEBUG1, "replaying changes from file \"%s\"", path); + /* + * If the two XIDs are the same, it's in fact abort of toplevel xact, + * so just free the subxactlist. + */ + if (subxid == xid) + { + set_apply_error_context_xact(subxid, InvalidXLogRecPtr); - fd = BufFileOpenFileSet(MyLogicalRepWorker->stream_fileset, path, O_RDONLY, - false); + AbortCurrentTransaction(); - buffer = palloc(BLCKSZ); + EndTransactionBlock(false); + CommitTransactionCommand(); + + in_remote_transaction = false; + + list_free(subxactlist); + subxactlist = NIL; + + apply_bgworker_set_status(APPLY_BGWORKER_FINISHED); + } + else + { + /* + * OK, so it's a subxact. Rollback to the savepoint. + * + * We also need to read the subxactlist, determine the offset + * tracked for the subxact, and truncate the list. + */ + int i; + bool found = false; + char spname[MAXPGPATH]; + + set_apply_error_context_xact(subxid, InvalidXLogRecPtr); + snprintf(spname, MAXPGPATH, "savepoint_for_xid_%u", subxid); + + elog(DEBUG1, "[Apply BGW #%u] rolling back to savepoint %s", + MyParallelState->n, spname); + + for (i = list_length(subxactlist) - 1; i >= 0; i--) + { + xid = (TransactionId) list_nth_int(subxactlist, i); + if (xid == subxid) + { + found = true; + break; + } + } + + if (found) + { + RollbackToSavepoint(spname); + CommitTransactionCommand(); + subxactlist = list_truncate(subxactlist, i + 1); + } + + apply_bgworker_set_status(APPLY_BGWORKER_READY); + } + + reset_apply_error_context_info(); + } + else + { + ApplyBgworkerState *wstate = apply_bgworker_find_or_start(xid, false); + + /* + * This is the main apply worker. Check if we are processing this + * transaction in a apply background worker. + */ + if (wstate) + { + apply_bgworker_send_data(wstate, s->len, s->data); + + if (subxid == xid) + { + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_FINISHED); + apply_bgworker_free(wstate); + } + else + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_READY); + } + + /* + * We are in main apply worker and the transaction has been serialized + * to file. + */ + else + serialize_stream_abort(xid, subxid); + } +} + +/* + * Common spoolfile processing. + */ +static void +apply_spooled_messages(TransactionId xid, XLogRecPtr lsn) +{ + StringInfoData s2; + int nchanges; + char path[MAXPGPATH]; + char *buffer = NULL; + MemoryContext oldcxt; + BufFile *fd; + + maybe_start_skipping_changes(lsn); + + /* Make sure we have an open transaction */ + begin_replication_step(); + + /* + * Allocate file handle and memory required to process all the messages in + * TopTransactionContext to avoid them getting reset after each message is + * processed. + */ + oldcxt = MemoryContextSwitchTo(TopTransactionContext); + + /* Open the spool file for the committed/prepared transaction */ + changes_filename(path, MyLogicalRepWorker->subid, xid); + elog(DEBUG1, "replaying changes from file \"%s\"", path); + + fd = BufFileOpenFileSet(MyLogicalRepWorker->stream_fileset, path, O_RDONLY, + false); + + buffer = palloc(BLCKSZ); initStringInfo(&s2); MemoryContextSwitchTo(oldcxt); @@ -1462,40 +1964,6 @@ apply_spooled_messages(TransactionId xid, XLogRecPtr lsn) return; } -/* - * Handle STREAM COMMIT message. - */ -static void -apply_handle_stream_commit(StringInfo s) -{ - TransactionId xid; - LogicalRepCommitData commit_data; - - if (in_streamed_transaction) - ereport(ERROR, - (errcode(ERRCODE_PROTOCOL_VIOLATION), - errmsg_internal("STREAM COMMIT message without STREAM STOP"))); - - xid = logicalrep_read_stream_commit(s, &commit_data); - set_apply_error_context_xact(xid, commit_data.commit_lsn); - - elog(DEBUG1, "received commit for streamed transaction %u", xid); - - apply_spooled_messages(xid, commit_data.commit_lsn); - - apply_handle_commit_internal(&commit_data); - - /* unlink the files with serialized changes and subxact info */ - stream_cleanup_files(MyLogicalRepWorker->subid, xid); - - /* Process any tables that are being synchronized in parallel. */ - process_syncing_tables(commit_data.end_lsn); - - pgstat_report_activity(STATE_IDLE, NULL); - - reset_apply_error_context_info(); -} - /* * Helper function for apply_handle_commit and apply_handle_stream_commit. */ @@ -2445,6 +2913,100 @@ apply_handle_truncate(StringInfo s) end_replication_step(); } +/* + * Handle STREAM COMMIT message. + */ +static void +apply_handle_stream_commit(StringInfo s) +{ + LogicalRepCommitData commit_data; + TransactionId xid; + + if (in_streamed_transaction) + ereport(ERROR, + (errcode(ERRCODE_PROTOCOL_VIOLATION), + errmsg_internal("STREAM COMMIT message without STREAM STOP"))); + + xid = logicalrep_read_stream_commit(s, &commit_data); + set_apply_error_context_xact(xid, commit_data.commit_lsn); + + if (am_apply_bgworker()) + { + /* + * Update origin state so we can restart streaming from correct + * position in case of crash. + */ + replorigin_session_origin_lsn = commit_data.end_lsn; + replorigin_session_origin_timestamp = commit_data.committime; + + CommitTransactionCommand(); + EndTransactionBlock(false); + CommitTransactionCommand(); + + in_remote_transaction = false; + + pgstat_report_stat(false); + + list_free(subxactlist); + subxactlist = NIL; + + apply_bgworker_set_status(APPLY_BGWORKER_FINISHED); + } + else + { + /* + * This is the main apply worker. Check if we are processing this + * transaction in an apply background worker. + */ + ApplyBgworkerState *wstate = apply_bgworker_find_or_start(xid, false); + + elog(DEBUG1, "received commit for streamed transaction %u", xid); + + if (wstate) + { + /* Send commit message */ + apply_bgworker_send_data(wstate, s->len, s->data); + + /* Wait for apply background worker to finish */ + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_FINISHED); + + pgstat_report_stat(false); + store_flush_position(commit_data.end_lsn); + stop_skipping_changes(); + + apply_bgworker_free(wstate); + + /* + * The transaction is either non-empty or skipped, so we clear the + * subskiplsn. + */ + clear_subscription_skip_lsn(commit_data.commit_lsn); + } + else + { + /* + * This is the main apply worker and the transaction has been + * serialized to file, replay all the spooled operations. + */ + apply_spooled_messages(xid, commit_data.commit_lsn); + + apply_handle_commit_internal(&commit_data); + + /* unlink the files with serialized changes and subxact info */ + stream_cleanup_files(MyLogicalRepWorker->subid, xid); + } + } + + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); + + /* Process any tables that are being synchronized in parallel. */ + process_syncing_tables(commit_data.end_lsn); + + pgstat_report_activity(STATE_IDLE, NULL); + + reset_apply_error_context_info(); +} /* * Logical replication protocol message dispatcher. @@ -2618,6 +3180,10 @@ store_flush_position(XLogRecPtr remote_lsn) { FlushPosition *flushpos; + /* We only need to collect the LSN in main apply worker */ + if (am_apply_bgworker()) + return; + /* Need to do this in permanent context */ MemoryContextSwitchTo(ApplyContext); @@ -2794,6 +3360,9 @@ LogicalRepApplyLoop(XLogRecPtr last_received) AcceptInvalidationMessages(); maybe_reread_subscription(); + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); + /* Process any table synchronization changes. */ process_syncing_tables(last_received); } @@ -3691,7 +4260,7 @@ ApplyWorkerMain(Datum main_arg) originid = replorigin_by_name(originname, true); if (!OidIsValid(originid)) originid = replorigin_create(originname); - replorigin_session_setup(originid); + replorigin_session_setup(originid, true); replorigin_session_origin = originid; origin_startpos = replorigin_session_get_progress(false); CommitTransactionCommand(); @@ -3738,7 +4307,7 @@ ApplyWorkerMain(Datum main_arg) options.proto.logical.publication_names = MySubscription->publications; options.proto.logical.binary = MySubscription->binary; - options.proto.logical.streaming = MySubscription->stream; + options.proto.logical.streaming = (MySubscription->stream != SUBSTREAM_OFF); options.proto.logical.twophase = false; if (!am_tablesync_worker()) @@ -3896,7 +4465,8 @@ clear_subscription_skip_lsn(XLogRecPtr finish_lsn) XLogRecPtr myskiplsn = MySubscription->skiplsn; bool started_tx = false; - if (likely(XLogRecPtrIsInvalid(myskiplsn))) + if (likely(XLogRecPtrIsInvalid(myskiplsn)) || + am_apply_bgworker()) return; if (!IsTransactionState()) @@ -4032,3 +4602,533 @@ reset_apply_error_context_info(void) apply_error_callback_arg.remote_attnum = -1; set_apply_error_context_xact(InvalidTransactionId, InvalidXLogRecPtr); } + +/* Apply Background Worker main loop */ +static void +LogicalApplyBgwLoop(shm_mq_handle *mqh, volatile ApplyBgworkerShared *pst) +{ + shm_mq_result shmq_res; + PGPROC *registrant; + ErrorContextCallback errcallback; + XLogRecPtr last_received = InvalidXLogRecPtr; + + registrant = BackendPidGetProc(MyBgworkerEntry->bgw_notify_pid); + SetLatch(®istrant->procLatch); + + /* + * Push apply error context callback. Fields will be filled during + * applying a change. + */ + errcallback.callback = apply_error_callback; + errcallback.previous = error_context_stack; + error_context_stack = &errcallback; + + for (;;) + { + void *data; + Size len; + int c; + StringInfoData s; + MemoryContext oldctx; + XLogRecPtr start_lsn; + XLogRecPtr end_lsn; + TimestampTz send_time; + + CHECK_FOR_INTERRUPTS(); + + /* Ensure we are reading the data into our memory context. */ + oldctx = MemoryContextSwitchTo(ApplyMessageContext); + + shmq_res = shm_mq_receive(mqh, &len, &data, false); + + if (shmq_res != SHM_MQ_SUCCESS) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("lost connection to the main apply worker"))); + + if (len == 0) + break; + + s.cursor = 0; + s.maxlen = -1; + s.data = (char *) data; + s.len = len; + + /* + * We use first byte of message for additional communication between + * main Logical replication worker and apply bgworkers, so if it + * differs from 'w', then process it first. + */ + c = pq_getmsgbyte(&s); + switch (c) + { + /* End message of streaming chunk */ + case LOGICAL_REP_MSG_STREAM_STOP: + elog(DEBUG1, "[Apply BGW #%u] ended processing streaming chunk," + "waiting on shm_mq_receive", pst->n); + + apply_bgworker_set_status(APPLY_BGWORKER_READY); + + SetLatch(®istrant->procLatch); + + in_streamed_transaction = false; + pgstat_report_activity(STATE_IDLE, NULL); + continue; + case 'w': + break; + default: + elog(ERROR, "[Apply BGW #%u] unexpected message \"%c\"", + pst->n, c); + break; + } + + start_lsn = pq_getmsgint64(&s); + end_lsn = pq_getmsgint64(&s); + send_time = pq_getmsgint64(&s); + + if (last_received < start_lsn) + last_received = start_lsn; + + if (last_received < end_lsn) + last_received = end_lsn; + + /* + * TO IMPROVE: Do we need to display the apply background worker's + * information in pg_stat_replication ? + */ + UpdateWorkerStats(last_received, send_time, false); + + apply_dispatch(&s); + + if (ConfigReloadPending) + { + ConfigReloadPending = false; + ProcessConfigFile(PGC_SIGHUP); + } + + MemoryContextSwitchTo(oldctx); + MemoryContextReset(ApplyMessageContext); + } + + MemoryContextSwitchTo(TopMemoryContext); + MemoryContextReset(ApplyContext); + + /* Pop the error context stack */ + error_context_stack = errcallback.previous; + + elog(DEBUG1, "[Apply BGW #%u] exiting", pst->n); + + /* Signal main process that we are done. */ + SetLatch(®istrant->procLatch); +} + +/* + * Set the exit status so that the main apply worker can realize we have + * shutdown. + */ +static void +ApplyBgwShutdown(int code, Datum arg) +{ + SpinLockAcquire(&MyParallelState->mutex); + MyParallelState->status = APPLY_BGWORKER_EXIT; + SpinLockRelease(&MyParallelState->mutex); + + dsm_detach((dsm_segment *) DatumGetPointer(arg)); +} + +/* + * Apply Background Worker entry point + */ +void +ApplyBgworkerMain(Datum main_arg) +{ + volatile ApplyBgworkerShared *pst; + + dsm_handle handle; + dsm_segment *seg; + shm_toc *toc; + shm_mq *mq; + shm_mq_handle *mqh; + MemoryContext oldcontext; + RepOriginId originid; + int worker_slot = DatumGetInt32(main_arg); + char originname[NAMEDATALEN]; + + /* Attach to slot */ + logicalrep_worker_attach(worker_slot); + + MemoryContextSwitchTo(TopMemoryContext); + + /* Load the subscription into persistent memory context. */ + ApplyContext = AllocSetContextCreate(TopMemoryContext, + "ApplyContext", + ALLOCSET_DEFAULT_SIZES); + + /* + * Init the ApplyMessageContext which we clean up after each replication + * protocol message. + */ + ApplyMessageContext = AllocSetContextCreate(ApplyContext, + "ApplyMessageContext", + ALLOCSET_DEFAULT_SIZES); + + /* Setup signal handling */ + pqsignal(SIGHUP, SignalHandlerForConfigReload); + pqsignal(SIGTERM, die); + BackgroundWorkerUnblockSignals(); + + /* + * Connect to the dynamic shared memory segment. + * + * The backend that registered this worker passed us the ID of a shared + * memory segment to which we must attach for further instructions. In + * order to attach to dynamic shared memory, we need a resource owner. + * Once we've mapped the segment in our address space, attach to the table + * of contents so we can locate the various data structures we'll need to + * find within the segment. + */ + CurrentResourceOwner = ResourceOwnerCreate(NULL, "Logical apply worker"); + memcpy(&handle, MyBgworkerEntry->bgw_extra, sizeof(dsm_handle)); + seg = dsm_attach(handle); + if (seg == NULL) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("unable to map dynamic shared memory segment"))); + toc = shm_toc_attach(PG_LOGICAL_APPLY_SHM_MAGIC, dsm_segment_address(seg)); + if (toc == NULL) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("bad magic number in dynamic shared memory segment"))); + + before_shmem_exit(ApplyBgwShutdown, PointerGetDatum(seg)); + + /* Look up the parallel state. */ + pst = shm_toc_lookup(toc, APPLY_BGWORKER_KEY_SHARED, false); + MyParallelState = pst; + + /* + * Attach to the message queue. + */ + mq = shm_toc_lookup(toc, APPLY_BGWORKER_KEY_MQ, false); + shm_mq_set_receiver(mq, MyProc); + mqh = shm_mq_attach(mq, seg, NULL); + + /* Connect to our database. */ + BackgroundWorkerInitializeConnectionByOid(MyLogicalRepWorker->dbid, + MyLogicalRepWorker->userid, + 0); + + /* + * Set the client encoding to the database encoding, since that is what + * the leader will expect. + */ + SetClientEncoding(GetDatabaseEncoding()); + + stream_xid = pst->stream_xid; + + StartTransactionCommand(); + oldcontext = MemoryContextSwitchTo(ApplyContext); + + MySubscription = GetSubscription(MyLogicalRepWorker->subid, true); + if (!MySubscription) + { + ereport(LOG, + (errmsg("logical replication apply worker for subscription %u will not " + "start because the subscription was removed during startup", + MyLogicalRepWorker->subid))); + proc_exit(0); + } + + MySubscriptionValid = true; + MemoryContextSwitchTo(oldcontext); + + /* Setup synchronous commit according to the user's wishes */ + SetConfigOption("synchronous_commit", MySubscription->synccommit, + PGC_BACKEND, PGC_S_OVERRIDE); + + /* Keep us informed about subscription changes. */ + CacheRegisterSyscacheCallback(SUBSCRIPTIONOID, + subscription_change_cb, + (Datum) 0); + + CommitTransactionCommand(); + + /* Setup replication origin tracking. */ + StartTransactionCommand(); + snprintf(originname, sizeof(originname), "pg_%u", MySubscription->oid); + originid = replorigin_by_name(originname, true); + if (!OidIsValid(originid)) + originid = replorigin_create(originname); + + /* + * The apply background worker doesn't need to monopolize this replication + * origin which was already acquired by its leader process. + */ + replorigin_session_setup(originid, false); + replorigin_session_origin = originid; + CommitTransactionCommand(); + + /* + * Allocate the origin name in long-lived context for error context + * message. + */ + apply_error_callback_arg.origin_name = MemoryContextStrdup(ApplyContext, + originname); + + /* + * Indicate that we're fully initialized and ready to begin the main part + * of the apply operation. + */ + apply_bgworker_set_status(APPLY_BGWORKER_ATTACHED); + + elog(DEBUG1, "[Apply BGW #%u] started", pst->n); + + PG_TRY(); + { + LogicalApplyBgwLoop(mqh, pst); + } + PG_CATCH(); + { + /* + * Report the worker failed while applying streaming transaction + * changes. Abort the current transaction so that the stats message is + * sent in an idle state. + */ + AbortOutOfAnyTransaction(); + pgstat_report_subscription_error(MySubscription->oid, false); + + PG_RE_THROW(); + } + PG_END_TRY(); + + /* + * We're done. Explicitly detach the shared memory segment so that we + * don't get a resource leak warning at commit time. This will fire any + * on_dsm_detach callbacks we've registered, as well. Once that's done, + * we can go ahead and exit. + */ + dsm_detach(seg); + proc_exit(0); +} + +/* + * Set up a dynamic shared memory segment. + * + * We set up a control region that contains a ApplyBgworkerShared, + * plus one region per message queue. There are as many message queues as + * the number of workers. + */ +static void +apply_bgworker_setup_dsm(ApplyBgworkerState *wstate) +{ + shm_toc_estimator e; + Size segsize; + dsm_segment *seg; + shm_toc *toc; + ApplyBgworkerShared *pst; + shm_mq *mq; + int64 queue_size = 160000000; /* 16 MB for now */ + + /* + * Estimate how much shared memory we need. + * + * Because the TOC machinery may choose to insert padding of oddly-sized + * requests, we must estimate each chunk separately. + * + * We need one key to register the location of the header, and we need + * another key to track the location of the message queue. + */ + shm_toc_initialize_estimator(&e); + shm_toc_estimate_chunk(&e, sizeof(ApplyBgworkerShared)); + shm_toc_estimate_chunk(&e, (Size) queue_size); + + shm_toc_estimate_keys(&e, 1 + 1); + segsize = shm_toc_estimate(&e); + + /* Create the shared memory segment and establish a table of contents. */ + seg = dsm_create(shm_toc_estimate(&e), 0); + toc = shm_toc_create(PG_LOGICAL_APPLY_SHM_MAGIC, dsm_segment_address(seg), + segsize); + + /* Set up the header region. */ + pst = shm_toc_allocate(toc, sizeof(ApplyBgworkerShared)); + SpinLockInit(&pst->mutex); + pst->status = APPLY_BGWORKER_BUSY; + pst->stream_xid = stream_xid; + pst->n = list_length(ApplyWorkersList) + 1; + + shm_toc_insert(toc, APPLY_BGWORKER_KEY_SHARED, pst); + + /* Set up one message queue per worker, plus one. */ + mq = shm_mq_create(shm_toc_allocate(toc, (Size) queue_size), + (Size) queue_size); + shm_toc_insert(toc, APPLY_BGWORKER_KEY_MQ, mq); + shm_mq_set_sender(mq, MyProc); + + /* Attach the queues. */ + wstate->mq_handle = shm_mq_attach(mq, seg, NULL); + + /* Return results to caller. */ + wstate->dsm_seg = seg; + wstate->pstate = pst; +} + +/* + * Start apply background worker process and allocate shared memory for it. + */ +static ApplyBgworkerState * +apply_bgworker_setup(void) +{ + MemoryContext oldcontext; + bool launched; + ApplyBgworkerState *wstate; + + elog(DEBUG1, "setting up apply worker #%u", list_length(ApplyWorkersList) + 1); + + oldcontext = MemoryContextSwitchTo(ApplyContext); + + wstate = (ApplyBgworkerState *) palloc0(sizeof(ApplyBgworkerState)); + + /* Setup shared memory */ + apply_bgworker_setup_dsm(wstate); + + launched = logicalrep_worker_launch(MyLogicalRepWorker->dbid, + MySubscription->oid, + MySubscription->name, + MyLogicalRepWorker->userid, + InvalidOid, + dsm_segment_handle(wstate->dsm_seg)); + + if (launched) + { + /* Wait for worker to attach. */ + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_ATTACHED); + + ApplyWorkersList = lappend(ApplyWorkersList, wstate); + } + else + { + pfree(wstate); + wstate = NULL; + } + + MemoryContextSwitchTo(oldcontext); + + return wstate; +} + +/* + * Send the data to the specified apply background worker via shared-memory queue. + */ +static void +apply_bgworker_send_data(ApplyBgworkerState *wstate, Size nbytes, const void *data) +{ + shm_mq_result result; + + result = shm_mq_send(wstate->mq_handle, nbytes, data, false, true); + + if (result != SHM_MQ_SUCCESS) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("could not send tuples to shared-memory queue"))); +} + +/* + * Wait until the status of apply background worker reaches the + * 'wait_for_status' + */ +static void +apply_bgworker_wait_for(ApplyBgworkerState *wstate, + ApplyBgworkerStatus wait_for_status) +{ + for (;;) + { + char status; + + SpinLockAcquire(&wstate->pstate->mutex); + status = wstate->pstate->status; + SpinLockRelease(&wstate->pstate->mutex); + + /* Done if already in correct status. */ + if (status == wait_for_status) + break; + + /* If any workers (or the postmaster) have died, we have failed. */ + if (status == APPLY_BGWORKER_EXIT) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("background worker %u failed to apply transaction %u", + wstate->pstate->n, wstate->pstate->stream_xid))); + + /* Wait to be signalled. */ + WaitLatch(MyLatch, WL_LATCH_SET | WL_EXIT_ON_PM_DEATH, 0, + WAIT_EVENT_LOGICAL_APPLY_WORKER_STATE_CHANGE); + + /* Reset the latch so we don't spin. */ + ResetLatch(MyLatch); + + /* An interrupt may have occurred while we were waiting. */ + CHECK_FOR_INTERRUPTS(); + } +} + +/* + * Check the status of workers and report an error if any apply background + * worker has exited unexpectedly. + * + * Exit if any relation is not in the READY state and if any worker is handling + * the streaming transaction at the same time. Because for streaming + * transactions that is being applied in apply background worker, we cannot + * decide whether to apply the change for a relation that is not in the READY + * state (see should_apply_changes_for_rel) as we won't know remote_final_lsn + * by that time. + */ +static void +apply_bgworker_check_status(void) +{ + ListCell *lc; + + if (am_apply_bgworker() || MySubscription->stream != SUBSTREAM_APPLY) + return; + + foreach(lc, ApplyWorkersList) + { + ApplyBgworkerState *wstate = (ApplyBgworkerState *) lfirst(lc); + + /* + * We don't lock here as in the worst case we will just detect the + * failure of worker a bit later. + */ + if (wstate->pstate->status == APPLY_BGWORKER_EXIT) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("background worker %u exited unexpectedly", + wstate->pstate->n))); + } + + if (list_length(ApplyWorkersFreeList) != list_length(ApplyWorkersList) && + !AllTablesyncsReady()) + { + ereport(LOG, + (errmsg("logical replication apply workers for subscription \"%s\" will restart", + MySubscription->name), + errdetail("Cannot handle streamed replication transaction by apply " + "bgworkers until all tables are synchronized"))); + + proc_exit(0); + } +} + +/* Set the apply background worker status */ +static void +apply_bgworker_set_status(ApplyBgworkerStatus status) +{ + if (!am_apply_bgworker()) + return; + + elog(DEBUG1, "[Apply BGW #%u] set status to %d", MyParallelState->n, status); + + SpinLockAcquire(&MyParallelState->mutex); + MyParallelState->status = status; + SpinLockRelease(&MyParallelState->mutex); +} diff --git a/src/backend/utils/activity/wait_event.c b/src/backend/utils/activity/wait_event.c index 87c15b9c6f..ba781e6f08 100644 --- a/src/backend/utils/activity/wait_event.c +++ b/src/backend/utils/activity/wait_event.c @@ -388,6 +388,9 @@ pgstat_get_wait_ipc(WaitEventIPC w) case WAIT_EVENT_HASH_GROW_BUCKETS_REINSERT: event_name = "HashGrowBucketsReinsert"; break; + case WAIT_EVENT_LOGICAL_APPLY_WORKER_STATE_CHANGE: + event_name = "LogicalApplyWorkerStateChange"; + break; case WAIT_EVENT_LOGICAL_SYNC_DATA: event_name = "LogicalSyncData"; break; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 7cc9c72e49..6f8b30abb0 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -4450,7 +4450,7 @@ getSubscriptions(Archive *fout) if (fout->remoteVersion >= 140000) appendPQExpBufferStr(query, " s.substream,\n"); else - appendPQExpBufferStr(query, " false AS substream,\n"); + appendPQExpBufferStr(query, " 'f' AS substream,\n"); if (fout->remoteVersion >= 150000) appendPQExpBufferStr(query, @@ -4580,8 +4580,10 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo) if (strcmp(subinfo->subbinary, "t") == 0) appendPQExpBufferStr(query, ", binary = true"); - if (strcmp(subinfo->substream, "f") != 0) + if (strcmp(subinfo->substream, "t") == 0) appendPQExpBufferStr(query, ", streaming = on"); + else if (strcmp(subinfo->substream, "a") == 0) + appendPQExpBufferStr(query, ", streaming = apply"); if (strcmp(subinfo->subtwophasestate, two_phase_disabled) != 0) appendPQExpBufferStr(query, ", two_phase = on"); diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h index d1260f590c..9b394a45fe 100644 --- a/src/include/catalog/pg_subscription.h +++ b/src/include/catalog/pg_subscription.h @@ -68,7 +68,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW bool subbinary; /* True if the subscription wants the * publisher to send data in binary */ - bool substream; /* Stream in-progress transactions. */ + char substream; /* Stream in-progress transactions. + * See SUBSTREAM_xxx constants. */ char subtwophasestate; /* Stream two-phase transactions */ @@ -109,7 +110,7 @@ typedef struct Subscription bool enabled; /* Indicates if the subscription is enabled */ bool binary; /* Indicates if the subscription wants data in * binary format */ - bool stream; /* Allow streaming in-progress transactions. */ + char stream; /* Allow streaming in-progress transactions. */ char twophasestate; /* Allow streaming two-phase transactions */ bool disableonerr; /* Indicates if the subscription should be * automatically disabled if a worker error @@ -120,6 +121,18 @@ typedef struct Subscription List *publications; /* List of publication names to subscribe to */ } Subscription; +/* Disallow streaming in-progress transactions */ +#define SUBSTREAM_OFF 'f' + +/* + * Streaming transactions are written to a temporary file and applied only + * after the transaction is committed on upstream. + */ +#define SUBSTREAM_ON 't' + +/* Streaming transactions are applied immediately via a background worker */ +#define SUBSTREAM_APPLY 'a' + extern Subscription *GetSubscription(Oid subid, bool missing_ok); extern void FreeSubscription(Subscription *sub); extern void DisableSubscription(Oid subid); diff --git a/src/include/replication/logicalworker.h b/src/include/replication/logicalworker.h index cd1b6e8afc..6a1af7f13c 100644 --- a/src/include/replication/logicalworker.h +++ b/src/include/replication/logicalworker.h @@ -13,6 +13,7 @@ #define LOGICALWORKER_H extern void ApplyWorkerMain(Datum main_arg); +extern void ApplyBgworkerMain(Datum main_arg); extern bool IsLogicalWorker(void); diff --git a/src/include/replication/origin.h b/src/include/replication/origin.h index 2d1b5e5c28..c7389b40a7 100644 --- a/src/include/replication/origin.h +++ b/src/include/replication/origin.h @@ -53,7 +53,7 @@ extern XLogRecPtr replorigin_get_progress(RepOriginId node, bool flush); extern void replorigin_session_advance(XLogRecPtr remote_commit, XLogRecPtr local_commit); -extern void replorigin_session_setup(RepOriginId node); +extern void replorigin_session_setup(RepOriginId node, bool must_acquire); extern void replorigin_session_reset(void); extern XLogRecPtr replorigin_session_get_progress(bool flush); diff --git a/src/include/replication/worker_internal.h b/src/include/replication/worker_internal.h index 901845abc2..1feafccaf6 100644 --- a/src/include/replication/worker_internal.h +++ b/src/include/replication/worker_internal.h @@ -60,6 +60,9 @@ typedef struct LogicalRepWorker */ FileSet *stream_fileset; + /* Indicates if this slot is used for an apply background worker. */ + bool subworker; + /* Stats. */ XLogRecPtr last_lsn; TimestampTz last_send_time; @@ -84,8 +87,9 @@ extern void logicalrep_worker_attach(int slot); extern LogicalRepWorker *logicalrep_worker_find(Oid subid, Oid relid, bool only_running); extern List *logicalrep_workers_find(Oid subid, bool only_running); -extern void logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, - Oid userid, Oid relid); +extern bool logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, + Oid userid, Oid relid, + dsm_handle subworker_dsm); extern void logicalrep_worker_stop(Oid subid, Oid relid); extern void logicalrep_worker_wakeup(Oid subid, Oid relid); extern void logicalrep_worker_wakeup_ptr(LogicalRepWorker *worker); @@ -109,4 +113,10 @@ am_tablesync_worker(void) return OidIsValid(MyLogicalRepWorker->relid); } +static inline bool +am_apply_bgworker(void) +{ + return MyLogicalRepWorker->subworker; +} + #endif /* WORKER_INTERNAL_H */ diff --git a/src/include/utils/wait_event.h b/src/include/utils/wait_event.h index b578e2ec75..c2d2a114d7 100644 --- a/src/include/utils/wait_event.h +++ b/src/include/utils/wait_event.h @@ -105,6 +105,7 @@ typedef enum WAIT_EVENT_HASH_GROW_BUCKETS_ALLOCATE, WAIT_EVENT_HASH_GROW_BUCKETS_ELECT, WAIT_EVENT_HASH_GROW_BUCKETS_REINSERT, + WAIT_EVENT_LOGICAL_APPLY_WORKER_STATE_CHANGE, WAIT_EVENT_LOGICAL_SYNC_DATA, WAIT_EVENT_LOGICAL_SYNC_STATE_CHANGE, WAIT_EVENT_MQ_INTERNAL, diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out index 7fcfad1591..f769835af0 100644 --- a/src/test/regress/expected/subscription.out +++ b/src/test/regress/expected/subscription.out @@ -197,7 +197,7 @@ ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); DROP SUBSCRIPTION regress_testsub; -- fail - streaming must be boolean CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = foo); -ERROR: streaming requires a Boolean value +ERROR: streaming requires a Boolean value or "apply" -- now it works CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = true); WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 4fb746930a..12940d6f4f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -116,6 +116,10 @@ Append AppendPath AppendRelInfo AppendState +ApplyBgworkerEntry +ApplyBgworkerShared +ApplyBgworkerState +ApplyBgworkerStatus ApplyErrorCallbackArg ApplyExecutionData ApplySubXactData -- 2.23.0.windows.1 [application/octet-stream] v7-0002-Test-streaming-apply-option-in-tap-test.patch (64.7K, ../../OS3PR01MB6275797F66EF0A47EEB2D8FC9EDD9@OS3PR01MB6275.jpnprd01.prod.outlook.com/3-v7-0002-Test-streaming-apply-option-in-tap-test.patch) download | inline diff: From 4049ef69d3aac74b16b719787f1733aec1f193f8 Mon Sep 17 00:00:00 2001 From: "shiy.fnst" <[email protected]> Date: Fri, 13 May 2022 14:50:30 +0800 Subject: [PATCH v7] Test streaming apply option in tap test Change all TAP tests using the SUBSCRIPTION "streaming" option, so they now test both 'on' and 'apply' values. --- src/test/subscription/t/015_stream.pl | 176 ++++--- src/test/subscription/t/016_stream_subxact.pl | 98 ++-- src/test/subscription/t/017_stream_ddl.pl | 169 ++++--- .../t/018_stream_subxact_abort.pl | 176 ++++--- .../t/019_stream_subxact_ddl_abort.pl | 89 +++- .../subscription/t/022_twophase_cascade.pl | 330 ++++++------ .../subscription/t/023_twophase_stream.pl | 472 ++++++++++-------- 7 files changed, 877 insertions(+), 633 deletions(-) diff --git a/src/test/subscription/t/015_stream.pl b/src/test/subscription/t/015_stream.pl index 6561b189de..c8b117c780 100644 --- a/src/test/subscription/t/015_stream.pl +++ b/src/test/subscription/t/015_stream.pl @@ -8,6 +8,96 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" option +# so the same code can be run both for the streaming=on and streaming=apply +# cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname) = @_; + + # Interleave a pair of transactions, each exceeding the 64kB limit. + my $in = ''; + my $out = ''; + + my $timer = IPC::Run::timeout($PostgreSQL::Test::Utils::timeout_default); + + my $h = $node_publisher->background_psql('postgres', \$in, \$out, $timer, + on_error_stop => 0); + + $in .= q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + }; + $h->pump_nb; + + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001, 9999) s(i); + DELETE FROM test_tab WHERE a > 5000; + COMMIT; + }); + + $in .= q{ + COMMIT; + \q + }; + $h->finish; # errors make the next test fail, so ignore them here + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), 'check extra columns contain local defaults'); + + # Test the streaming in binary mode + $node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET (binary = on)"); + + # Insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001, 10000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + COMMIT; + }); + + $node_publisher->wait_for_catchup($appname); + + $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(6667|6667|6667), 'check extra columns contain local defaults'); + + # Change the local values of the extra columns on the subscriber, + # update publisher, and check that subscriber retains the expected + # values. This is to ensure that non-streaming transactions behave + # properly after a streaming transaction. + $node_subscriber->safe_psql('postgres', + "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'" + ); + $node_publisher->safe_psql('postgres', + "UPDATE test_tab SET b = md5(a::text)"); + + $node_publisher->wait_for_catchup($appname); + + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab" + ); + is($result, qq(6667|6667|6667), + 'check extra columns contain locally changed data'); + + # Cleanup the test data + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE (a > 2)"); + $node_publisher->wait_for_catchup($appname); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -37,6 +127,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -54,82 +148,24 @@ my $result = "SELECT count(*), count(c), count(d = 999) FROM test_tab"); is($result, qq(2|2|2), 'check initial data was copied to subscriber'); -# Interleave a pair of transactions, each exceeding the 64kB limit. -my $in = ''; -my $out = ''; - -my $timer = IPC::Run::timeout($PostgreSQL::Test::Utils::timeout_default); - -my $h = $node_publisher->background_psql('postgres', \$in, \$out, $timer, - on_error_stop => 0); - -$in .= q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -}; -$h->pump_nb; - -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001, 9999) s(i); -DELETE FROM test_tab WHERE a > 5000; -COMMIT; -}); - -$in .= q{ -COMMIT; -\q -}; -$h->finish; # errors make the next test fail, so ignore them here - -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), 'check extra columns contain local defaults'); - -# Test the streaming in binary mode -$node_subscriber->safe_psql('postgres', - "ALTER SUBSCRIPTION tap_sub SET (binary = on)"); - -# Insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001, 10000) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -COMMIT; -}); +test_streaming($node_publisher, $node_subscriber, $appname); -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(6667|6667|6667), 'check extra columns contain local defaults'); +################################### +# Test using streaming mode 'apply' +################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" +); -# Change the local values of the extra columns on the subscriber, -# update publisher, and check that subscriber retains the expected -# values. This is to ensure that non-streaming transactions behave -# properly after a streaming transaction. $node_subscriber->safe_psql('postgres', - "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'" + "ALTER SUBSCRIPTION tap_sub SET(streaming = apply, binary = off)" ); -$node_publisher->safe_psql('postgres', - "UPDATE test_tab SET b = md5(a::text)"); -$node_publisher->wait_for_catchup($appname); +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" +) or die "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab" -); -is($result, qq(6667|6667|6667), - 'check extra columns contain locally changed data'); +test_streaming($node_publisher, $node_subscriber, $appname); $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/016_stream_subxact.pl b/src/test/subscription/t/016_stream_subxact.pl index f27f1694f2..64a4d54cc2 100644 --- a/src/test/subscription/t/016_stream_subxact.pl +++ b/src/test/subscription/t/016_stream_subxact.pl @@ -8,6 +8,54 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" option +# so the same code can be run both for the streaming=on and streaming=apply +# cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname) = @_; + + # Insert, update and delete enough rows to exceed 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series( 3, 500) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501, 1000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001, 1500) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + SAVEPOINT s3; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501, 2000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + SAVEPOINT s4; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001, 2500) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + COMMIT; + }); + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(1667|1667|1667), + 'check data was copied to subscriber in streaming mode and extra columns contain local defaults' + ); + + # Cleanup the test data + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE (a > 2)"); + $node_publisher->wait_for_catchup($appname); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -37,6 +85,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -54,41 +106,25 @@ my $result = "SELECT count(*), count(c), count(d = 999) FROM test_tab"); is($result, qq(2|2|2), 'check initial data was copied to subscriber'); -# Insert, update and delete enough rows to exceed 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series( 3, 500) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501, 1000) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001, 1500) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -SAVEPOINT s3; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501, 2000) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -SAVEPOINT s4; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001, 2500) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -COMMIT; -}); +test_streaming($node_publisher, $node_subscriber, $appname); -$node_publisher->wait_for_catchup($appname); +################################### +# Test using streaming mode 'apply' +################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" +); -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(1667|1667|1667), - 'check data was copied to subscriber in streaming mode and extra columns contain local defaults' +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = apply)" ); +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" +) or die "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; + +test_streaming($node_publisher, $node_subscriber, $appname); + $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/017_stream_ddl.pl b/src/test/subscription/t/017_stream_ddl.pl index 0bce63b716..50b71dfa48 100644 --- a/src/test/subscription/t/017_stream_ddl.pl +++ b/src/test/subscription/t/017_stream_ddl.pl @@ -8,6 +8,93 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" option +# so the same code can be run both for the streaming=on and streaming=apply +# cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname) = @_; + + # a small (non-streamed) transaction with DDL and DML + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab VALUES (3, md5(3::text)); + ALTER TABLE test_tab ADD COLUMN c INT; + SAVEPOINT s1; + INSERT INTO test_tab VALUES (4, md5(4::text), -4); + COMMIT; + }); + + # large (streamed) transaction with DDL and DML + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(5, 1000) s(i); + ALTER TABLE test_tab ADD COLUMN d INT; + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001, 2000) s(i); + COMMIT; + }); + + # a small (non-streamed) transaction with DDL and DML + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab VALUES (2001, md5(2001::text), -2001, 2*2001); + ALTER TABLE test_tab ADD COLUMN e INT; + SAVEPOINT s1; + INSERT INTO test_tab VALUES (2002, md5(2002::text), -2002, 2*2002, -3*2002); + COMMIT; + }); + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d), count(e) FROM test_tab"); + is($result, qq(2002|1999|1002|1), + 'check data was copied to subscriber in streaming mode and extra columns contain local defaults' + ); + + # A large (streamed) transaction with DDL and DML. One of the DDL is performed + # after DML to ensure that we invalidate the schema sent for test_tab so that + # the next transaction has to send the schema again. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(2003,5000) s(i); + ALTER TABLE test_tab ADD COLUMN f INT; + COMMIT; + }); + + # A small transaction that won't get streamed. This is just to ensure that we + # send the schema again to reflect the last column added in the previous test. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i, 4*i FROM generate_series(5001,5005) s(i); + COMMIT; + }); + + $node_publisher->wait_for_catchup($appname); + + $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d), count(e), count(f) FROM test_tab"); + is($result, qq(5005|5002|4005|3004|5), + 'check data was copied to subscriber for both streaming and non-streaming transactions' + ); + + # Cleanup the test data + $node_publisher->safe_psql( + 'postgres', q{ + DELETE FROM test_tab WHERE (a > 2); + ALTER TABLE test_tab DROP COLUMN c, DROP COLUMN d, DROP COLUMN e, DROP COLUMN f; + }); + $node_publisher->wait_for_catchup($appname); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -37,6 +124,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -54,76 +145,24 @@ my $result = "SELECT count(*), count(c), count(d = 999) FROM test_tab"); is($result, qq(2|0|0), 'check initial data was copied to subscriber'); -# a small (non-streamed) transaction with DDL and DML -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab VALUES (3, md5(3::text)); -ALTER TABLE test_tab ADD COLUMN c INT; -SAVEPOINT s1; -INSERT INTO test_tab VALUES (4, md5(4::text), -4); -COMMIT; -}); - -# large (streamed) transaction with DDL and DML -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(5, 1000) s(i); -ALTER TABLE test_tab ADD COLUMN d INT; -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001, 2000) s(i); -COMMIT; -}); - -# a small (non-streamed) transaction with DDL and DML -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab VALUES (2001, md5(2001::text), -2001, 2*2001); -ALTER TABLE test_tab ADD COLUMN e INT; -SAVEPOINT s1; -INSERT INTO test_tab VALUES (2002, md5(2002::text), -2002, 2*2002, -3*2002); -COMMIT; -}); +test_streaming($node_publisher, $node_subscriber, $appname); -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d), count(e) FROM test_tab"); -is($result, qq(2002|1999|1002|1), - 'check data was copied to subscriber in streaming mode and extra columns contain local defaults' +################################### +# Test using streaming mode 'apply' +################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" ); -# A large (streamed) transaction with DDL and DML. One of the DDL is performed -# after DML to ensure that we invalidate the schema sent for test_tab so that -# the next transaction has to send the schema again. -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(2003,5000) s(i); -ALTER TABLE test_tab ADD COLUMN f INT; -COMMIT; -}); - -# A small transaction that won't get streamed. This is just to ensure that we -# send the schema again to reflect the last column added in the previous test. -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i, 4*i FROM generate_series(5001,5005) s(i); -COMMIT; -}); +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = apply)" +); -$node_publisher->wait_for_catchup($appname); +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" +) or die "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d), count(e), count(f) FROM test_tab"); -is($result, qq(5005|5002|4005|3004|5), - 'check data was copied to subscriber for both streaming and non-streaming transactions' -); +test_streaming($node_publisher, $node_subscriber, $appname); $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/018_stream_subxact_abort.pl b/src/test/subscription/t/018_stream_subxact_abort.pl index 7155442e76..1c3b8a3e98 100644 --- a/src/test/subscription/t/018_stream_subxact_abort.pl +++ b/src/test/subscription/t/018_stream_subxact_abort.pl @@ -8,6 +8,95 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" option +# so the same code can be run both for the streaming=on and streaming=apply +# cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname) = @_; + + # large (streamed) transaction with DDL, DML and ROLLBACKs + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i); + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501,1000) s(i); + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001,1500) s(i); + SAVEPOINT s3; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501,2000) s(i); + ROLLBACK TO s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001,2500) s(i); + ROLLBACK TO s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2501,3000) s(i); + SAVEPOINT s4; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3001,3500) s(i); + SAVEPOINT s5; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3501,4000) s(i); + COMMIT; + }); + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c) FROM test_tab"); + is($result, qq(2000|0), + 'check rollback to savepoint was reflected on subscriber and extra columns contain local defaults' + ); + + # large (streamed) transaction with subscriber receiving out of order + # subtransaction ROLLBACKs + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(4001,4500) s(i); + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001,5500) s(i); + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(6001,6500) s(i); + SAVEPOINT s3; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(7001,7500) s(i); + RELEASE s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(8001,8500) s(i); + ROLLBACK TO s1; + COMMIT; + }); + + $node_publisher->wait_for_catchup($appname); + + $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c) FROM test_tab"); + is($result, qq(2500|0), + 'check rollback to savepoint was reflected on subscriber'); + + # large (streamed) transaction with subscriber receiving rollback + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(8501,9000) s(i); + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(9001,9500) s(i); + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(9501,10000) s(i); + ROLLBACK; + }); + + $node_publisher->wait_for_catchup($appname); + + $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c) FROM test_tab"); + is($result, qq(2500|0), 'check rollback was reflected on subscriber'); + + # Cleanup the test data + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE (a > 2)"); + $node_publisher->wait_for_catchup($appname); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -36,6 +125,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -53,81 +146,24 @@ my $result = "SELECT count(*), count(c) FROM test_tab"); is($result, qq(2|0), 'check initial data was copied to subscriber'); -# large (streamed) transaction with DDL, DML and ROLLBACKs -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i); -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501,1000) s(i); -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001,1500) s(i); -SAVEPOINT s3; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501,2000) s(i); -ROLLBACK TO s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001,2500) s(i); -ROLLBACK TO s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2501,3000) s(i); -SAVEPOINT s4; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3001,3500) s(i); -SAVEPOINT s5; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3501,4000) s(i); -COMMIT; -}); - -$node_publisher->wait_for_catchup($appname); +test_streaming($node_publisher, $node_subscriber, $appname); -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c) FROM test_tab"); -is($result, qq(2000|0), - 'check rollback to savepoint was reflected on subscriber and extra columns contain local defaults' +################################### +# Test using streaming mode 'apply' +################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" ); -# large (streamed) transaction with subscriber receiving out of order -# subtransaction ROLLBACKs -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(4001,4500) s(i); -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001,5500) s(i); -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(6001,6500) s(i); -SAVEPOINT s3; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(7001,7500) s(i); -RELEASE s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(8001,8500) s(i); -ROLLBACK TO s1; -COMMIT; -}); - -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c) FROM test_tab"); -is($result, qq(2500|0), - 'check rollback to savepoint was reflected on subscriber'); - -# large (streamed) transaction with subscriber receiving rollback -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(8501,9000) s(i); -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(9001,9500) s(i); -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(9501,10000) s(i); -ROLLBACK; -}); +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = apply)" +); -$node_publisher->wait_for_catchup($appname); +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" +) or die "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c) FROM test_tab"); -is($result, qq(2500|0), 'check rollback was reflected on subscriber'); +test_streaming($node_publisher, $node_subscriber, $appname); $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/019_stream_subxact_ddl_abort.pl b/src/test/subscription/t/019_stream_subxact_ddl_abort.pl index dbd0fca4d1..c532f862da 100644 --- a/src/test/subscription/t/019_stream_subxact_ddl_abort.pl +++ b/src/test/subscription/t/019_stream_subxact_ddl_abort.pl @@ -9,6 +9,51 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" option +# so the same code can be run both for the streaming=on and streaming=apply +# cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname) = @_; + + # large (streamed) transaction with DDL, DML and ROLLBACKs + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i); + ALTER TABLE test_tab ADD COLUMN c INT; + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(501,1000) s(i); + ALTER TABLE test_tab ADD COLUMN d INT; + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001,1500) s(i); + ALTER TABLE test_tab ADD COLUMN e INT; + SAVEPOINT s3; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(1501,2000) s(i); + ALTER TABLE test_tab DROP COLUMN c; + ROLLBACK TO s1; + INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(501,1000) s(i); + COMMIT; + }); + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c) FROM test_tab"); + is($result, qq(1000|500), + 'check rollback to savepoint was reflected on subscriber and extra columns contain local defaults' + ); + + # Cleanup the test data + $node_publisher->safe_psql( + 'postgres', q{ + DELETE FROM test_tab WHERE (a > 2); + ALTER TABLE test_tab DROP COLUMN c; + }); + $node_publisher->wait_for_catchup($appname); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -37,6 +82,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -54,35 +103,25 @@ my $result = "SELECT count(*), count(c) FROM test_tab"); is($result, qq(2|0), 'check initial data was copied to subscriber'); -# large (streamed) transaction with DDL, DML and ROLLBACKs -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i); -ALTER TABLE test_tab ADD COLUMN c INT; -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(501,1000) s(i); -ALTER TABLE test_tab ADD COLUMN d INT; -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001,1500) s(i); -ALTER TABLE test_tab ADD COLUMN e INT; -SAVEPOINT s3; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(1501,2000) s(i); -ALTER TABLE test_tab DROP COLUMN c; -ROLLBACK TO s1; -INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(501,1000) s(i); -COMMIT; -}); +test_streaming($node_publisher, $node_subscriber, $appname); -$node_publisher->wait_for_catchup($appname); +################################### +# Test using streaming mode 'apply' +################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" +); -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c) FROM test_tab"); -is($result, qq(1000|500), - 'check rollback to savepoint was reflected on subscriber and extra columns contain local defaults' +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = apply)" ); +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" +) or die "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; + +test_streaming($node_publisher, $node_subscriber, $appname); + $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/022_twophase_cascade.pl b/src/test/subscription/t/022_twophase_cascade.pl index 7a797f37ba..663808acb9 100644 --- a/src/test/subscription/t/022_twophase_cascade.pl +++ b/src/test/subscription/t/022_twophase_cascade.pl @@ -11,6 +11,175 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" option +# so the same code can be run both for the streaming=on and streaming=apply +# cases. +sub test_streaming +{ + my ($node_A, $node_B, $node_C, $appname_B, $appname_C, $streaming_mode) = @_; + + my $oldpid_B = $node_A->safe_psql( + 'postgres', " + SELECT pid FROM pg_stat_replication + WHERE application_name = '$appname_B' AND state = 'streaming';"); + my $oldpid_C = $node_B->safe_psql( + 'postgres', " + SELECT pid FROM pg_stat_replication + WHERE application_name = '$appname_C' AND state = 'streaming';"); + + # Setup logical replication streaming mode + + $node_B->safe_psql( + 'postgres', " + ALTER SUBSCRIPTION tap_sub_B + SET (streaming = $streaming_mode);"); + $node_C->safe_psql( + 'postgres', " + ALTER SUBSCRIPTION tap_sub_C + SET (streaming = $streaming_mode)"); + + # Wait for subscribers to finish initialization + + $node_A->poll_query_until( + 'postgres', " + SELECT pid != $oldpid_B FROM pg_stat_replication + WHERE application_name = '$appname_B' AND state = 'streaming';" + ) or die "Timed out while waiting for apply to restart"; + $node_B->poll_query_until( + 'postgres', " + SELECT pid != $oldpid_C FROM pg_stat_replication + WHERE application_name = '$appname_C' AND state = 'streaming';" + ) or die "Timed out while waiting for apply to restart"; + + ############################### + # Test 2PC PREPARE / COMMIT PREPARED. + # 1. Data is streamed as a 2PC transaction. + # 2. Then do commit prepared. + # + # Expect all data is replicated on subscriber(s) after the commit. + ############################### + + # Insert, update and delete enough rows to exceed the 64kB limit. + # Then 2PC PREPARE + $node_A->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); + + # check the transaction state is prepared on subscriber(s) + my $result = + $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber B'); + $result = + $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber C'); + + # 2PC COMMIT + $node_A->safe_psql('postgres', "COMMIT PREPARED 'test_prepared_tab';"); + + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); + + # check that transaction was committed on subscriber(s) + $result = $node_B->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), + 'Rows inserted by 2PC have committed on subscriber B, and extra columns have local defaults' + ); + $result = $node_C->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), + 'Rows inserted by 2PC have committed on subscriber C, and extra columns have local defaults' + ); + + # check the transaction state is ended on subscriber(s) + $result = + $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is committed on subscriber B'); + $result = + $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is committed on subscriber C'); + + ############################### + # Test 2PC PREPARE with a nested ROLLBACK TO SAVEPOINT. + # 0. Cleanup from previous test leaving only 2 rows. + # 1. Insert one more row. + # 2. Record a SAVEPOINT. + # 3. Data is streamed using 2PC. + # 4. Do rollback to SAVEPOINT prior to the streamed inserts. + # 5. Then COMMIT PREPARED. + # + # Expect data after the SAVEPOINT is aborted leaving only 3 rows (= 2 original + 1 from step 1). + ############################### + + # First, delete the data except for 2 rows (delete will be replicated) + $node_A->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); + + # 2PC PREPARE with a nested ROLLBACK TO SAVEPOINT + $node_A->safe_psql( + 'postgres', " + BEGIN; + INSERT INTO test_tab VALUES (9999, 'foobar'); + SAVEPOINT sp_inner; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + ROLLBACK TO SAVEPOINT sp_inner; + PREPARE TRANSACTION 'outer'; + "); + + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); + + # check the transaction state prepared on subscriber(s) + $result = + $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber B'); + $result = + $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber C'); + + # 2PC COMMIT + $node_A->safe_psql('postgres', "COMMIT PREPARED 'outer';"); + + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); + + # check the transaction state is ended on subscriber + $result = + $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is ended on subscriber B'); + $result = + $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is ended on subscriber C'); + + # check inserts are visible at subscriber(s). + # All the streamed data (prior to the SAVEPOINT) should be rolled back. + # (9999, 'foobar') should be committed. + $result = $node_B->safe_psql('postgres', + "SELECT count(*) FROM test_tab where b = 'foobar';"); + is($result, qq(1), 'Rows committed are present on subscriber B'); + $result = $node_B->safe_psql('postgres', "SELECT count(*) FROM test_tab;"); + is($result, qq(3), 'Rows committed are present on subscriber B'); + $result = $node_C->safe_psql('postgres', + "SELECT count(*) FROM test_tab where b = 'foobar';"); + is($result, qq(1), 'Rows committed are present on subscriber C'); + $result = $node_C->safe_psql('postgres', "SELECT count(*) FROM test_tab;"); + is($result, qq(3), 'Rows committed are present on subscriber C'); + + # Cleanup the test data + $node_A->safe_psql('postgres', + "DELETE FROM test_tab WHERE a > 2;"); + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); +} + ############################### # Setup a cascade of pub/sub nodes. # node_A -> node_B -> node_C @@ -260,160 +429,15 @@ is($result, qq(21), 'Rows committed are present on subscriber C'); # 2PC + STREAMING TESTS # --------------------- -my $oldpid_B = $node_A->safe_psql( - 'postgres', " - SELECT pid FROM pg_stat_replication - WHERE application_name = '$appname_B' AND state = 'streaming';"); -my $oldpid_C = $node_B->safe_psql( - 'postgres', " - SELECT pid FROM pg_stat_replication - WHERE application_name = '$appname_C' AND state = 'streaming';"); - -# Setup logical replication (streaming = on) - -$node_B->safe_psql( - 'postgres', " - ALTER SUBSCRIPTION tap_sub_B - SET (streaming = on);"); -$node_C->safe_psql( - 'postgres', " - ALTER SUBSCRIPTION tap_sub_C - SET (streaming = on)"); - -# Wait for subscribers to finish initialization - -$node_A->poll_query_until( - 'postgres', " - SELECT pid != $oldpid_B FROM pg_stat_replication - WHERE application_name = '$appname_B' AND state = 'streaming';" -) or die "Timed out while waiting for apply to restart"; -$node_B->poll_query_until( - 'postgres', " - SELECT pid != $oldpid_C FROM pg_stat_replication - WHERE application_name = '$appname_C' AND state = 'streaming';" -) or die "Timed out while waiting for apply to restart"; - -############################### -# Test 2PC PREPARE / COMMIT PREPARED. -# 1. Data is streamed as a 2PC transaction. -# 2. Then do commit prepared. -# -# Expect all data is replicated on subscriber(s) after the commit. -############################### - -# Insert, update and delete enough rows to exceed the 64kB limit. -# Then 2PC PREPARE -$node_A->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_A->wait_for_catchup($appname_B); -$node_B->wait_for_catchup($appname_C); +################################ +# Test using streaming mode 'on' +################################ +test_streaming($node_A, $node_B, $node_C, $appname_B, $appname_C, 'on'); -# check the transaction state is prepared on subscriber(s) -$result = - $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber B'); -$result = - $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber C'); - -# 2PC COMMIT -$node_A->safe_psql('postgres', "COMMIT PREPARED 'test_prepared_tab';"); - -$node_A->wait_for_catchup($appname_B); -$node_B->wait_for_catchup($appname_C); - -# check that transaction was committed on subscriber(s) -$result = $node_B->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), - 'Rows inserted by 2PC have committed on subscriber B, and extra columns have local defaults' -); -$result = $node_C->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), - 'Rows inserted by 2PC have committed on subscriber C, and extra columns have local defaults' -); - -# check the transaction state is ended on subscriber(s) -$result = - $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is committed on subscriber B'); -$result = - $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is committed on subscriber C'); - -############################### -# Test 2PC PREPARE with a nested ROLLBACK TO SAVEPOINT. -# 0. Cleanup from previous test leaving only 2 rows. -# 1. Insert one more row. -# 2. Record a SAVEPOINT. -# 3. Data is streamed using 2PC. -# 4. Do rollback to SAVEPOINT prior to the streamed inserts. -# 5. Then COMMIT PREPARED. -# -# Expect data after the SAVEPOINT is aborted leaving only 3 rows (= 2 original + 1 from step 1). -############################### - -# First, delete the data except for 2 rows (delete will be replicated) -$node_A->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); - -# 2PC PREPARE with a nested ROLLBACK TO SAVEPOINT -$node_A->safe_psql( - 'postgres', " - BEGIN; - INSERT INTO test_tab VALUES (9999, 'foobar'); - SAVEPOINT sp_inner; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - ROLLBACK TO SAVEPOINT sp_inner; - PREPARE TRANSACTION 'outer'; - "); - -$node_A->wait_for_catchup($appname_B); -$node_B->wait_for_catchup($appname_C); - -# check the transaction state prepared on subscriber(s) -$result = - $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber B'); -$result = - $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber C'); - -# 2PC COMMIT -$node_A->safe_psql('postgres', "COMMIT PREPARED 'outer';"); - -$node_A->wait_for_catchup($appname_B); -$node_B->wait_for_catchup($appname_C); - -# check the transaction state is ended on subscriber -$result = - $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is ended on subscriber B'); -$result = - $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is ended on subscriber C'); - -# check inserts are visible at subscriber(s). -# All the streamed data (prior to the SAVEPOINT) should be rolled back. -# (9999, 'foobar') should be committed. -$result = $node_B->safe_psql('postgres', - "SELECT count(*) FROM test_tab where b = 'foobar';"); -is($result, qq(1), 'Rows committed are present on subscriber B'); -$result = $node_B->safe_psql('postgres', "SELECT count(*) FROM test_tab;"); -is($result, qq(3), 'Rows committed are present on subscriber B'); -$result = $node_C->safe_psql('postgres', - "SELECT count(*) FROM test_tab where b = 'foobar';"); -is($result, qq(1), 'Rows committed are present on subscriber C'); -$result = $node_C->safe_psql('postgres', "SELECT count(*) FROM test_tab;"); -is($result, qq(3), 'Rows committed are present on subscriber C'); +################################### +# Test using streaming mode 'apply' +################################### +test_streaming($node_A, $node_B, $node_C, $appname_B, $appname_C, 'apply'); ############################### # check all the cleanup diff --git a/src/test/subscription/t/023_twophase_stream.pl b/src/test/subscription/t/023_twophase_stream.pl index d8475d25a4..30200d5937 100644 --- a/src/test/subscription/t/023_twophase_stream.pl +++ b/src/test/subscription/t/023_twophase_stream.pl @@ -8,6 +8,243 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" option +# so the same code can be run both for the streaming=on and streaming=apply +# cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname) = @_; + + ############################### + # Test 2PC PREPARE / COMMIT PREPARED. + # 1. Data is streamed as a 2PC transaction. + # 2. Then do commit prepared. + # + # Expect all data is replicated on subscriber side after the commit. + ############################### + + # check that 2PC gets replicated to subscriber + # Insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is in prepared state on subscriber + my $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber'); + + # 2PC transaction gets committed + $node_publisher->safe_psql('postgres', + "COMMIT PREPARED 'test_prepared_tab';"); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is committed on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), + 'Rows inserted by 2PC have committed on subscriber, and extra columns contain local defaults' + ); + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is committed on subscriber'); + + ############################### + # Test 2PC PREPARE / ROLLBACK PREPARED. + # 1. Table is deleted back to 2 rows which are replicated on subscriber. + # 2. Data is streamed using 2PC. + # 3. Do rollback prepared. + # + # Expect data rolls back leaving only the original 2 rows. + ############################### + + # First, delete the data except for 2 rows (will be replicated) + $node_publisher->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); + + # Then insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is in prepared state on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber'); + + # 2PC transaction gets aborted + $node_publisher->safe_psql('postgres', + "ROLLBACK PREPARED 'test_prepared_tab';"); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is aborted on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(2|2|2), + 'Rows inserted by 2PC are rolled back, leaving only the original 2 rows'); + + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is aborted on subscriber'); + + ############################### + # Check that 2PC COMMIT PREPARED is decoded properly on crash restart. + # 1. insert, update and delete enough rows to exceed the 64kB limit. + # 2. Then server crashes before the 2PC transaction is committed. + # 3. After servers are restarted the pending transaction is committed. + # + # Expect all data is replicated on subscriber side after the commit. + # Note: both publisher and subscriber do crash/restart. + ############################### + + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + $node_subscriber->stop('immediate'); + $node_publisher->stop('immediate'); + + $node_publisher->start; + $node_subscriber->start; + + # commit post the restart + $node_publisher->safe_psql('postgres', + "COMMIT PREPARED 'test_prepared_tab';"); + $node_publisher->wait_for_catchup($appname); + + # check inserts are visible + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), + 'Rows inserted by 2PC have committed on subscriber, and extra columns contain local defaults' + ); + + ############################### + # Do INSERT after the PREPARE but before ROLLBACK PREPARED. + # 1. Table is deleted back to 2 rows which are replicated on subscriber. + # 2. Data is streamed using 2PC. + # 3. A single row INSERT is done which is after the PREPARE. + # 4. Then do a ROLLBACK PREPARED. + # + # Expect the 2PC data rolls back leaving only 3 rows on the subscriber + # (the original 2 + inserted 1). + ############################### + + # First, delete the data except for 2 rows (will be replicated) + $node_publisher->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); + + # Then insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is in prepared state on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber'); + + # Insert a different record (now we are outside of the 2PC transaction) + # Note: the 2PC transaction still holds row locks so make sure this insert is for a separate primary key + $node_publisher->safe_psql('postgres', + "INSERT INTO test_tab VALUES (99999, 'foobar')"); + + # 2PC transaction gets aborted + $node_publisher->safe_psql('postgres', + "ROLLBACK PREPARED 'test_prepared_tab';"); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is aborted on subscriber, + # but the extra INSERT outside of the 2PC still was replicated + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3|3|3), 'check the outside insert was copied to subscriber'); + + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is aborted on subscriber'); + + ############################### + # Do INSERT after the PREPARE but before COMMIT PREPARED. + # 1. Table is deleted back to 2 rows which are replicated on subscriber. + # 2. Data is streamed using 2PC. + # 3. A single row INSERT is done which is after the PREPARE. + # 4. Then do a COMMIT PREPARED. + # + # Expect 2PC data + the extra row are on the subscriber + # (the 3334 + inserted 1 = 3335). + ############################### + + # First, delete the data except for 2 rows (will be replicated) + $node_publisher->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); + + # Then insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is in prepared state on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber'); + + # Insert a different record (now we are outside of the 2PC transaction) + # Note: the 2PC transaction still holds row locks so make sure this insert is for a separare primary key + $node_publisher->safe_psql('postgres', + "INSERT INTO test_tab VALUES (99999, 'foobar')"); + + # 2PC transaction gets committed + $node_publisher->safe_psql('postgres', + "COMMIT PREPARED 'test_prepared_tab';"); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is committed on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3335|3335|3335), + 'Rows inserted by 2PC (as well as outside insert) have committed on subscriber, and extra columns contain local defaults' + ); + + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is committed on subscriber'); + + # Cleanup the test data + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE a > 2;"); + $node_publisher->wait_for_catchup($appname); +} + ############################### # Setup ############################### @@ -48,6 +285,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql( 'postgres', " CREATE SUBSCRIPTION tap_sub @@ -70,236 +311,29 @@ my $twophase_query = $node_subscriber->poll_query_until('postgres', $twophase_query) or die "Timed out while waiting for subscriber to enable twophase"; -############################### # Check initial data was copied to subscriber -############################### my $result = $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab"); is($result, qq(2|2|2), 'check initial data was copied to subscriber'); -############################### -# Test 2PC PREPARE / COMMIT PREPARED. -# 1. Data is streamed as a 2PC transaction. -# 2. Then do commit prepared. -# -# Expect all data is replicated on subscriber side after the commit. -############################### - -# check that 2PC gets replicated to subscriber -# Insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is in prepared state on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber'); +test_streaming($node_publisher, $node_subscriber, $appname); -# 2PC transaction gets committed -$node_publisher->safe_psql('postgres', - "COMMIT PREPARED 'test_prepared_tab';"); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is committed on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), - 'Rows inserted by 2PC have committed on subscriber, and extra columns contain local defaults' +################################### +# Test using streaming mode 'apply' +################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" ); -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is committed on subscriber'); - -############################### -# Test 2PC PREPARE / ROLLBACK PREPARED. -# 1. Table is deleted back to 2 rows which are replicated on subscriber. -# 2. Data is streamed using 2PC. -# 3. Do rollback prepared. -# -# Expect data rolls back leaving only the original 2 rows. -############################### - -# First, delete the data except for 2 rows (will be replicated) -$node_publisher->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); - -# Then insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); -$node_publisher->wait_for_catchup($appname); - -# check that transaction is in prepared state on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber'); - -# 2PC transaction gets aborted -$node_publisher->safe_psql('postgres', - "ROLLBACK PREPARED 'test_prepared_tab';"); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is aborted on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(2|2|2), - 'Rows inserted by 2PC are rolled back, leaving only the original 2 rows'); - -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is aborted on subscriber'); - -############################### -# Check that 2PC COMMIT PREPARED is decoded properly on crash restart. -# 1. insert, update and delete enough rows to exceed the 64kB limit. -# 2. Then server crashes before the 2PC transaction is committed. -# 3. After servers are restarted the pending transaction is committed. -# -# Expect all data is replicated on subscriber side after the commit. -# Note: both publisher and subscriber do crash/restart. -############################### - -$node_publisher->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_subscriber->stop('immediate'); -$node_publisher->stop('immediate'); - -$node_publisher->start; -$node_subscriber->start; - -# commit post the restart -$node_publisher->safe_psql('postgres', - "COMMIT PREPARED 'test_prepared_tab';"); -$node_publisher->wait_for_catchup($appname); - -# check inserts are visible -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), - 'Rows inserted by 2PC have committed on subscriber, and extra columns contain local defaults' +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = apply)" ); -############################### -# Do INSERT after the PREPARE but before ROLLBACK PREPARED. -# 1. Table is deleted back to 2 rows which are replicated on subscriber. -# 2. Data is streamed using 2PC. -# 3. A single row INSERT is done which is after the PREPARE. -# 4. Then do a ROLLBACK PREPARED. -# -# Expect the 2PC data rolls back leaving only 3 rows on the subscriber -# (the original 2 + inserted 1). -############################### - -# First, delete the data except for 2 rows (will be replicated) -$node_publisher->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); - -# Then insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is in prepared state on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber'); - -# Insert a different record (now we are outside of the 2PC transaction) -# Note: the 2PC transaction still holds row locks so make sure this insert is for a separate primary key -$node_publisher->safe_psql('postgres', - "INSERT INTO test_tab VALUES (99999, 'foobar')"); - -# 2PC transaction gets aborted -$node_publisher->safe_psql('postgres', - "ROLLBACK PREPARED 'test_prepared_tab';"); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is aborted on subscriber, -# but the extra INSERT outside of the 2PC still was replicated -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3|3|3), 'check the outside insert was copied to subscriber'); - -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is aborted on subscriber'); - -############################### -# Do INSERT after the PREPARE but before COMMIT PREPARED. -# 1. Table is deleted back to 2 rows which are replicated on subscriber. -# 2. Data is streamed using 2PC. -# 3. A single row INSERT is done which is after the PREPARE. -# 4. Then do a COMMIT PREPARED. -# -# Expect 2PC data + the extra row are on the subscriber -# (the 3334 + inserted 1 = 3335). -############################### - -# First, delete the data except for 2 rows (will be replicated) -$node_publisher->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); - -# Then insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is in prepared state on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber'); - -# Insert a different record (now we are outside of the 2PC transaction) -# Note: the 2PC transaction still holds row locks so make sure this insert is for a separare primary key -$node_publisher->safe_psql('postgres', - "INSERT INTO test_tab VALUES (99999, 'foobar')"); - -# 2PC transaction gets committed -$node_publisher->safe_psql('postgres', - "COMMIT PREPARED 'test_prepared_tab';"); - -$node_publisher->wait_for_catchup($appname); +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" +) or die "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; -# check that transaction is committed on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3335|3335|3335), - 'Rows inserted by 2PC (as well as outside insert) have committed on subscriber, and extra columns contain local defaults' -); - -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is committed on subscriber'); +test_streaming($node_publisher, $node_subscriber, $appname); ############################### # check all the cleanup -- 2.24.0.windows.2 [application/octet-stream] v7-0003-Add-some-checks-before-using-apply-background-wor.patch (17.8K, ../../OS3PR01MB6275797F66EF0A47EEB2D8FC9EDD9@OS3PR01MB6275.jpnprd01.prod.outlook.com/4-v7-0003-Add-some-checks-before-using-apply-background-wor.patch) download | inline diff: From d8cc81f517c4ee7cad2695abb5ae3304344005a2 Mon Sep 17 00:00:00 2001 From: wangw <[email protected]> Date: Mon, 30 May 2022 13:49:55 +0800 Subject: [PATCH v7 3/4] Add some checks before using apply background worker to apply changes. If any of the following checks are violated, an error will be reported. 1. The unique columns between publisher and subscriber are difference. 2. There is any non-immutable function present in expression in subscriber's relation. Check from the following 3 items: a. The function in triggers; b. Column default value expressions and domain constraints; c. Constraint expressions. --- src/backend/replication/logical/proto.c | 60 +++++- src/backend/replication/logical/relation.c | 179 +++++++++++++++++- src/backend/replication/logical/tablesync.c | 1 + src/backend/replication/logical/worker.c | 47 +++++ src/backend/utils/cache/typcache.c | 17 ++ src/include/replication/logicalproto.h | 1 + src/include/replication/logicalrelation.h | 15 ++ src/include/utils/typcache.h | 2 + .../subscription/t/022_twophase_cascade.pl | 7 + 9 files changed, 324 insertions(+), 5 deletions(-) diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c index ff8513e2d2..7da0deb830 100644 --- a/src/backend/replication/logical/proto.c +++ b/src/backend/replication/logical/proto.c @@ -23,7 +23,8 @@ /* * Protocol message flags. */ -#define LOGICALREP_IS_REPLICA_IDENTITY 1 +#define LOGICALREP_IS_REPLICA_IDENTITY 0x0001 +#define LOGICALREP_IS_UNIQUE 0x0002 #define MESSAGE_TRANSACTIONAL (1<<0) #define TRUNCATE_CASCADE (1<<0) @@ -933,11 +934,53 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns) TupleDesc desc; int i; uint16 nliveatts = 0; - Bitmapset *idattrs = NULL; + Bitmapset *idattrs = NULL, + *attunique = NULL; bool replidentfull; desc = RelationGetDescr(rel); + if (rel->rd_rel->relhasindex) + { + List *indexoidlist = RelationGetIndexList(rel); + ListCell *indexoidscan; + foreach(indexoidscan, indexoidlist) + { + Oid indexoid = lfirst_oid(indexoidscan); + Relation indexRel; + + /* Look up the description for index */ + indexRel = RelationIdGetRelation(indexoid); + + if (!RelationIsValid(indexRel)) + elog(ERROR, "could not open relation with OID %u", indexoid); + + if (indexRel->rd_index->indisunique) + { + int i; + /* Add referenced attributes to idindexattrs */ + for (i = 0; i < indexRel->rd_index->indnatts; i++) + { + int attrnum = indexRel->rd_index->indkey.values[i]; + + /* + * We don't include non-key columns into idindexattrs bitmaps. See + * RelationGetIndexAttrBitmap. + */ + if (attrnum != 0) + { + if (i < indexRel->rd_index->indnkeyatts && + !bms_is_member(attrnum - FirstLowInvalidHeapAttributeNumber, attunique)) + attunique = bms_add_member(attunique, + attrnum - FirstLowInvalidHeapAttributeNumber); + } + } + } + RelationClose(indexRel); + } + list_free(indexoidlist); + } + /* send number of live attributes */ for (i = 0; i < desc->natts; i++) { @@ -976,6 +1019,10 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns) idattrs)) flags |= LOGICALREP_IS_REPLICA_IDENTITY; + if (bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber, + attunique)) + flags |= LOGICALREP_IS_UNIQUE; + pq_sendbyte(out, flags); /* attribute name */ @@ -1001,7 +1048,8 @@ logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel) int natts; char **attnames; Oid *atttyps; - Bitmapset *attkeys = NULL; + Bitmapset *attkeys = NULL, + *attunique = NULL; natts = pq_getmsgint(in, 2); attnames = palloc(natts * sizeof(char *)); @@ -1012,11 +1060,14 @@ logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel) { uint8 flags; - /* Check for replica identity column */ + /* Check for replica identity and unique column */ flags = pq_getmsgbyte(in); if (flags & LOGICALREP_IS_REPLICA_IDENTITY) attkeys = bms_add_member(attkeys, i); + if (flags & LOGICALREP_IS_UNIQUE) + attunique = bms_add_member(attunique, i); + /* attribute name */ attnames[i] = pstrdup(pq_getmsgstring(in)); @@ -1030,6 +1081,7 @@ logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel) rel->attnames = attnames; rel->atttyps = atttyps; rel->attkeys = attkeys; + rel->attunique = attunique; rel->natts = natts; } diff --git a/src/backend/replication/logical/relation.c b/src/backend/replication/logical/relation.c index 80fb561a9a..d038f7dc7a 100644 --- a/src/backend/replication/logical/relation.c +++ b/src/backend/replication/logical/relation.c @@ -19,12 +19,18 @@ #include "access/table.h" #include "catalog/namespace.h" +#include "catalog/pg_proc.h" #include "catalog/pg_subscription_rel.h" #include "executor/executor.h" #include "nodes/makefuncs.h" +#include "optimizer/optimizer.h" #include "replication/logicalrelation.h" #include "replication/worker_internal.h" +#include "rewrite/rewriteHandler.h" #include "utils/inval.h" +#include "utils/lsyscache.h" +#include "utils/syscache.h" +#include "utils/typcache.h" static MemoryContext LogicalRepRelMapContext = NULL; @@ -91,6 +97,23 @@ logicalrep_relmap_invalidate_cb(Datum arg, Oid reloid) } } +/* + * Reset the flag volatility of all existing entry in the relation map cache. + */ +static void +logicalrep_relmap_reset_volatility_cb(Datum arg, int cacheid, uint32 hashvalue) +{ + HASH_SEQ_STATUS hash_seq; + LogicalRepRelMapEntry *entry; + + if (LogicalRepRelMap == NULL) + return; + + hash_seq_init(&hash_seq, LogicalRepRelMap); + while ((entry = hash_seq_search(&hash_seq)) != NULL) + entry->volatility = FUNCTION_UNKNOWN; +} + /* * Initialize the relation map cache. */ @@ -116,6 +139,9 @@ logicalrep_relmap_init(void) /* Watch for invalidation events. */ CacheRegisterRelcacheCallback(logicalrep_relmap_invalidate_cb, (Datum) 0); + CacheRegisterSyscacheCallback(PROCOID, + logicalrep_relmap_reset_volatility_cb, + (Datum) 0); } /* @@ -142,6 +168,7 @@ logicalrep_relmap_free_entry(LogicalRepRelMapEntry *entry) pfree(remoterel->atttyps); } bms_free(remoterel->attkeys); + bms_free(remoterel->attunique); if (entry->attrmap) pfree(entry->attrmap); @@ -190,6 +217,7 @@ logicalrep_relmap_update(LogicalRepRelation *remoterel) } entry->remoterel.replident = remoterel->replident; entry->remoterel.attkeys = bms_copy(remoterel->attkeys); + entry->remoterel.attunique = bms_copy(remoterel->attunique); MemoryContextSwitchTo(oldctx); } @@ -307,7 +335,8 @@ logicalrep_rel_open(LogicalRepRelId remoteid, LOCKMODE lockmode) if (!entry->localrelvalid) { Oid relid; - Bitmapset *idkey; + Bitmapset *idkey, + *ukey; TupleDesc desc; MemoryContext oldctx; int i; @@ -415,6 +444,153 @@ logicalrep_rel_open(LogicalRepRelId remoteid, LOCKMODE lockmode) } } + /* + * Check whether the unique index of publisher and subscriber are + * consistent. + */ + entry->sameunique = true; + ukey = RelationGetIndexAttrBitmap(entry->localrel, + INDEX_ATTR_BITMAP_KEY); + + if (ukey) + { + i = -1; + while ((i = bms_next_member(ukey, i)) >= 0) + { + int attnum = i + FirstLowInvalidHeapAttributeNumber; + + attnum = AttrNumberGetAttrOffset(attnum); + + if (entry->attrmap->attnums[attnum] < 0 || + !bms_is_member(entry->attrmap->attnums[attnum], remoterel->attunique)) + { + entry->sameunique = false; + break; + } + } + } + + if (entry->volatility == FUNCTION_UNKNOWN) + { + /* + * Check from the following points: + * + * a. The function in triggers; + * b. Column default value expressions and domain constraints; + * c. Constraint expressions. + */ + bool nonimmutable = false; + + /* Check the trigger functions. */ + if (!nonimmutable) + { + int i; + + if (entry->localrel->trigdesc != NULL) + for (i = 0; i < entry->localrel->trigdesc->numtriggers; i++) + { + Trigger *trig = entry->localrel->trigdesc->triggers + i; + + if (func_volatile(trig->tgfoid) != PROVOLATILE_IMMUTABLE) + { + nonimmutable = true; + break; + } + } + } + + /* Check the columns. */ + if (!nonimmutable) + { + TupleDesc tupdesc = RelationGetDescr(entry->localrel); + int attnum; + + for (attnum = 0; attnum < tupdesc->natts; attnum++) + { + Form_pg_attribute att = TupleDescAttr(tupdesc, attnum); + + /* break if we set nonimmutable in last check for domain. */ + if (nonimmutable) + break; + + /* We don't need info for dropped or generated attributes */ + if (att->attisdropped || att->attgenerated) + continue; + + /* We don't need to check columns that only exist on the subscriber */ + if (entry->attrmap->attnums[attnum] < 0) + continue; + + if (att->atthasdef) + { + Node *defaultexpr; + + defaultexpr = build_column_default(entry->localrel, attnum + 1); + if (contain_mutable_functions(defaultexpr)) + { + nonimmutable = true; + break; + } + } + + /* + * If the column is of a DOMAIN type, determine whether that + * domain has any CHECK expressions that are not immutable. + */ + if (get_typtype(att->atttypid) == TYPTYPE_DOMAIN) + { + List *domain_constraints; + ListCell *lc; + + domain_constraints = GetDomainConstraints(att->atttypid); + + foreach(lc, domain_constraints) + { + DomainConstraintState *con = (DomainConstraintState *) lfirst(lc); + + if (con->check_expr && contain_mutable_functions((Node *) con->check_expr)) + { + nonimmutable = true; + break; + } + } + } + } + } + + /* Check the constraints. */ + if (!nonimmutable) + { + TupleDesc tupdesc = RelationGetDescr(entry->localrel); + + if (tupdesc->constr) + { + ConstrCheck *check = tupdesc->constr->check; + int i; + + /* + * Determine if there are any CHECK constraints which contains + * non-immutable function. + */ + for (i = 0; i < tupdesc->constr->num_check; i++) + { + Expr *check_expr = stringToNode(check[i].ccbin); + + if (contain_mutable_functions((Node *) check_expr)) + { + nonimmutable = true; + break; + } + } + } + } + + if (nonimmutable) + entry->volatility = FUNCTION_NONIMMUTABLE; + else + entry->volatility = FUNCTION_IMMUTABLE; + } + entry->localrelvalid = true; } @@ -570,6 +746,7 @@ logicalrep_partition_open(LogicalRepRelMapEntry *root, } entry->remoterel.replident = remoterel->replident; entry->remoterel.attkeys = bms_copy(remoterel->attkeys); + entry->remoterel.attunique = bms_copy(remoterel->attunique); entry->localrel = partrel; entry->localreloid = partOid; diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c index 5eb60327b6..4885eaa078 100644 --- a/src/backend/replication/logical/tablesync.c +++ b/src/backend/replication/logical/tablesync.c @@ -880,6 +880,7 @@ fetch_remote_table_info(char *nspname, char *relname, lrel->attnames = palloc0(MaxTupleAttributeNumber * sizeof(char *)); lrel->atttyps = palloc0(MaxTupleAttributeNumber * sizeof(Oid)); lrel->attkeys = NULL; + lrel->attunique = NULL; /* * Store the columns as a list of names. Ignore those that are not diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c index 788c1c65c9..40c811aa0e 100644 --- a/src/backend/replication/logical/worker.c +++ b/src/backend/replication/logical/worker.c @@ -299,6 +299,7 @@ static void apply_bgworker_send_data(ApplyBgworkerState *wstate, Size nbytes, static void apply_bgworker_free(ApplyBgworkerState *wstate); static void apply_bgworker_check_status(void); static void apply_bgworker_set_status(ApplyBgworkerStatus status); +static void apply_bgworker_relation_check(LogicalRepRelMapEntry *rel); typedef struct FlushPosition { @@ -2142,6 +2143,8 @@ apply_handle_insert(StringInfo s) /* Set relation for error callback */ apply_error_callback_arg.rel = rel; + apply_bgworker_relation_check(rel); + /* Initialize the executor state. */ edata = create_edata_for_relation(rel); estate = edata->estate; @@ -2278,6 +2281,8 @@ apply_handle_update(StringInfo s) /* Check if we can do the update. */ check_relation_updatable(rel); + apply_bgworker_relation_check(rel); + /* Initialize the executor state. */ edata = create_edata_for_relation(rel); estate = edata->estate; @@ -2446,6 +2451,8 @@ apply_handle_delete(StringInfo s) /* Check if we can do the delete. */ check_relation_updatable(rel); + apply_bgworker_relation_check(rel); + /* Initialize the executor state. */ edata = create_edata_for_relation(rel); estate = edata->estate; @@ -5132,3 +5139,43 @@ apply_bgworker_set_status(ApplyBgworkerStatus status) MyParallelState->status = status; SpinLockRelease(&MyParallelState->mutex); } + +/* + * Check if changes on this logical replication relation can be applied by + * apply background worker. + */ +static void +apply_bgworker_relation_check(LogicalRepRelMapEntry *rel) +{ + /* Check only we are in apply bgworker. */ + if (!am_apply_bgworker()) + return; + + /* + * FIXME: Checks on partitioned tables are currently not supported. + * Adding this check requires additional fixes for other issues. + * I will add this later, after the related issues are fixed. + */ + if (rel->localrel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + return; + + /* + * If any unique index exist, check that they are same as remoterel. + */ + if (!rel->sameunique) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot replicate relation with different unique index"), + errhint("Please change the streaming option to 'on' instead of 'apply'."))); + + /* + * Check if there is any non-immutable function present in expression in + * this relation. + */ + if (rel->volatility == FUNCTION_NONIMMUTABLE) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot replicate relation. There is at least one non-immutable function"), + errhint("Please change the streaming option to 'on' instead of 'apply'."))); + +} diff --git a/src/backend/utils/cache/typcache.c b/src/backend/utils/cache/typcache.c index 808f9ebd0d..b248899d82 100644 --- a/src/backend/utils/cache/typcache.c +++ b/src/backend/utils/cache/typcache.c @@ -2540,6 +2540,23 @@ compare_values_of_enum(TypeCacheEntry *tcache, Oid arg1, Oid arg2) return 0; } +/* + * GetDomainConstraints --- get DomainConstraintState list of specified domain type + */ +List * +GetDomainConstraints(Oid type_id) +{ + TypeCacheEntry *typentry; + List *constraints = NIL; + + typentry = lookup_type_cache(type_id, TYPECACHE_DOMAIN_CONSTR_INFO); + + if(typentry->domainData != NULL) + constraints = typentry->domainData->constraints; + + return constraints; +} + /* * Load (or re-load) the enumData member of the typcache entry. */ diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h index a771ab8ff3..a88535820b 100644 --- a/src/include/replication/logicalproto.h +++ b/src/include/replication/logicalproto.h @@ -108,6 +108,7 @@ typedef struct LogicalRepRelation char replident; /* replica identity */ char relkind; /* remote relation kind */ Bitmapset *attkeys; /* Bitmap of key columns */ + Bitmapset *attunique; /* Bitmap of unique columns */ } LogicalRepRelation; /* Type mapping info */ diff --git a/src/include/replication/logicalrelation.h b/src/include/replication/logicalrelation.h index 7bf8cd22bd..9f568e9c00 100644 --- a/src/include/replication/logicalrelation.h +++ b/src/include/replication/logicalrelation.h @@ -15,6 +15,17 @@ #include "access/attmap.h" #include "replication/logicalproto.h" +/* + * States to determine volatility of the function in expressions in one + * relation. + */ +typedef enum RelFuncVolatility +{ + FUNCTION_UNKNOWN = 0, /* initializing */ + FUNCTION_IMMUTABLE, /* all functions are immutable function */ + FUNCTION_NONIMMUTABLE /* at least one non-immutable function */ +} RelFuncVolatility; + typedef struct LogicalRepRelMapEntry { LogicalRepRelation remoterel; /* key is remoterel.remoteid */ @@ -31,6 +42,10 @@ typedef struct LogicalRepRelMapEntry Relation localrel; /* relcache entry (NULL when closed) */ AttrMap *attrmap; /* map of local attributes to remote ones */ bool updatable; /* Can apply updates/deletes? */ + bool sameunique; /* Is the unique column of local and remote + consistent? */ + RelFuncVolatility volatility; /* all functions in localrel are + immutable function? */ /* Sync state. */ char state; diff --git a/src/include/utils/typcache.h b/src/include/utils/typcache.h index 431ad7f1b3..ed7c2e7f48 100644 --- a/src/include/utils/typcache.h +++ b/src/include/utils/typcache.h @@ -199,6 +199,8 @@ extern uint64 assign_record_type_identifier(Oid type_id, int32 typmod); extern int compare_values_of_enum(TypeCacheEntry *tcache, Oid arg1, Oid arg2); +extern List *GetDomainConstraints(Oid type_id); + extern size_t SharedRecordTypmodRegistryEstimate(void); extern void SharedRecordTypmodRegistryInit(SharedRecordTypmodRegistry *, diff --git a/src/test/subscription/t/022_twophase_cascade.pl b/src/test/subscription/t/022_twophase_cascade.pl index 663808acb9..b317144230 100644 --- a/src/test/subscription/t/022_twophase_cascade.pl +++ b/src/test/subscription/t/022_twophase_cascade.pl @@ -38,6 +38,13 @@ sub test_streaming ALTER SUBSCRIPTION tap_sub_C SET (streaming = $streaming_mode)"); + if ($streaming_mode eq 'apply') + { + $node_C->safe_psql( + 'postgres', " + ALTER TABLE test_tab ALTER c DROP DEFAULT"); + } + # Wait for subscribers to finish initialization $node_A->poll_query_until( -- 2.23.0.windows.1 [application/octet-stream] v7-0004-Add-a-GUC-max_apply_bgworkers_per_subscription-to.patch (6.9K, ../../OS3PR01MB6275797F66EF0A47EEB2D8FC9EDD9@OS3PR01MB6275.jpnprd01.prod.outlook.com/5-v7-0004-Add-a-GUC-max_apply_bgworkers_per_subscription-to.patch) download | inline diff: From a975b33b26147b039524c26789cab4eefd18e4aa Mon Sep 17 00:00:00 2001 From: wangw <[email protected]> Date: Mon, 30 May 2022 13:42:41 +0800 Subject: [PATCH v7 4/4] Add a GUC "max_apply_bgworkers_per_subscription" to control parallelism. This GUC controls how many apply background workers can be launched per subscription. --- doc/src/sgml/config.sgml | 25 +++++++++++++++++++ src/backend/replication/logical/launcher.c | 25 +++++++++++++++++++ src/backend/replication/logical/worker.c | 9 +++++++ src/backend/utils/misc/guc.c | 12 +++++++++ src/backend/utils/misc/postgresql.conf.sample | 1 + src/include/replication/logicallauncher.h | 1 + src/include/replication/worker_internal.h | 1 + 7 files changed, 74 insertions(+) diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml index 5b7ce6531d..ce7f6827f2 100644 --- a/doc/src/sgml/config.sgml +++ b/doc/src/sgml/config.sgml @@ -4970,6 +4970,31 @@ ANY <replaceable class="parameter">num_sync</replaceable> ( <replaceable class=" </listitem> </varlistentry> + <varlistentry id="guc-max-apply-bgworkers-per-subscription" xreflabel="max_apply_bgworkers_per_subscription"> + <term><varname>max_apply_bgworkers_per_subscription</varname> (<type>integer</type>) + <indexterm> + <primary><varname>max_apply_bgworkers_per_subscription</varname> configuration parameter</primary> + </indexterm> + </term> + <listitem> + <para> + Maximum number of apply background workers per subscription. This + parameter controls the amount of parallelism of the streaming of + in-progress transactions if we set subscription option + <literal>streaming</literal> to <literal>apply</literal>. + </para> + <para> + The apply background workers are taken from the pool defined by + <varname>max_logical_replication_workers</varname>. + </para> + <para> + The default value is 3. This parameter can only be set in the + <filename>postgresql.conf</filename> file or on the server command + line. + </para> + </listitem> + </varlistentry> + </variablelist> </sect2> diff --git a/src/backend/replication/logical/launcher.c b/src/backend/replication/logical/launcher.c index 58337bef91..dd75e16cbf 100644 --- a/src/backend/replication/logical/launcher.c +++ b/src/backend/replication/logical/launcher.c @@ -54,6 +54,7 @@ int max_logical_replication_workers = 4; int max_sync_workers_per_subscription = 2; +int max_apply_bgworkers_per_subscription = 3; LogicalRepWorker *MyLogicalRepWorker = NULL; @@ -735,6 +736,30 @@ logicalrep_sync_worker_count(Oid subid) return res; } +/* + * Count the number of registered (not necessarily running) apply background + * worker for a subscription. + */ +int +logicalrep_apply_background_worker_count(Oid subid) +{ + int i; + int res = 0; + + Assert(LWLockHeldByMe(LogicalRepWorkerLock)); + + /* Search for attached worker for a given subscription id. */ + for (i = 0; i < max_logical_replication_workers; i++) + { + LogicalRepWorker *w = &LogicalRepCtx->workers[i]; + + if (w->subid == subid && w->subworker) + res++; + } + + return res; +} + /* * ApplyLauncherShmemSize * Compute space needed for replication launcher shared memory diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c index 40c811aa0e..1ba4df44b6 100644 --- a/src/backend/replication/logical/worker.c +++ b/src/backend/replication/logical/worker.c @@ -180,6 +180,7 @@ #include "postmaster/walwriter.h" #include "replication/decode.h" #include "replication/logical.h" +#include "replication/logicallauncher.h" #include "replication/logicalproto.h" #include "replication/logicalrelation.h" #include "replication/logicalworker.h" @@ -4989,9 +4990,17 @@ apply_bgworker_setup(void) MemoryContext oldcontext; bool launched; ApplyBgworkerState *wstate; + int napplyworkers; elog(DEBUG1, "setting up apply worker #%u", list_length(ApplyWorkersList) + 1); + /* Check If there are free worker slot(s) */ + LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); + napplyworkers = logicalrep_apply_background_worker_count(MyLogicalRepWorker->subid); + LWLockRelease(LogicalRepWorkerLock); + if (napplyworkers >= max_apply_bgworkers_per_subscription) + return NULL; + oldcontext = MemoryContextSwitchTo(ApplyContext); wstate = (ApplyBgworkerState *) palloc0(sizeof(ApplyBgworkerState)); diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c index 55d41ae7d6..539ec07ff5 100644 --- a/src/backend/utils/misc/guc.c +++ b/src/backend/utils/misc/guc.c @@ -3220,6 +3220,18 @@ static struct config_int ConfigureNamesInt[] = NULL, NULL, NULL }, + { + {"max_apply_bgworkers_per_subscription", + PGC_SIGHUP, + REPLICATION_SUBSCRIBERS, + gettext_noop("Maximum number of apply backgrand workers per subscription."), + NULL, + }, + &max_apply_bgworkers_per_subscription, + 3, 0, MAX_BACKENDS, + NULL, NULL, NULL + }, + { {"log_rotation_age", PGC_SIGHUP, LOGGING_WHERE, gettext_noop("Sets the amount of time to wait before forcing " diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample index 48ad80cf2e..b71340549d 100644 --- a/src/backend/utils/misc/postgresql.conf.sample +++ b/src/backend/utils/misc/postgresql.conf.sample @@ -360,6 +360,7 @@ #max_logical_replication_workers = 4 # taken from max_worker_processes # (change requires restart) #max_sync_workers_per_subscription = 2 # taken from max_logical_replication_workers +#max_apply_bgworkers_per_subscription = 3 # taken from max_logical_replication_workers #------------------------------------------------------------------------------ diff --git a/src/include/replication/logicallauncher.h b/src/include/replication/logicallauncher.h index f1e2821e25..ac8ef94381 100644 --- a/src/include/replication/logicallauncher.h +++ b/src/include/replication/logicallauncher.h @@ -14,6 +14,7 @@ extern PGDLLIMPORT int max_logical_replication_workers; extern PGDLLIMPORT int max_sync_workers_per_subscription; +extern PGDLLIMPORT int max_apply_bgworkers_per_subscription; extern void ApplyLauncherRegister(void); extern void ApplyLauncherMain(Datum main_arg); diff --git a/src/include/replication/worker_internal.h b/src/include/replication/worker_internal.h index 1feafccaf6..f1b9a242c6 100644 --- a/src/include/replication/worker_internal.h +++ b/src/include/replication/worker_internal.h @@ -95,6 +95,7 @@ extern void logicalrep_worker_wakeup(Oid subid, Oid relid); extern void logicalrep_worker_wakeup_ptr(LogicalRepWorker *worker); extern int logicalrep_sync_worker_count(Oid subid); +extern int logicalrep_apply_background_worker_count(Oid subid); extern void ReplicationOriginNameForTablesync(Oid suboid, Oid relid, char *originname, int szorgname); -- 2.23.0.windows.1 ^ permalink raw reply [nested|flat] 66+ messages in thread
* Re: Perform streaming logical transactions by background workers and parallel apply @ 2022-05-30 11:38 Amit Kapila <[email protected]> parent: [email protected] <[email protected]> 0 siblings, 2 replies; 66+ messages in thread From: Amit Kapila @ 2022-05-30 11:38 UTC (permalink / raw) To: [email protected] <[email protected]>; +Cc: Peter Smith <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; PostgreSQL Hackers <[email protected]> On Mon, May 30, 2022 at 2:22 PM [email protected] <[email protected]> wrote: > > Attach the new patches(only changed 0001 and 0002) > Few comments/suggestions for 0001 and 0003 ===================================== 0001 -------- 1. + else + snprintf(bgw.bgw_name, BGW_MAXLEN, + "logical replication apply worker for subscription %u", subid); Can we slightly change the message to: "logical replication background apply worker for subscription %u"? 2. Can we think of separating the new logic for applying the xact by bgworker into a new file like applybgwroker or applyparallel? We have previously done the same in the case of vacuum (see vacuumparallel.c). 3. + /* + * XXX The publisher side doesn't always send relation update messages + * after the streaming transaction, so update the relation in main + * apply worker here. + */ + if (action == LOGICAL_REP_MSG_RELATION) + { + LogicalRepRelation *rel = logicalrep_read_rel(s); + logicalrep_relmap_update(rel); + } I think the publisher side won't send the relation update message after streaming transaction only if it has already been sent for a non-streaming transaction in which case we don't need to update the local cache here. This is as per my understanding of maybe_send_schema(), do let me know if I am missing something? If my understanding is correct then we don't need this change. 4. + * For the main apply worker, if in streaming mode (receiving a block of + * streamed transaction), we send the data to the apply background worker. * - * If in streaming mode (receiving a block of streamed transaction), we - * simply redirect it to a file for the proper toplevel transaction. This comment is slightly confusing. Can we change it to something like: "In streaming case (receiving a block of streamed transaction), for SUBSTREAM_ON mode, we simply redirect it to a file for the proper toplevel transaction, and for SUBSTREAM_APPLY mode, we send the changes to background apply worker."? 5. +apply_handle_stream_abort(StringInfo s) { ... ... + /* + * If the two XIDs are the same, it's in fact abort of toplevel xact, + * so just free the subxactlist. + */ + if (subxid == xid) + { + set_apply_error_context_xact(subxid, InvalidXLogRecPtr); - fd = BufFileOpenFileSet(MyLogicalRepWorker->stream_fileset, path, O_RDONLY, - false); + AbortCurrentTransaction(); - buffer = palloc(BLCKSZ); + EndTransactionBlock(false); + CommitTransactionCommand(); + + in_remote_transaction = false; ... ... } Here, can we update the replication origin as we are doing in apply_handle_rollback_prepared? Currently, we don't do it because we are just cleaning up temporary files for which we don't even have a transaction. Also, we don't have the required infrastructure to advance origins for aborts as we have for abort prepared. See commits [1eb6d6527a][8a812e5106]. If we think it is a good idea then I think we need to send abort_lsn and abort_time from the publisher and we need to be careful to make it work with lower subscriber versions that don't have the facility to process these additional values. 0003 -------- 6. + /* + * If any unique index exist, check that they are same as remoterel. + */ + if (!rel->sameunique) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot replicate relation with different unique index"), + errhint("Please change the streaming option to 'on' instead of 'apply'."))); I think we can do better here. Instead of simply erroring out and asking the user to change streaming mode, we can remember this in the system catalog probably in pg_subscription, and then on restart, we can change the streaming mode to 'on', perform the transaction, and again change the streaming mode to apply. I am not sure whether we want to do it in the first version or not, so if you agree with this, developing it as a separate patch would be a good idea. Also, please update comments here as to why we don't handle such cases. -- With Regards, Amit Kapila. ^ permalink raw reply [nested|flat] 66+ messages in thread
* Re: Perform streaming logical transactions by background workers and parallel apply @ 2022-05-31 08:52 Amit Kapila <[email protected]> parent: Amit Kapila <[email protected]> 1 sibling, 1 reply; 66+ messages in thread From: Amit Kapila @ 2022-05-31 08:52 UTC (permalink / raw) To: [email protected] <[email protected]>; +Cc: Peter Smith <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; PostgreSQL Hackers <[email protected]> On Mon, May 30, 2022 at 5:08 PM Amit Kapila <[email protected]> wrote: > > On Mon, May 30, 2022 at 2:22 PM [email protected] > <[email protected]> wrote: > > > > Attach the new patches(only changed 0001 and 0002) > > > This patch allows the same replication origin to be used by the main apply worker and the bgworker that uses it to apply streaming transactions. See the changes [1] in the patch. I am not completely sure whether that is a good idea even though I could not spot or think of problems that can't be fixed in your patch. I see that currently both the main apply worker and bgworker will assign MyProcPid to the assigned origin slot, this can create the problem because ReplicationOriginExitCleanup() can clean it up even though the main apply worker or another bgworker is still using that origin slot. Now, one way to fix is that we assign only the main apply worker's MyProcPid to session_replication_state->acquired_by. I have tried to think about the concurrency issues as multiple workers could now point to the same replication origin state. I think it is safe because the patch maintains the commit order by allowing only one process to commit at a time, so no two workers will be operating on the same origin at the same time. Even, though there is no case where the patch will try to advance the session's origin concurrently, it appears safe to do so as we change/advance the session_origin LSNs under replicate_state LWLock. Another idea could be that we allow multiple replication origins (one for each bgworker and one for the main apply worker) for the apply workers corresponding to a subscription. Then on restart, we can find the highest LSN among all the origins for a subscription. This should work primarily because we will maintain the commit order. Now, for this to work we need to somehow map all the origins for a subscription and one possibility is that we have a subscription id in each of the origin names. Currently we use ("pg_%u", MySubscription->oid) as origin_name. We can probably append some unique identifier number for each worker to allow each origin to have a subscription id. We need to drop all origins for a particular subscription on DROP SUBSCRIPTION. I think having multiple origins for the same subscription will have some additional work when we try to filter changes based on origin. The advantage of the first idea is that it won't increase the need to have more origins per subscription but it is quite possible that I am missing something and there are problems due to which we can't use that approach. Thoughts? [1]: -replorigin_session_setup(RepOriginId node) +replorigin_session_setup(RepOriginId node, bool acquire) { static bool registered_cleanup; int i; @@ -1110,7 +1110,7 @@ replorigin_session_setup(RepOriginId node) if (curstate->roident != node) continue; - else if (curstate->acquired_by != 0) + else if (curstate->acquired_by != 0 && acquire) { ... ... + /* + * The apply bgworker don't need to monopolize this replication origin + * which was already acquired by its leader process. + */ + replorigin_session_setup(originid, false); + replorigin_session_origin = originid; -- With Regards, Amit Kapila. ^ permalink raw reply [nested|flat] 66+ messages in thread
* Re: Perform streaming logical transactions by background workers and parallel apply @ 2022-06-01 02:00 Masahiko Sawada <[email protected]> parent: Amit Kapila <[email protected]> 0 siblings, 1 reply; 66+ messages in thread From: Masahiko Sawada @ 2022-06-01 02:00 UTC (permalink / raw) To: Amit Kapila <[email protected]>; +Cc: [email protected] <[email protected]>; Peter Smith <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; PostgreSQL Hackers <[email protected]> On Tue, May 31, 2022 at 5:53 PM Amit Kapila <[email protected]> wrote: > > On Mon, May 30, 2022 at 5:08 PM Amit Kapila <[email protected]> wrote: > > > > On Mon, May 30, 2022 at 2:22 PM [email protected] > > <[email protected]> wrote: > > > > > > Attach the new patches(only changed 0001 and 0002) > > > > > > > This patch allows the same replication origin to be used by the main > apply worker and the bgworker that uses it to apply streaming > transactions. See the changes [1] in the patch. I am not completely > sure whether that is a good idea even though I could not spot or think > of problems that can't be fixed in your patch. I see that currently > both the main apply worker and bgworker will assign MyProcPid to the > assigned origin slot, this can create the problem because > ReplicationOriginExitCleanup() can clean it up even though the main > apply worker or another bgworker is still using that origin slot. Good point. > Now, > one way to fix is that we assign only the main apply worker's > MyProcPid to session_replication_state->acquired_by. I have tried to > think about the concurrency issues as multiple workers could now point > to the same replication origin state. I think it is safe because the > patch maintains the commit order by allowing only one process to > commit at a time, so no two workers will be operating on the same > origin at the same time. Even, though there is no case where the patch > will try to advance the session's origin concurrently, it appears safe > to do so as we change/advance the session_origin LSNs under > replicate_state LWLock. Right. That way, the cleanup is done only by the main apply worker. Probably the bgworker can check if the origin is already acquired by its (leader) main apply worker process for safety. > > Another idea could be that we allow multiple replication origins (one > for each bgworker and one for the main apply worker) for the apply > workers corresponding to a subscription. Then on restart, we can find > the highest LSN among all the origins for a subscription. This should > work primarily because we will maintain the commit order. Now, for > this to work we need to somehow map all the origins for a subscription > and one possibility is that we have a subscription id in each of the > origin names. Currently we use ("pg_%u", MySubscription->oid) as > origin_name. We can probably append some unique identifier number for > each worker to allow each origin to have a subscription id. We need to > drop all origins for a particular subscription on DROP SUBSCRIPTION. I > think having multiple origins for the same subscription will have some > additional work when we try to filter changes based on origin. It also seems to work but need additional work and resource. > The advantage of the first idea is that it won't increase the need to > have more origins per subscription but it is quite possible that I am > missing something and there are problems due to which we can't use > that approach. I prefer the first idea as it's simpler than the second one. I don't see any concurrency problem so far unless I'm not missing something. Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/ ^ permalink raw reply [nested|flat] 66+ messages in thread
* Re: Perform streaming logical transactions by background workers and parallel apply @ 2022-06-01 05:19 Amit Kapila <[email protected]> parent: Masahiko Sawada <[email protected]> 0 siblings, 1 reply; 66+ messages in thread From: Amit Kapila @ 2022-06-01 05:19 UTC (permalink / raw) To: Masahiko Sawada <[email protected]>; +Cc: [email protected] <[email protected]>; Peter Smith <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; PostgreSQL Hackers <[email protected]> On Wed, Jun 1, 2022 at 7:30 AM Masahiko Sawada <[email protected]> wrote: > > On Tue, May 31, 2022 at 5:53 PM Amit Kapila <[email protected]> wrote: > > > > On Mon, May 30, 2022 at 5:08 PM Amit Kapila <[email protected]> wrote: > > > > > > On Mon, May 30, 2022 at 2:22 PM [email protected] > > > <[email protected]> wrote: > > > > > > > > Attach the new patches(only changed 0001 and 0002) > > > > > > > > > > > This patch allows the same replication origin to be used by the main > > apply worker and the bgworker that uses it to apply streaming > > transactions. See the changes [1] in the patch. I am not completely > > sure whether that is a good idea even though I could not spot or think > > of problems that can't be fixed in your patch. I see that currently > > both the main apply worker and bgworker will assign MyProcPid to the > > assigned origin slot, this can create the problem because > > ReplicationOriginExitCleanup() can clean it up even though the main > > apply worker or another bgworker is still using that origin slot. > > Good point. > > > Now, > > one way to fix is that we assign only the main apply worker's > > MyProcPid to session_replication_state->acquired_by. I have tried to > > think about the concurrency issues as multiple workers could now point > > to the same replication origin state. I think it is safe because the > > patch maintains the commit order by allowing only one process to > > commit at a time, so no two workers will be operating on the same > > origin at the same time. Even, though there is no case where the patch > > will try to advance the session's origin concurrently, it appears safe > > to do so as we change/advance the session_origin LSNs under > > replicate_state LWLock. > > Right. That way, the cleanup is done only by the main apply worker. > Probably the bgworker can check if the origin is already acquired by > its (leader) main apply worker process for safety. > Yeah, that makes sense. > > > > Another idea could be that we allow multiple replication origins (one > > for each bgworker and one for the main apply worker) for the apply > > workers corresponding to a subscription. Then on restart, we can find > > the highest LSN among all the origins for a subscription. This should > > work primarily because we will maintain the commit order. Now, for > > this to work we need to somehow map all the origins for a subscription > > and one possibility is that we have a subscription id in each of the > > origin names. Currently we use ("pg_%u", MySubscription->oid) as > > origin_name. We can probably append some unique identifier number for > > each worker to allow each origin to have a subscription id. We need to > > drop all origins for a particular subscription on DROP SUBSCRIPTION. I > > think having multiple origins for the same subscription will have some > > additional work when we try to filter changes based on origin. > > It also seems to work but need additional work and resource. > > > The advantage of the first idea is that it won't increase the need to > > have more origins per subscription but it is quite possible that I am > > missing something and there are problems due to which we can't use > > that approach. > > I prefer the first idea as it's simpler than the second one. I don't > see any concurrency problem so far unless I'm not missing something. > Thanks for evaluating it and sharing your opinion. -- With Regards, Amit Kapila. ^ permalink raw reply [nested|flat] 66+ messages in thread
* RE: Perform streaming logical transactions by background workers and parallel apply @ 2022-06-02 10:01 [email protected] <[email protected]> parent: Amit Kapila <[email protected]> 0 siblings, 1 reply; 66+ messages in thread From: [email protected] @ 2022-06-02 10:01 UTC (permalink / raw) To: Amit Kapila <[email protected]>; Masahiko Sawada <[email protected]>; +Cc: Peter Smith <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; PostgreSQL Hackers <[email protected]> On Wed, Jun 1, 2022 1:19 PM Amit Kapila <[email protected]> wrote: > On Wed, Jun 1, 2022 at 7:30 AM Masahiko Sawada <[email protected]> > wrote: > > > > On Tue, May 31, 2022 at 5:53 PM Amit Kapila <[email protected]> > wrote: > > > > > > On Mon, May 30, 2022 at 5:08 PM Amit Kapila <[email protected]> > wrote: > > > > > > > > On Mon, May 30, 2022 at 2:22 PM [email protected] > > > > <[email protected]> wrote: > > > > > > > > > > Attach the new patches(only changed 0001 and 0002) > > > > > > > > > > > > > > > This patch allows the same replication origin to be used by the main > > > apply worker and the bgworker that uses it to apply streaming > > > transactions. See the changes [1] in the patch. I am not completely > > > sure whether that is a good idea even though I could not spot or think > > > of problems that can't be fixed in your patch. I see that currently > > > both the main apply worker and bgworker will assign MyProcPid to the > > > assigned origin slot, this can create the problem because > > > ReplicationOriginExitCleanup() can clean it up even though the main > > > apply worker or another bgworker is still using that origin slot. > > > > Good point. > > > > > Now, > > > one way to fix is that we assign only the main apply worker's > > > MyProcPid to session_replication_state->acquired_by. I have tried to > > > think about the concurrency issues as multiple workers could now point > > > to the same replication origin state. I think it is safe because the > > > patch maintains the commit order by allowing only one process to > > > commit at a time, so no two workers will be operating on the same > > > origin at the same time. Even, though there is no case where the patch > > > will try to advance the session's origin concurrently, it appears safe > > > to do so as we change/advance the session_origin LSNs under > > > replicate_state LWLock. > > > > Right. That way, the cleanup is done only by the main apply worker. > > Probably the bgworker can check if the origin is already acquired by > > its (leader) main apply worker process for safety. > > > > Yeah, that makes sense. > > > > > > > Another idea could be that we allow multiple replication origins (one > > > for each bgworker and one for the main apply worker) for the apply > > > workers corresponding to a subscription. Then on restart, we can find > > > the highest LSN among all the origins for a subscription. This should > > > work primarily because we will maintain the commit order. Now, for > > > this to work we need to somehow map all the origins for a subscription > > > and one possibility is that we have a subscription id in each of the > > > origin names. Currently we use ("pg_%u", MySubscription->oid) as > > > origin_name. We can probably append some unique identifier number for > > > each worker to allow each origin to have a subscription id. We need to > > > drop all origins for a particular subscription on DROP SUBSCRIPTION. I > > > think having multiple origins for the same subscription will have some > > > additional work when we try to filter changes based on origin. > > > > It also seems to work but need additional work and resource. > > > > > The advantage of the first idea is that it won't increase the need to > > > have more origins per subscription but it is quite possible that I am > > > missing something and there are problems due to which we can't use > > > that approach. > > > > I prefer the first idea as it's simpler than the second one. I don't > > see any concurrency problem so far unless I'm not missing something. > > > > Thanks for evaluating it and sharing your opinion. Thanks for your comments and opinions. I fixed this problem by following the first suggestion. I also added the relevant checks and changed the relevant comments. Thanks for Shi Yu to add some tests as suggested by Osumi-san in [1].#4 and improve the 0002 patch by adding some checks to see if the apply background worker starts. Attach the new patches. 1. Add some descriptions related to "apply" mode to logical-replication.sgml and create_subscription.sgml.(suggested by Osumi-san in [1].#1,#4) 2. Fix the problem that values in pg_stat_subscription_stats are not updated properly. (suggested by Osumi-san in [1].#2) 3. Improve the code formatting of the patches. (suggested by Osumi-san in [1].#3) 4. Add some tests in 0003 patch. And improve some tests by adding some checks to see if the apply background worker starts in 0002 patch. (suggested by Osumi-san in [1].#4 and Shi Yu) 5. Improve the log message. (suggested by Amit-san in [2].#1) 6. Separate the new logic related to apply background worker to new file applybgwroker.c. (suggested by Amit-san in [2].#2) 7. Improve function handle_streamed_transaction. (suggested by Amit-san in[2].#3) 8. Improve some comments. (suggested by Amit-san in [2].#4,#6 and me) 9. Fix the problem that the structure member "acquired_by" is incorrectly set when apply background worker tries to get replication origin. (suggested by Amit-san in [3]) [1] - https://www.postgresql.org/message-id/TYCPR01MB83735AEE38370254ED495B06EDDA9%40TYCPR01MB8373.jpnprd0... [2] - https://www.postgresql.org/message-id/CAA4eK1Jt08SYbRt_-rbSWNg%3DX9-m8%2BRdP5PosfnQgyF-z8bkxQ%40mail... [3] - https://www.postgresql.org/message-id/CAA4eK1%2BZ6ahpTQK2KzkvQ1kN-urVS9-N_RDM11MS%2BbtqaB8Bpw%40mail... Regards, Wang wei Attachments: [application/octet-stream] v8-0001-Perform-streaming-logical-transactions-by-backgro.patch (81.4K, ../../OS3PR01MB62758A881FF3240171B7B21B9EDE9@OS3PR01MB6275.jpnprd01.prod.outlook.com/2-v8-0001-Perform-streaming-logical-transactions-by-backgro.patch) download | inline diff: From 3e7322232c5248575a11d51dcc8c7a7c876006c6 Mon Sep 17 00:00:00 2001 From: "houzj.fnst" <[email protected]> Date: Wed, 20 Apr 2022 16:45:07 +0800 Subject: [PATCH v8 1/4] Perform streaming logical transactions by background workers Currently, for large transactions, the publisher sends the data in multiple streams (changes divided into chunks depending upon logical_decoding_work_mem), and then on the subscriber-side, the apply worker writes the changes into temporary files and once it receives the commit, it reads from the file and applies the entire transaction. To improve the performance of such transactions, we can instead allow them to be applied via background workers. In this approach, we assign a new apply background worker (if available) as soon as the xact's first stream is received and the main apply worker will send changes to this new worker via shared memory. The apply background worker will directly apply the change instead of writing it to temporary files. We keep this worker assigned till the transaction commit is received and also wait for the worker to finish at commit. This preserves commit ordering and avoids writing to and reading from file in most cases. We still need to spill if there is no worker available. We also need to allow stream_stop to complete by the apply background worker to avoid deadlocks because T-1's current stream of changes can update rows in conflicting order with T-2's next stream of changes. This patch also extends the SUBSCRIPTION 'streaming' option so that the user can control whether to apply the streaming transaction in an apply background worker or spill the change to disk. User can set the streaming option to 'on/off', 'apply'. For now, 'apply' means the streaming will be applied via a apply background worker if available. 'on' means the streaming transaction will be spilled to disk. --- doc/src/sgml/catalogs.sgml | 10 +- doc/src/sgml/logical-replication.sgml | 7 + doc/src/sgml/ref/create_subscription.sgml | 24 +- src/backend/commands/subscriptioncmds.c | 66 +- src/backend/postmaster/bgworker.c | 3 + src/backend/replication/logical/Makefile | 3 +- .../replication/logical/applybgwroker.c | 743 ++++++++++++++++++ src/backend/replication/logical/launcher.c | 87 +- src/backend/replication/logical/origin.c | 26 +- src/backend/replication/logical/tablesync.c | 10 +- src/backend/replication/logical/worker.c | 626 +++++++++++---- src/backend/utils/activity/wait_event.c | 3 + src/bin/pg_dump/pg_dump.c | 6 +- src/include/catalog/pg_subscription.h | 17 +- src/include/replication/logicalworker.h | 1 + src/include/replication/origin.h | 2 +- src/include/replication/worker_internal.h | 100 ++- src/include/utils/wait_event.h | 1 + src/test/regress/expected/subscription.out | 2 +- src/tools/pgindent/typedefs.list | 4 + 20 files changed, 1546 insertions(+), 195 deletions(-) create mode 100644 src/backend/replication/logical/applybgwroker.c diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index c00c93dd7b..5cb39b18bd 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -7873,11 +7873,15 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l <row> <entry role="catalog_table_entry"><para role="column_definition"> - <structfield>substream</structfield> <type>bool</type> + <structfield>substream</structfield> <type>char</type> </para> <para> - If true, the subscription will allow streaming of in-progress - transactions + Controls how to handle the streaming of in-progress transactions: + <literal>f</literal> = disallow streaming of in-progress transactions, + <literal>t</literal> = spill the changes of in-progress transactions to + disk and apply at once after the transaction is committed on the + publisher, + <literal>a</literal> = apply changes directly using a background worker </para></entry> </row> diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml index 145ea71d61..cca6c97d79 100644 --- a/doc/src/sgml/logical-replication.sgml +++ b/doc/src/sgml/logical-replication.sgml @@ -948,6 +948,13 @@ CONTEXT: processing remote data for replication origin "pg_16395" during "INSER might not violate any constraint. This can easily make the subscriber inconsistent. </para> + + <para> + Setting streaming mode to <literal>apply</literal> could export invalid LSN + as finish LSN of failed transaction. Changing the streaming mode and making + the same conflict writes the finish LSN of the failed transaction in the + server log if required. + </para> </sect1> <sect1 id="logical-replication-restrictions"> diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml index 35b39c28da..d4caae0222 100644 --- a/doc/src/sgml/ref/create_subscription.sgml +++ b/doc/src/sgml/ref/create_subscription.sgml @@ -217,13 +217,29 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl </varlistentry> <varlistentry> - <term><literal>streaming</literal> (<type>boolean</type>)</term> + <term><literal>streaming</literal> (<type>enum</type>)</term> <listitem> <para> Specifies whether to enable streaming of in-progress transactions - for this subscription. By default, all transactions - are fully decoded on the publisher and only then sent to the - subscriber as a whole. + for this subscription. The default value is <literal>off</literal>, + all transactions are fully decoded on the publisher and only then + sent to the subscriber as a whole. + </para> + + <para> + If set to <literal>on</literal>, the changes of transaction are + written to temporary files and then applied at once after the + transaction is committed on the publisher. + </para> + + <para> + If set to <literal>apply</literal> incoming + changes are directly applied via one of the background workers, if + available. If no background worker is free to handle streaming + transaction then the changes are written to a file and applied after + the transaction is committed. Note that if an error happens when + applying changes in a background worker, it might not report the + finish LSN of the remote transaction in the server log. </para> </listitem> </varlistentry> diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c index 83e6eae855..4080bba987 100644 --- a/src/backend/commands/subscriptioncmds.c +++ b/src/backend/commands/subscriptioncmds.c @@ -83,7 +83,7 @@ typedef struct SubOpts bool copy_data; bool refresh; bool binary; - bool streaming; + char streaming; bool twophase; bool disableonerr; XLogRecPtr lsn; @@ -95,6 +95,62 @@ static List *merge_publications(List *oldpublist, List *newpublist, bool addpub, static void ReportSlotConnectionError(List *rstates, Oid subid, char *slotname, char *err); +/* + * Extract the streaming mode value from a DefElem. This is like + * defGetBoolean() but also accepts the special value of "apply". + */ +static char +defGetStreamingMode(DefElem *def) +{ + /* + * If no parameter given, assume "true" is meant. + */ + if (def->arg == NULL) + return SUBSTREAM_ON; + + /* + * Allow 0, 1, "true", "false", "on", "off" or "apply". + */ + switch (nodeTag(def->arg)) + { + case T_Integer: + switch (intVal(def->arg)) + { + case 0: + return SUBSTREAM_OFF; + case 1: + return SUBSTREAM_ON; + default: + /* otherwise, error out below */ + break; + } + break; + default: + { + char *sval = defGetString(def); + + /* + * The set of strings accepted here should match up with the + * grammar's opt_boolean_or_string production. + */ + if (pg_strcasecmp(sval, "false") == 0 || + pg_strcasecmp(sval, "off") == 0) + return SUBSTREAM_OFF; + if (pg_strcasecmp(sval, "true") == 0 || + pg_strcasecmp(sval, "on") == 0) + return SUBSTREAM_ON; + if (pg_strcasecmp(sval, "apply") == 0) + return SUBSTREAM_APPLY; + } + break; + } + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s requires a Boolean value or \"apply\"", + def->defname))); + return SUBSTREAM_OFF; /* keep compiler quiet */ +} + /* * Common option parsing function for CREATE and ALTER SUBSCRIPTION commands. * @@ -132,7 +188,7 @@ parse_subscription_options(ParseState *pstate, List *stmt_options, if (IsSet(supported_opts, SUBOPT_BINARY)) opts->binary = false; if (IsSet(supported_opts, SUBOPT_STREAMING)) - opts->streaming = false; + opts->streaming = SUBSTREAM_OFF; if (IsSet(supported_opts, SUBOPT_TWOPHASE_COMMIT)) opts->twophase = false; if (IsSet(supported_opts, SUBOPT_DISABLE_ON_ERR)) @@ -233,7 +289,7 @@ parse_subscription_options(ParseState *pstate, List *stmt_options, errorConflictingDefElem(defel, pstate); opts->specified_opts |= SUBOPT_STREAMING; - opts->streaming = defGetBoolean(defel); + opts->streaming = defGetStreamingMode(defel); } else if (strcmp(defel->defname, "two_phase") == 0) { @@ -601,7 +657,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt, values[Anum_pg_subscription_subowner - 1] = ObjectIdGetDatum(owner); values[Anum_pg_subscription_subenabled - 1] = BoolGetDatum(opts.enabled); values[Anum_pg_subscription_subbinary - 1] = BoolGetDatum(opts.binary); - values[Anum_pg_subscription_substream - 1] = BoolGetDatum(opts.streaming); + values[Anum_pg_subscription_substream - 1] = CharGetDatum(opts.streaming); values[Anum_pg_subscription_subtwophasestate - 1] = CharGetDatum(opts.twophase ? LOGICALREP_TWOPHASE_STATE_PENDING : @@ -1060,7 +1116,7 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt, if (IsSet(opts.specified_opts, SUBOPT_STREAMING)) { values[Anum_pg_subscription_substream - 1] = - BoolGetDatum(opts.streaming); + CharGetDatum(opts.streaming); replaces[Anum_pg_subscription_substream - 1] = true; } diff --git a/src/backend/postmaster/bgworker.c b/src/backend/postmaster/bgworker.c index 40601aefd9..40ccb8993c 100644 --- a/src/backend/postmaster/bgworker.c +++ b/src/backend/postmaster/bgworker.c @@ -128,6 +128,9 @@ static const struct }, { "ApplyWorkerMain", ApplyWorkerMain + }, + { + "ApplyBgworkerMain", ApplyBgworkerMain } }; diff --git a/src/backend/replication/logical/Makefile b/src/backend/replication/logical/Makefile index c4e2fdeb71..a24a41969e 100644 --- a/src/backend/replication/logical/Makefile +++ b/src/backend/replication/logical/Makefile @@ -26,6 +26,7 @@ OBJS = \ reorderbuffer.o \ snapbuild.o \ tablesync.o \ - worker.o + worker.o \ + applybgwroker.o include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/replication/logical/applybgwroker.c b/src/backend/replication/logical/applybgwroker.c new file mode 100644 index 0000000000..9fcc600227 --- /dev/null +++ b/src/backend/replication/logical/applybgwroker.c @@ -0,0 +1,743 @@ +/*------------------------------------------------------------------------- + * applybgwroker.c + * Support routines for applying xact by apply background worker + * + * Copyright (c) 2016-2022, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/replication/logical/applybgwroker.c + * + * This file contains routines that are intended to support setting up, using, + * and tearing down a ApplyBgworkerState. + * Refer to the comments in file header of logical/worker.c to see more + * informations about apply background worker. + * + *------------------------------------------------------------------------- + */ + +#include "postgres.h" + +#include "libpq/pqformat.h" +#include "mb/pg_wchar.h" +#include "pgstat.h" +#include "postmaster/interrupt.h" +#include "replication/logicalworker.h" +#include "replication/origin.h" +#include "replication/worker_internal.h" +#include "storage/ipc.h" +#include "storage/procarray.h" +#include "tcop/tcopprot.h" +#include "utils/inval.h" +#include "utils/memutils.h" +#include "utils/resowner.h" +#include "utils/syscache.h" + +#define PG_LOGICAL_APPLY_SHM_MAGIC 0x79fb2447 // TODO Consider change + +/* + * DSM keys for apply background worker. Unlike other parallel execution code, + * since we don't need to worry about DSM keys conflicting with plan_node_id we + * can use small integers. + */ +#define APPLY_BGWORKER_KEY_SHARED 1 +#define APPLY_BGWORKER_KEY_MQ 2 + +/* + * entry for a hash table we use to map from xid to our apply background worker + * state. + */ +typedef struct ApplyBgworkerEntry +{ + TransactionId xid; + ApplyBgworkerState *wstate; +} ApplyBgworkerEntry; + +/* Apply background workers hash table (initialized on first use) */ +static HTAB *ApplyWorkersHash = NULL; +static List *ApplyWorkersFreeList = NIL; +static List *ApplyWorkersList = NIL; + +/* + * Fields to record the share informations between main apply worker and apply + * background worker. + */ +volatile ApplyBgworkerShared *MyParallelState = NULL; + +List *subxactlist = NIL; + +/* apply background worker setup */ +static ApplyBgworkerState *apply_bgworker_setup(void); +static void apply_bgworker_setup_dsm(ApplyBgworkerState *wstate); + +/* + * Look up worker inside ApplyWorkersHash for requested xid. + * + * If start flag is true, try to start a new worker if not found in hash table. + */ +ApplyBgworkerState * +apply_bgworker_find_or_start(TransactionId xid, bool start) +{ + bool found; + ApplyBgworkerState *wstate; + ApplyBgworkerEntry *entry = NULL; + + if (!TransactionIdIsValid(xid)) + return NULL; + + /* + * We don't start new background worker if we are not in streaming apply + * mode. + */ + if (MySubscription->stream != SUBSTREAM_APPLY) + return NULL; + + /* + * We don't start new background worker if user has set skiplsn as it's + * possible that user want to skip the streaming transaction. For + * streaming transaction, we need to spill the transaction to disk so that + * we can get the last LSN of the transaction to judge whether to skip + * before starting to apply the change. + */ + if (start && !XLogRecPtrIsInvalid(MySubscription->skiplsn)) + return NULL; + + /* + * For streaming transactions that are being applied in apply background + * worker, we cannot decide whether to apply the change for a relation + * that is not in the READY state (see should_apply_changes_for_rel) as we + * won't know remote_final_lsn by that time. So, we don't start new apply + * background worker in this case. + */ + if (start && !AllTablesyncsReady()) + return NULL; + + /* First time through, initialize apply workers hashtable */ + if (ApplyWorkersHash == NULL) + { + HASHCTL ctl; + + MemSet(&ctl, 0, sizeof(ctl)); + ctl.keysize = sizeof(TransactionId); + ctl.entrysize = sizeof(ApplyBgworkerEntry); + ctl.hcxt = ApplyContext; + + ApplyWorkersHash = hash_create("logical apply workers hash", 8, &ctl, + HASH_ELEM | HASH_BLOBS | HASH_CONTEXT); + } + + /* + * Find entry for requested transaction. + */ + entry = hash_search(ApplyWorkersHash, &xid, start ? HASH_ENTER : HASH_FIND, + &found); + if (found) + { + entry->wstate->pstate->status = APPLY_BGWORKER_BUSY; + return entry->wstate; + } + else if (!start) + return NULL; + + /* + * Now, we try to get a apply background worker. If there is at least one + * worker in the idle list, then take one. Otherwise, we try to start a + * new apply background worker. + */ + if (list_length(ApplyWorkersFreeList) > 0) + { + wstate = (ApplyBgworkerState *) llast(ApplyWorkersFreeList); + ApplyWorkersFreeList = list_delete_last(ApplyWorkersFreeList); + } + else + { + wstate = apply_bgworker_setup(); + + if (wstate == NULL) + { + /* + * If the apply background worker cannot be launched, remove entry + * in hash table. + */ + hash_search(ApplyWorkersHash, &xid, HASH_REMOVE, &found); + return NULL; + } + } + + /* Fill up the hash entry */ + wstate->pstate->status = APPLY_BGWORKER_BUSY; + wstate->pstate->stream_xid = xid; + entry->wstate = wstate; + entry->xid = xid; + + return wstate; +} + +/* + * Add the worker to the free list and remove the entry from hash table. + */ +void +apply_bgworker_free(ApplyBgworkerState *wstate) +{ + MemoryContext oldctx; + TransactionId xid = wstate->pstate->stream_xid; + + Assert(wstate->pstate->status == APPLY_BGWORKER_FINISHED); + + oldctx = MemoryContextSwitchTo(ApplyContext); + + hash_search(ApplyWorkersHash, &xid, HASH_REMOVE, NULL); + + elog(DEBUG1, "adding finished apply worker #%u for xid %u to the idle list", + wstate->pstate->n, wstate->pstate->stream_xid); + + ApplyWorkersFreeList = lappend(ApplyWorkersFreeList, wstate); + + MemoryContextSwitchTo(oldctx); +} + +/* Apply Background Worker main loop */ +static void +LogicalApplyBgwLoop(shm_mq_handle *mqh, volatile ApplyBgworkerShared *pst) +{ + shm_mq_result shmq_res; + PGPROC *registrant; + ErrorContextCallback errcallback; + XLogRecPtr last_received = InvalidXLogRecPtr; + + registrant = BackendPidGetProc(MyBgworkerEntry->bgw_notify_pid); + SetLatch(®istrant->procLatch); + + /* + * Push apply error context callback. Fields will be filled during + * applying a change. + */ + errcallback.callback = apply_error_callback; + errcallback.previous = error_context_stack; + error_context_stack = &errcallback; + + for (;;) + { + void *data; + Size len; + int c; + StringInfoData s; + MemoryContext oldctx; + XLogRecPtr start_lsn; + XLogRecPtr end_lsn; + TimestampTz send_time; + + CHECK_FOR_INTERRUPTS(); + + /* Ensure we are reading the data into our memory context. */ + oldctx = MemoryContextSwitchTo(ApplyMessageContext); + + shmq_res = shm_mq_receive(mqh, &len, &data, false); + + if (shmq_res != SHM_MQ_SUCCESS) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("lost connection to the main apply worker"))); + + if (len == 0) + break; + + s.cursor = 0; + s.maxlen = -1; + s.data = (char *) data; + s.len = len; + + /* + * We use first byte of message for additional communication between + * main Logical replication worker and apply bgworkers, so if it + * differs from 'w', then process it first. + */ + c = pq_getmsgbyte(&s); + switch (c) + { + /* End message of streaming chunk */ + case LOGICAL_REP_MSG_STREAM_STOP: + elog(DEBUG1, "[Apply BGW #%u] ended processing streaming chunk," + "waiting on shm_mq_receive", pst->n); + + apply_bgworker_set_status(APPLY_BGWORKER_READY); + + SetLatch(®istrant->procLatch); + + in_streamed_transaction = false; + pgstat_report_activity(STATE_IDLE, NULL); + continue; + case 'w': + break; + default: + elog(ERROR, "[Apply BGW #%u] unexpected message \"%c\"", + pst->n, c); + break; + } + + start_lsn = pq_getmsgint64(&s); + end_lsn = pq_getmsgint64(&s); + send_time = pq_getmsgint64(&s); + + if (last_received < start_lsn) + last_received = start_lsn; + + if (last_received < end_lsn) + last_received = end_lsn; + + /* + * TO IMPROVE: Do we need to display the apply background worker's + * information in pg_stat_replication ? + */ + UpdateWorkerStats(last_received, send_time, false); + + apply_dispatch(&s); + + if (ConfigReloadPending) + { + ConfigReloadPending = false; + ProcessConfigFile(PGC_SIGHUP); + } + + MemoryContextSwitchTo(oldctx); + MemoryContextReset(ApplyMessageContext); + } + + MemoryContextSwitchTo(TopMemoryContext); + MemoryContextReset(ApplyContext); + + /* Pop the error context stack */ + error_context_stack = errcallback.previous; + + elog(DEBUG1, "[Apply BGW #%u] exiting", pst->n); + + /* Signal main process that we are done. */ + SetLatch(®istrant->procLatch); +} + +/* + * Set the exit status so that the main apply worker can realize we have + * shutdown. + */ +static void +ApplyBgwShutdown(int code, Datum arg) +{ + SpinLockAcquire(&MyParallelState->mutex); + MyParallelState->status = APPLY_BGWORKER_EXIT; + SpinLockRelease(&MyParallelState->mutex); + + dsm_detach((dsm_segment *) DatumGetPointer(arg)); +} + +/* + * Apply Background Worker entry point + */ +void +ApplyBgworkerMain(Datum main_arg) +{ + volatile ApplyBgworkerShared *pst; + + dsm_handle handle; + dsm_segment *seg; + shm_toc *toc; + shm_mq *mq; + shm_mq_handle *mqh; + MemoryContext oldcontext; + RepOriginId originid; + int worker_slot = DatumGetInt32(main_arg); + char originname[NAMEDATALEN]; + + /* Attach to slot */ + logicalrep_worker_attach(worker_slot); + + MemoryContextSwitchTo(TopMemoryContext); + + /* Load the subscription into persistent memory context. */ + ApplyContext = AllocSetContextCreate(TopMemoryContext, + "ApplyContext", + ALLOCSET_DEFAULT_SIZES); + + /* + * Init the ApplyMessageContext which we clean up after each replication + * protocol message. + */ + ApplyMessageContext = AllocSetContextCreate(ApplyContext, + "ApplyMessageContext", + ALLOCSET_DEFAULT_SIZES); + + /* Setup signal handling */ + pqsignal(SIGHUP, SignalHandlerForConfigReload); + pqsignal(SIGTERM, die); + BackgroundWorkerUnblockSignals(); + + /* + * Connect to the dynamic shared memory segment. + * + * The backend that registered this worker passed us the ID of a shared + * memory segment to which we must attach for further instructions. In + * order to attach to dynamic shared memory, we need a resource owner. + * Once we've mapped the segment in our address space, attach to the table + * of contents so we can locate the various data structures we'll need to + * find within the segment. + */ + CurrentResourceOwner = ResourceOwnerCreate(NULL, "Logical apply worker"); + memcpy(&handle, MyBgworkerEntry->bgw_extra, sizeof(dsm_handle)); + seg = dsm_attach(handle); + if (seg == NULL) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("unable to map dynamic shared memory segment"))); + toc = shm_toc_attach(PG_LOGICAL_APPLY_SHM_MAGIC, dsm_segment_address(seg)); + if (toc == NULL) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("bad magic number in dynamic shared memory segment"))); + + before_shmem_exit(ApplyBgwShutdown, PointerGetDatum(seg)); + + /* Look up the parallel state. */ + pst = shm_toc_lookup(toc, APPLY_BGWORKER_KEY_SHARED, false); + MyParallelState = pst; + + /* + * Attach to the message queue. + */ + mq = shm_toc_lookup(toc, APPLY_BGWORKER_KEY_MQ, false); + shm_mq_set_receiver(mq, MyProc); + mqh = shm_mq_attach(mq, seg, NULL); + + /* Connect to our database. */ + BackgroundWorkerInitializeConnectionByOid(MyLogicalRepWorker->dbid, + MyLogicalRepWorker->userid, + 0); + + /* + * Set the client encoding to the database encoding, since that is what + * the leader will expect. + */ + SetClientEncoding(GetDatabaseEncoding()); + + stream_xid = pst->stream_xid; + + StartTransactionCommand(); + oldcontext = MemoryContextSwitchTo(ApplyContext); + + MySubscription = GetSubscription(MyLogicalRepWorker->subid, true); + if (!MySubscription) + { + ereport(LOG, + (errmsg("logical replication apply worker for subscription %u will not " + "start because the subscription was removed during startup", + MyLogicalRepWorker->subid))); + proc_exit(0); + } + + MySubscriptionValid = true; + MemoryContextSwitchTo(oldcontext); + + /* Setup synchronous commit according to the user's wishes */ + SetConfigOption("synchronous_commit", MySubscription->synccommit, + PGC_BACKEND, PGC_S_OVERRIDE); + + /* Keep us informed about subscription changes. */ + CacheRegisterSyscacheCallback(SUBSCRIPTIONOID, + subscription_change_cb, + (Datum) 0); + + CommitTransactionCommand(); + + /* Setup replication origin tracking. */ + StartTransactionCommand(); + snprintf(originname, sizeof(originname), "pg_%u", MySubscription->oid); + originid = replorigin_by_name(originname, true); + if (!OidIsValid(originid)) + originid = replorigin_create(originname); + + /* + * The apply background worker doesn't need to monopolize this replication + * origin which was already acquired by its leader process. + */ + replorigin_session_setup(originid, false); + replorigin_session_origin = originid; + CommitTransactionCommand(); + + /* + * Allocate the origin name in long-lived context for error context + * message. + */ + apply_error_callback_arg.origin_name = MemoryContextStrdup(ApplyContext, + originname); + + /* + * Indicate that we're fully initialized and ready to begin the main part + * of the apply operation. + */ + apply_bgworker_set_status(APPLY_BGWORKER_ATTACHED); + + elog(DEBUG1, "[Apply BGW #%u] started", pst->n); + + LogicalApplyBgwLoop(mqh, pst); + + /* + * We're done. Explicitly detach the shared memory segment so that we + * don't get a resource leak warning at commit time. This will fire any + * on_dsm_detach callbacks we've registered, as well. Once that's done, + * we can go ahead and exit. + */ + dsm_detach(seg); + proc_exit(0); +} + +/* + * Set up a dynamic shared memory segment. + * + * We set up a control region that contains a ApplyBgworkerShared, + * plus one region per message queue. There are as many message queues as + * the number of workers. + */ +static void +apply_bgworker_setup_dsm(ApplyBgworkerState *wstate) +{ + shm_toc_estimator e; + Size segsize; + dsm_segment *seg; + shm_toc *toc; + ApplyBgworkerShared *pst; + shm_mq *mq; + int64 queue_size = 160000000; /* 16 MB for now */ + + /* + * Estimate how much shared memory we need. + * + * Because the TOC machinery may choose to insert padding of oddly-sized + * requests, we must estimate each chunk separately. + * + * We need one key to register the location of the header, and we need + * another key to track the location of the message queue. + */ + shm_toc_initialize_estimator(&e); + shm_toc_estimate_chunk(&e, sizeof(ApplyBgworkerShared)); + shm_toc_estimate_chunk(&e, (Size) queue_size); + + shm_toc_estimate_keys(&e, 1 + 1); + segsize = shm_toc_estimate(&e); + + /* Create the shared memory segment and establish a table of contents. */ + seg = dsm_create(shm_toc_estimate(&e), 0); + toc = shm_toc_create(PG_LOGICAL_APPLY_SHM_MAGIC, dsm_segment_address(seg), + segsize); + + /* Set up the header region. */ + pst = shm_toc_allocate(toc, sizeof(ApplyBgworkerShared)); + SpinLockInit(&pst->mutex); + pst->status = APPLY_BGWORKER_BUSY; + pst->stream_xid = stream_xid; + pst->n = list_length(ApplyWorkersList) + 1; + + shm_toc_insert(toc, APPLY_BGWORKER_KEY_SHARED, pst); + + /* Set up one message queue per worker, plus one. */ + mq = shm_mq_create(shm_toc_allocate(toc, (Size) queue_size), + (Size) queue_size); + shm_toc_insert(toc, APPLY_BGWORKER_KEY_MQ, mq); + shm_mq_set_sender(mq, MyProc); + + /* Attach the queues. */ + wstate->mq_handle = shm_mq_attach(mq, seg, NULL); + + /* Return results to caller. */ + wstate->dsm_seg = seg; + wstate->pstate = pst; +} + +/* + * Start apply background worker process and allocate shared memory for it. + */ +static ApplyBgworkerState * +apply_bgworker_setup(void) +{ + MemoryContext oldcontext; + bool launched; + ApplyBgworkerState *wstate; + + elog(DEBUG1, "setting up apply worker #%u", list_length(ApplyWorkersList) + 1); + + oldcontext = MemoryContextSwitchTo(ApplyContext); + + wstate = (ApplyBgworkerState *) palloc0(sizeof(ApplyBgworkerState)); + + /* Setup shared memory */ + apply_bgworker_setup_dsm(wstate); + + launched = logicalrep_worker_launch(MyLogicalRepWorker->dbid, + MySubscription->oid, + MySubscription->name, + MyLogicalRepWorker->userid, + InvalidOid, + dsm_segment_handle(wstate->dsm_seg)); + + if (launched) + { + /* Wait for worker to attach. */ + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_ATTACHED); + + ApplyWorkersList = lappend(ApplyWorkersList, wstate); + } + else + { + pfree(wstate); + wstate = NULL; + } + + MemoryContextSwitchTo(oldcontext); + + return wstate; +} + +/* + * Send the data to the specified apply background worker via shared-memory queue. + */ +void +apply_bgworker_send_data(ApplyBgworkerState *wstate, Size nbytes, const void *data) +{ + shm_mq_result result; + + result = shm_mq_send(wstate->mq_handle, nbytes, data, false, true); + + if (result != SHM_MQ_SUCCESS) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("could not send tuples to shared-memory queue"))); +} + +/* + * Wait until the status of apply background worker reaches the + * 'wait_for_status' + */ +void +apply_bgworker_wait_for(ApplyBgworkerState *wstate, + ApplyBgworkerStatus wait_for_status) +{ + for (;;) + { + char status; + + SpinLockAcquire(&wstate->pstate->mutex); + status = wstate->pstate->status; + SpinLockRelease(&wstate->pstate->mutex); + + /* Done if already in correct status. */ + if (status == wait_for_status) + break; + + /* If any workers (or the postmaster) have died, we have failed. */ + if (status == APPLY_BGWORKER_EXIT) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("background worker %u failed to apply transaction %u", + wstate->pstate->n, wstate->pstate->stream_xid))); + + /* Wait to be signalled. */ + WaitLatch(MyLatch, WL_LATCH_SET | WL_EXIT_ON_PM_DEATH, 0, + WAIT_EVENT_LOGICAL_APPLY_WORKER_STATE_CHANGE); + + /* Reset the latch so we don't spin. */ + ResetLatch(MyLatch); + + /* An interrupt may have occurred while we were waiting. */ + CHECK_FOR_INTERRUPTS(); + } +} + +/* + * Check the status of workers and report an error if any apply background + * worker has exited unexpectedly. + * + * Exit if any relation is not in the READY state and if any worker is handling + * the streaming transaction at the same time. Because for streaming + * transactions that is being applied in apply background worker, we cannot + * decide whether to apply the change for a relation that is not in the READY + * state (see should_apply_changes_for_rel) as we won't know remote_final_lsn + * by that time. + */ +void +apply_bgworker_check_status(void) +{ + ListCell *lc; + + if (am_apply_bgworker() || MySubscription->stream != SUBSTREAM_APPLY) + return; + + foreach(lc, ApplyWorkersList) + { + ApplyBgworkerState *wstate = (ApplyBgworkerState *) lfirst(lc); + + /* + * We don't lock here as in the worst case we will just detect the + * failure of worker a bit later. + */ + if (wstate->pstate->status == APPLY_BGWORKER_EXIT) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("background worker %u exited unexpectedly", + wstate->pstate->n))); + } + + if (list_length(ApplyWorkersFreeList) != list_length(ApplyWorkersList) && + !AllTablesyncsReady()) + { + ereport(LOG, + (errmsg("logical replication apply workers for subscription \"%s\" will restart", + MySubscription->name), + errdetail("Cannot handle streamed replication transaction by apply " + "bgworkers until all tables are synchronized"))); + + proc_exit(0); + } +} + +/* Set the apply background worker status */ +void +apply_bgworker_set_status(ApplyBgworkerStatus status) +{ + if (!am_apply_bgworker()) + return; + + elog(DEBUG1, "[Apply BGW #%u] set status to %d", MyParallelState->n, status); + + SpinLockAcquire(&MyParallelState->mutex); + MyParallelState->status = status; + SpinLockRelease(&MyParallelState->mutex); +} + +/* + * Define a savepoint for a subxact in apply background worker if needed. + * + * Inside apply background worker we can figure out that new subtransaction was + * started if new change arrived with different xid. In that case we can define + * named savepoint, so that we were able to commit/rollback it separately + * later. + * Special case is if the first change comes from subtransaction, then + * we check that current_xid differs from stream_xid. + */ +void +apply_bgworker_subxact_info_add(TransactionId current_xid) +{ + if (current_xid != stream_xid && + !list_member_int(subxactlist, (int) current_xid)) + { + MemoryContext oldctx; + char spname[MAXPGPATH]; + + snprintf(spname, MAXPGPATH, "savepoint_for_xid_%u", current_xid); + + elog(DEBUG1, "[Apply BGW #%u] defining savepoint %s", + MyParallelState->n, spname); + + DefineSavepoint(spname); + CommitTransactionCommand(); + + oldctx = MemoryContextSwitchTo(ApplyContext); + subxactlist = lappend_int(subxactlist, (int) current_xid); + MemoryContextSwitchTo(oldctx); + } +} diff --git a/src/backend/replication/logical/launcher.c b/src/backend/replication/logical/launcher.c index bd5f78cf9a..e810875ddd 100644 --- a/src/backend/replication/logical/launcher.c +++ b/src/backend/replication/logical/launcher.c @@ -73,6 +73,7 @@ static void logicalrep_launcher_onexit(int code, Datum arg); static void logicalrep_worker_onexit(int code, Datum arg); static void logicalrep_worker_detach(void); static void logicalrep_worker_cleanup(LogicalRepWorker *worker); +static void logicalrep_worker_stop_internal(LogicalRepWorker *worker); static bool on_commit_launcher_wakeup = false; @@ -223,6 +224,10 @@ logicalrep_worker_find(Oid subid, Oid relid, bool only_running) { LogicalRepWorker *w = &LogicalRepCtx->workers[i]; + /* We only need main apply worker or table sync worker here */ + if (w->subworker) + continue; + if (w->in_use && w->subid == subid && w->relid == relid && (!only_running || w->proc)) { @@ -259,11 +264,11 @@ logicalrep_workers_find(Oid subid, bool only_running) } /* - * Start new apply background worker, if possible. + * Start new background worker, if possible. */ -void +bool logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, Oid userid, - Oid relid) + Oid relid, dsm_handle subworker_dsm) { BackgroundWorker bgw; BackgroundWorkerHandle *bgw_handle; @@ -273,6 +278,10 @@ logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, Oid userid, LogicalRepWorker *worker = NULL; int nsyncworkers; TimestampTz now; + bool is_subworker = (subworker_dsm != DSM_HANDLE_INVALID); + + /* We don't support table sync in subworker */ + Assert(!(is_subworker && OidIsValid(relid))); ereport(DEBUG1, (errmsg_internal("starting logical replication worker for subscription \"%s\"", @@ -350,7 +359,7 @@ retry: if (OidIsValid(relid) && nsyncworkers >= max_sync_workers_per_subscription) { LWLockRelease(LogicalRepWorkerLock); - return; + return false; } /* @@ -364,7 +373,7 @@ retry: (errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED), errmsg("out of logical replication worker slots"), errhint("You might need to increase max_logical_replication_workers."))); - return; + return false; } /* Prepare the worker slot. */ @@ -379,6 +388,7 @@ retry: worker->relstate = SUBREL_STATE_UNKNOWN; worker->relstate_lsn = InvalidXLogRecPtr; worker->stream_fileset = NULL; + worker->subworker = is_subworker; worker->last_lsn = InvalidXLogRecPtr; TIMESTAMP_NOBEGIN(worker->last_send_time); TIMESTAMP_NOBEGIN(worker->last_recv_time); @@ -396,19 +406,30 @@ retry: BGWORKER_BACKEND_DATABASE_CONNECTION; bgw.bgw_start_time = BgWorkerStart_RecoveryFinished; snprintf(bgw.bgw_library_name, BGW_MAXLEN, "postgres"); - snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyWorkerMain"); + + if (!is_subworker) + snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyWorkerMain"); + else + snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyBgworkerMain"); + if (OidIsValid(relid)) snprintf(bgw.bgw_name, BGW_MAXLEN, "logical replication worker for subscription %u sync %u", subid, relid); - else + else if (!is_subworker) snprintf(bgw.bgw_name, BGW_MAXLEN, "logical replication worker for subscription %u", subid); + else + snprintf(bgw.bgw_name, BGW_MAXLEN, + "logical replication background apply worker for subscription %u ", subid); snprintf(bgw.bgw_type, BGW_MAXLEN, "logical replication worker"); bgw.bgw_restart_time = BGW_NEVER_RESTART; bgw.bgw_notify_pid = MyProcPid; bgw.bgw_main_arg = Int32GetDatum(slot); + if (is_subworker) + memcpy(bgw.bgw_extra, &subworker_dsm, sizeof(dsm_handle)); + if (!RegisterDynamicBackgroundWorker(&bgw, &bgw_handle)) { /* Failed to start worker, so clean up the worker slot. */ @@ -421,11 +442,13 @@ retry: (errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED), errmsg("out of background worker slots"), errhint("You might need to increase max_worker_processes."))); - return; + return false; } /* Now wait until it attaches. */ WaitForReplicationWorkerAttach(worker, generation, bgw_handle); + + return true; } /* @@ -436,7 +459,6 @@ void logicalrep_worker_stop(Oid subid, Oid relid) { LogicalRepWorker *worker; - uint16 generation; LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); @@ -449,6 +471,22 @@ logicalrep_worker_stop(Oid subid, Oid relid) return; } + logicalrep_worker_stop_internal(worker); + + LWLockRelease(LogicalRepWorkerLock); +} + +/* + * Workhorse for logicalrep_worker_stop() and logicalrep_worker_detach(). Stop + * the worker and wait for it to die. + */ +static void +logicalrep_worker_stop_internal(LogicalRepWorker *worker) +{ + uint16 generation; + + Assert(LWLockHeldByMe(LogicalRepWorkerLock)); + /* * Remember which generation was our worker so we can check if what we see * is still the same one. @@ -485,10 +523,7 @@ logicalrep_worker_stop(Oid subid, Oid relid) * different, meaning that a different worker has taken the slot. */ if (!worker->in_use || worker->generation != generation) - { - LWLockRelease(LogicalRepWorkerLock); return; - } /* Worker has assigned proc, so it has started. */ if (worker->proc) @@ -522,8 +557,6 @@ logicalrep_worker_stop(Oid subid, Oid relid) LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); } - - LWLockRelease(LogicalRepWorkerLock); } /* @@ -599,6 +632,29 @@ logicalrep_worker_attach(int slot) static void logicalrep_worker_detach(void) { + /* + * If we are the main apply worker, stop all the apply background workers + * we started before. + */ + if (!MyLogicalRepWorker->subworker) + { + List *workers; + ListCell *lc; + + LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); + + workers = logicalrep_workers_find(MyLogicalRepWorker->subid, true); + foreach(lc, workers) + { + LogicalRepWorker *w = (LogicalRepWorker *) lfirst(lc); + + if (w->subworker) + logicalrep_worker_stop_internal(w); + } + + LWLockRelease(LogicalRepWorkerLock); + } + /* Block concurrent access. */ LWLockAcquire(LogicalRepWorkerLock, LW_EXCLUSIVE); @@ -621,6 +677,7 @@ logicalrep_worker_cleanup(LogicalRepWorker *worker) worker->userid = InvalidOid; worker->subid = InvalidOid; worker->relid = InvalidOid; + worker->subworker = false; } /* @@ -868,7 +925,7 @@ ApplyLauncherMain(Datum main_arg) wait_time = wal_retrieve_retry_interval; logicalrep_worker_launch(sub->dbid, sub->oid, sub->name, - sub->owner, InvalidOid); + sub->owner, InvalidOid, DSM_HANDLE_INVALID); } } diff --git a/src/backend/replication/logical/origin.c b/src/backend/replication/logical/origin.c index 21937ab2d3..9273011b0a 100644 --- a/src/backend/replication/logical/origin.c +++ b/src/backend/replication/logical/origin.c @@ -1063,12 +1063,21 @@ ReplicationOriginExitCleanup(int code, Datum arg) * array doesn't have to be searched when calling * replorigin_session_advance(). * - * Obviously only one such cached origin can exist per process and the current + * Normally only one such cached origin can exist per process and the current * cached value can only be set again after the previous value is torn down * with replorigin_session_reset(). + * + * However, If must_acquire is false, we allow process to get the slot which is + * already acquired by other process. It's safe because 1) The only caller + * (apply background workers) will maintain the commit order by allowing only + * one process to commit at a time, so no two workers will be operating on the + * same origin at the same time (see comments in logical/worker.c). 2) Even + * though we try to advance the session's origin concurrently, it's safe to do + * so as we change/advance the session_origin LSNs under replicate_state + * LWLock. */ void -replorigin_session_setup(RepOriginId node) +replorigin_session_setup(RepOriginId node, bool must_acquire) { static bool registered_cleanup; int i; @@ -1110,7 +1119,7 @@ replorigin_session_setup(RepOriginId node) if (curstate->roident != node) continue; - else if (curstate->acquired_by != 0) + else if (curstate->acquired_by != 0 && must_acquire) { ereport(ERROR, (errcode(ERRCODE_OBJECT_IN_USE), @@ -1141,7 +1150,14 @@ replorigin_session_setup(RepOriginId node) Assert(session_replication_state->roident != InvalidRepOriginId); - session_replication_state->acquired_by = MyProcPid; + if (must_acquire) + session_replication_state->acquired_by = MyProcPid; + else if (session_replication_state->acquired_by == 0) + ereport(ERROR, + (errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED), + errmsg("could not find correct replication state slot for replication origin with OID %u for apply background worker", + node), + errhint("There is no replication state slot set by its main apply worker."))); LWLockRelease(ReplicationOriginLock); @@ -1321,7 +1337,7 @@ pg_replication_origin_session_setup(PG_FUNCTION_ARGS) name = text_to_cstring((text *) DatumGetPointer(PG_GETARG_DATUM(0))); origin = replorigin_by_name(name, false); - replorigin_session_setup(origin); + replorigin_session_setup(origin, true); replorigin_session_origin = origin; diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c index 670c6fcada..8ffba7e2e5 100644 --- a/src/backend/replication/logical/tablesync.c +++ b/src/backend/replication/logical/tablesync.c @@ -568,7 +568,8 @@ process_syncing_tables_for_apply(XLogRecPtr current_lsn) MySubscription->oid, MySubscription->name, MyLogicalRepWorker->userid, - rstate->relid); + rstate->relid, + DSM_HANDLE_INVALID); hentry->last_start_time = now; } } @@ -589,6 +590,9 @@ process_syncing_tables_for_apply(XLogRecPtr current_lsn) void process_syncing_tables(XLogRecPtr current_lsn) { + if (MyLogicalRepWorker->subworker) + return; + if (am_tablesync_worker()) process_syncing_tables_for_sync(current_lsn); else @@ -1273,7 +1277,7 @@ LogicalRepSyncTableStart(XLogRecPtr *origin_startpos) * time this tablesync was launched. */ originid = replorigin_by_name(originname, false); - replorigin_session_setup(originid); + replorigin_session_setup(originid, true); replorigin_session_origin = originid; *origin_startpos = replorigin_session_get_progress(false); @@ -1384,7 +1388,7 @@ LogicalRepSyncTableStart(XLogRecPtr *origin_startpos) true /* go backward */ , true /* WAL log */ ); UnlockRelationOid(ReplicationOriginRelationId, RowExclusiveLock); - replorigin_session_setup(originid); + replorigin_session_setup(originid, true); replorigin_session_origin = originid; } else diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c index fc210a9e7b..20c7865275 100644 --- a/src/backend/replication/logical/worker.c +++ b/src/backend/replication/logical/worker.c @@ -22,8 +22,25 @@ * STREAMED TRANSACTIONS * --------------------- * Streamed transactions (large transactions exceeding a memory limit on the - * upstream) are not applied immediately, but instead, the data is written - * to temporary files and then applied at once when the final commit arrives. + * upstream) are applied via one of two approaches. + * + * 1) Separate background workers + * + * Assign a new apply background worker (if available) as soon as the xact's + * first stream is received and the main apply worker will send changes to this + * new worker via shared memory. We keep this worker assigned till the + * transaction commit is received and also wait for the worker to finish at + * commit. This preserves commit ordering and avoids writing to and reading + * from file in most cases. We still need to spill if there is no worker + * available. We also need to allow stream_stop to complete by the background + * worker to avoid deadlocks because T-1's current stream of changes can update + * rows in conflicting order with T-2's next stream of changes. + * + * 2) Write to temporary files and apply when the final commit arrives + * + * If no worker is available to handle streamed transaction, the data is + * written to temporary files and then applied at once when the final commit + * arrives. * * Unlike the regular (non-streamed) case, handling streamed transactions has * to handle aborts of both the toplevel transaction and subtransactions. This @@ -219,20 +236,7 @@ typedef struct ApplyExecutionData PartitionTupleRouting *proute; /* partition routing info */ } ApplyExecutionData; -/* Struct for saving and restoring apply errcontext information */ -typedef struct ApplyErrorCallbackArg -{ - LogicalRepMsgType command; /* 0 if invalid */ - LogicalRepRelMapEntry *rel; - - /* Remote node information */ - int remote_attnum; /* -1 if invalid */ - TransactionId remote_xid; - XLogRecPtr finish_lsn; - char *origin_name; -} ApplyErrorCallbackArg; - -static ApplyErrorCallbackArg apply_error_callback_arg = +ApplyErrorCallbackArg apply_error_callback_arg = { .command = 0, .rel = NULL, @@ -242,7 +246,7 @@ static ApplyErrorCallbackArg apply_error_callback_arg = .origin_name = NULL, }; -static MemoryContext ApplyMessageContext = NULL; +MemoryContext ApplyMessageContext = NULL; MemoryContext ApplyContext = NULL; /* per stream context for streaming transactions */ @@ -251,27 +255,38 @@ static MemoryContext LogicalStreamingContext = NULL; WalReceiverConn *LogRepWorkerWalRcvConn = NULL; Subscription *MySubscription = NULL; -static bool MySubscriptionValid = false; +bool MySubscriptionValid = false; bool in_remote_transaction = false; static XLogRecPtr remote_final_lsn = InvalidXLogRecPtr; /* fields valid only when processing streamed transaction */ -static bool in_streamed_transaction = false; +bool in_streamed_transaction = false; -static TransactionId stream_xid = InvalidTransactionId; +TransactionId stream_xid = InvalidTransactionId; +static ApplyBgworkerState *stream_apply_worker = NULL; + +/* check if we are applying the transaction in apply background worker */ +#define apply_bgworker_active() (in_streamed_transaction && stream_apply_worker != NULL) + +/* + * The number of changes during one streaming block (only for apply background + * workers) + */ +static uint32 nchanges = 0; /* * We enable skipping all data modification changes (INSERT, UPDATE, etc.) for * the subscription if the remote transaction's finish LSN matches the subskiplsn. * Once we start skipping changes, we don't stop it until we skip all changes of * the transaction even if pg_subscription is updated and MySubscription->skiplsn - * gets changed or reset during that. Also, in streaming transaction cases, we - * don't skip receiving and spooling the changes since we decide whether or not + * gets changed or reset during that. Also, in streaming transaction cases (streaming = on), + * we don't skip receiving and spooling the changes since we decide whether or not * to skip applying the changes when starting to apply changes. The subskiplsn is * cleared after successfully skipping the transaction or applying non-empty * transaction. The latter prevents the mistakenly specified subskiplsn from - * being left. + * being left. Note that we cannot skip the streaming transaction in apply mode, + * because we cannot get the finish LSN before applying the changes. */ static XLogRecPtr skip_xact_finish_lsn = InvalidXLogRecPtr; #define is_skipping_changes() (unlikely(!XLogRecPtrIsInvalid(skip_xact_finish_lsn))) @@ -324,9 +339,6 @@ static void maybe_reread_subscription(void); static void DisableSubscriptionAndExit(void); -/* prototype needed because of stream_commit */ -static void apply_dispatch(StringInfo s); - static void apply_handle_commit_internal(LogicalRepCommitData *commit_data); static void apply_handle_insert_internal(ApplyExecutionData *edata, ResultRelInfo *relinfo, @@ -359,7 +371,6 @@ static void stop_skipping_changes(void); static void clear_subscription_skip_lsn(XLogRecPtr finish_lsn); /* Functions for apply error callback */ -static void apply_error_callback(void *arg); static inline void set_apply_error_context_xact(TransactionId xid, XLogRecPtr lsn); static inline void reset_apply_error_context_info(void); @@ -426,41 +437,76 @@ end_replication_step(void) } /* - * Handle streamed transactions. + * Handle streamed transactions for both main apply worker and apply background + * worker. * - * If in streaming mode (receiving a block of streamed transaction), we - * simply redirect it to a file for the proper toplevel transaction. + * In streaming case (receiving a block of streamed transaction), for + * SUBSTREAM_ON mode, we simply redirect it to a file for the proper toplevel + * transaction, and for SUBSTREAM_APPLY mode, we send the changes to background + * apply worker (LOGICAL_REP_MSG_RELATION or LOGICAL_REP_MSG_TYPE changes will + * also be applied in main apply worker). * - * Returns true for streamed transactions, false otherwise (regular mode). + * For non-streamed transactions, returns false; + * For streamed transactions, returns true if in main apply worker (except we + * apply streamed transaction in "apply" mode and address + * LOGICAL_REP_MSG_RELATION or LOGICAL_REP_MSG_TYPE changes), false otherwise. */ static bool handle_streamed_transaction(LogicalRepMsgType action, StringInfo s) { - TransactionId xid; + TransactionId current_xid = InvalidTransactionId; /* not in streaming mode */ - if (!in_streamed_transaction) + if (!(in_streamed_transaction || am_apply_bgworker())) return false; - Assert(stream_fd != NULL); Assert(TransactionIdIsValid(stream_xid)); /* * We should have received XID of the subxact as the first part of the * message, so extract it. */ - xid = pq_getmsgint(s, 4); + current_xid = pq_getmsgint(s, 4); - if (!TransactionIdIsValid(xid)) + if (!TransactionIdIsValid(current_xid)) ereport(ERROR, (errcode(ERRCODE_PROTOCOL_VIOLATION), errmsg_internal("invalid transaction ID in streamed replication transaction"))); - /* Add the new subxact to the array (unless already there). */ - subxact_info_add(xid); + if (am_apply_bgworker()) + { + /* define a savepoint for a subxact if needed. */ + apply_bgworker_subxact_info_add(current_xid); + + return false; + } + else if (apply_bgworker_active()) + { + /* + * If we decided to apply the changes of this transaction in an apply + * background worker, pass the data to the worker. + */ + apply_bgworker_send_data(stream_apply_worker, s->len, s->data); + nchanges += 1; - /* write the change to the current file */ - stream_write_change(action, s); + /* + * XXX The publisher side doesn't always send relation/type update + * messages after the streaming transaction, so also update the + * relation/type in main apply worker here. See function + * cleanup_rel_sync_cache. + */ + if (action == LOGICAL_REP_MSG_RELATION || + action == LOGICAL_REP_MSG_TYPE) + return false; + } + else + { + /* Add the new subxact to the array (unless already there). */ + subxact_info_add(current_xid); + + /* write the change to the current file */ + stream_write_change(action, s); + } return true; } @@ -844,6 +890,9 @@ apply_handle_commit(StringInfo s) apply_handle_commit_internal(&commit_data); + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); + /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(commit_data.end_lsn); @@ -898,7 +947,9 @@ apply_handle_prepare_internal(LogicalRepPreparedTxnData *prepare_data) * BeginTransactionBlock is necessary to balance the EndTransactionBlock * called within the PrepareTransactionBlock below. */ - BeginTransactionBlock(); + if (!IsTransactionBlock()) + BeginTransactionBlock(); + CommitTransactionCommand(); /* Completes the preceding Begin command. */ /* @@ -950,6 +1001,9 @@ apply_handle_prepare(StringInfo s) in_remote_transaction = false; + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); + /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(prepare_data.end_lsn); @@ -1064,10 +1118,6 @@ apply_handle_rollback_prepared(StringInfo s) /* * Handle STREAM PREPARE. - * - * Logic is in two parts: - * 1. Replay all the spooled operations - * 2. Mark the transaction as prepared */ static void apply_handle_stream_prepare(StringInfo s) @@ -1088,24 +1138,70 @@ apply_handle_stream_prepare(StringInfo s) logicalrep_read_stream_prepare(s, &prepare_data); set_apply_error_context_xact(prepare_data.xid, prepare_data.prepare_lsn); - elog(DEBUG1, "received prepare for streamed transaction %u", prepare_data.xid); + if (am_apply_bgworker()) + { + /* Mark the transaction as prepared. */ + apply_handle_prepare_internal(&prepare_data); - /* Replay all the spooled operations. */ - apply_spooled_messages(prepare_data.xid, prepare_data.prepare_lsn); + CommitTransactionCommand(); - /* Mark the transaction as prepared. */ - apply_handle_prepare_internal(&prepare_data); + pgstat_report_stat(false); - CommitTransactionCommand(); + list_free(subxactlist); + subxactlist = NIL; - pgstat_report_stat(false); + apply_bgworker_set_status(APPLY_BGWORKER_FINISHED); + } + else + { + ApplyBgworkerState *wstate = apply_bgworker_find_or_start(prepare_data.xid, false); - store_flush_position(prepare_data.end_lsn); + elog(DEBUG1, "received prepare for streamed transaction %u", + prepare_data.xid); + + /* + * This is the main apply worker. Check if we are processing this + * transaction in a apply background worker. + */ + if (wstate) + { + apply_bgworker_send_data(wstate, s->len, s->data); + + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_FINISHED); + apply_bgworker_free(wstate); + + pgstat_report_stat(false); + store_flush_position(prepare_data.end_lsn); + } + else + { + /* + * This is the main apply worker and the transaction has been + * serialized to file, replay all the spooled operations. + */ + apply_spooled_messages(prepare_data.xid, prepare_data.prepare_lsn); + + /* Mark the transaction as prepared. */ + apply_handle_prepare_internal(&prepare_data); + + CommitTransactionCommand(); + + pgstat_report_stat(false); + + store_flush_position(prepare_data.end_lsn); + + in_remote_transaction = false; + + /* unlink the files with serialized changes and subxact info. */ + stream_cleanup_files(MyLogicalRepWorker->subid, prepare_data.xid); + } + } in_remote_transaction = false; + stream_apply_worker = NULL; - /* unlink the files with serialized changes and subxact info. */ - stream_cleanup_files(MyLogicalRepWorker->subid, prepare_data.xid); + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(prepare_data.end_lsn); @@ -1155,15 +1251,6 @@ apply_handle_stream_start(StringInfo s) (errcode(ERRCODE_PROTOCOL_VIOLATION), errmsg_internal("duplicate STREAM START message"))); - /* - * Start a transaction on stream start, this transaction will be committed - * on the stream stop unless it is a tablesync worker in which case it - * will be committed after processing all the messages. We need the - * transaction for handling the buffile, used for serializing the - * streaming data and subxact info. - */ - begin_replication_step(); - /* notify handle methods we're processing a remote transaction */ in_streamed_transaction = true; @@ -1177,36 +1264,92 @@ apply_handle_stream_start(StringInfo s) set_apply_error_context_xact(stream_xid, InvalidXLogRecPtr); + if (am_apply_bgworker()) + { + /* + * Make sure the handle apply_dispatch methods are aware we're in a + * remote transaction. + */ + in_remote_transaction = true; + + /* If we are in a apply background worker, begin the transaction */ + AcceptInvalidationMessages(); + maybe_reread_subscription(); + + StartTransactionCommand(); + BeginTransactionBlock(); + CommitTransactionCommand(); + + return; + } + /* - * Initialize the worker's stream_fileset if we haven't yet. This will be - * used for the entire duration of the worker so create it in a permanent - * context. We create this on the very first streaming message from any - * transaction and then use it for this and other streaming transactions. - * Now, we could create a fileset at the start of the worker as well but - * then we won't be sure that it will ever be used. + * This is the main apply worker. Check if there is any free apply + * background worker we can use to process this transaction. */ - if (MyLogicalRepWorker->stream_fileset == NULL) + stream_apply_worker = apply_bgworker_find_or_start(stream_xid, first_segment); + + if (stream_apply_worker) { - MemoryContext oldctx; + /* + * If we have found a free worker or if we are already applying this + * transaction in an apply background worker, then we pass the data to + * that worker. + */ + if (first_segment) + apply_bgworker_send_data(stream_apply_worker, s->len, s->data); - oldctx = MemoryContextSwitchTo(ApplyContext); + nchanges = 0; + elog(DEBUG1, "starting streaming of xid %u", stream_xid); + } - MyLogicalRepWorker->stream_fileset = palloc(sizeof(FileSet)); - FileSetInit(MyLogicalRepWorker->stream_fileset); + /* + * If no worker is available for the first stream start, we start to + * serialize all the changes of the transaction. + */ + else + { + /* + * Start a transaction on stream start, this transaction will be + * committed on the stream stop unless it is a tablesync worker in + * which case it will be committed after processing all the messages. + * We need the transaction for handling the buffile, used for + * serializing the streaming data and subxact info. + */ + begin_replication_step(); - MemoryContextSwitchTo(oldctx); - } + /* + * Initialize the worker's stream_fileset if we haven't yet. This will + * be used for the entire duration of the worker so create it in a + * permanent context. We create this on the very first streaming + * message from any transaction and then use it for this and other + * streaming transactions. Now, we could create a fileset at the start + * of the worker as well but then we won't be sure that it will ever + * be used. + */ + if (MyLogicalRepWorker->stream_fileset == NULL) + { + MemoryContext oldctx; - /* open the spool file for this transaction */ - stream_open_file(MyLogicalRepWorker->subid, stream_xid, first_segment); + oldctx = MemoryContextSwitchTo(ApplyContext); - /* if this is not the first segment, open existing subxact file */ - if (!first_segment) - subxact_info_read(MyLogicalRepWorker->subid, stream_xid); + MyLogicalRepWorker->stream_fileset = palloc(sizeof(FileSet)); + FileSetInit(MyLogicalRepWorker->stream_fileset); - pgstat_report_activity(STATE_RUNNING, NULL); + MemoryContextSwitchTo(oldctx); + } - end_replication_step(); + /* open the spool file for this transaction */ + stream_open_file(MyLogicalRepWorker->subid, stream_xid, first_segment); + + /* if this is not the first segment, open existing subxact file */ + if (!first_segment) + subxact_info_read(MyLogicalRepWorker->subid, stream_xid); + + end_replication_step(); + } + + pgstat_report_activity(STATE_RUNNING, NULL); } /* @@ -1220,44 +1363,47 @@ apply_handle_stream_stop(StringInfo s) (errcode(ERRCODE_PROTOCOL_VIOLATION), errmsg_internal("STREAM STOP message without STREAM START"))); - /* - * Close the file with serialized changes, and serialize information about - * subxacts for the toplevel transaction. - */ - subxact_info_write(MyLogicalRepWorker->subid, stream_xid); - stream_close_file(); + if (apply_bgworker_active()) + { + char action = LOGICAL_REP_MSG_STREAM_STOP; - /* We must be in a valid transaction state */ - Assert(IsTransactionState()); + apply_bgworker_send_data(stream_apply_worker, 1, &action); + apply_bgworker_wait_for(stream_apply_worker, APPLY_BGWORKER_READY); - /* Commit the per-stream transaction */ - CommitTransactionCommand(); + elog(DEBUG1, "stopped streaming of xid %u, %u changes streamed", stream_xid, nchanges); + } + else + { + /* + * Close the file with serialized changes, and serialize information + * about subxacts for the toplevel transaction. + */ + subxact_info_write(MyLogicalRepWorker->subid, stream_xid); + stream_close_file(); - in_streamed_transaction = false; + /* We must be in a valid transaction state */ + Assert(IsTransactionState()); + + /* Commit the per-stream transaction */ + CommitTransactionCommand(); - /* Reset per-stream context */ - MemoryContextReset(LogicalStreamingContext); + /* Reset per-stream context */ + MemoryContextReset(LogicalStreamingContext); + } + + in_streamed_transaction = false; + stream_apply_worker = NULL; pgstat_report_activity(STATE_IDLE, NULL); reset_apply_error_context_info(); } /* - * Handle STREAM abort message. + * Handle STREAM ABORT message when the transaction was spilled to disk. */ static void -apply_handle_stream_abort(StringInfo s) +serialize_stream_abort(TransactionId xid, TransactionId subxid) { - TransactionId xid; - TransactionId subxid; - - if (in_streamed_transaction) - ereport(ERROR, - (errcode(ERRCODE_PROTOCOL_VIOLATION), - errmsg_internal("STREAM ABORT message without STREAM STOP"))); - - logicalrep_read_stream_abort(s, &xid, &subxid); - /* * If the two XIDs are the same, it's in fact abort of toplevel xact, so * just delete the files with serialized info. @@ -1339,8 +1485,118 @@ apply_handle_stream_abort(StringInfo s) end_replication_step(); CommitTransactionCommand(); } +} - reset_apply_error_context_info(); +/* + * Handle STREAM ABORT message. + */ +static void +apply_handle_stream_abort(StringInfo s) +{ + TransactionId xid; + TransactionId subxid; + + if (in_streamed_transaction) + ereport(ERROR, + (errcode(ERRCODE_PROTOCOL_VIOLATION), + errmsg_internal("STREAM ABORT message without STREAM STOP"))); + + logicalrep_read_stream_abort(s, &xid, &subxid); + + if (am_apply_bgworker()) + { + elog(DEBUG1, "[Apply BGW #%u] aborting current transaction xid=%u, subxid=%u", + MyParallelState->n, GetCurrentTransactionIdIfAny(), + GetCurrentSubTransactionId()); + + /* + * If the two XIDs are the same, it's in fact abort of toplevel xact, + * so just free the subxactlist. + */ + if (subxid == xid) + { + set_apply_error_context_xact(subxid, InvalidXLogRecPtr); + + AbortCurrentTransaction(); + + EndTransactionBlock(false); + CommitTransactionCommand(); + + in_remote_transaction = false; + + list_free(subxactlist); + subxactlist = NIL; + + apply_bgworker_set_status(APPLY_BGWORKER_FINISHED); + } + else + { + /* + * OK, so it's a subxact. Rollback to the savepoint. + * + * We also need to read the subxactlist, determine the offset + * tracked for the subxact, and truncate the list. + */ + int i; + bool found = false; + char spname[MAXPGPATH]; + + set_apply_error_context_xact(subxid, InvalidXLogRecPtr); + snprintf(spname, MAXPGPATH, "savepoint_for_xid_%u", subxid); + + elog(DEBUG1, "[Apply BGW #%u] rolling back to savepoint %s", + MyParallelState->n, spname); + + for (i = list_length(subxactlist) - 1; i >= 0; i--) + { + xid = (TransactionId) list_nth_int(subxactlist, i); + if (xid == subxid) + { + found = true; + break; + } + } + + if (found) + { + RollbackToSavepoint(spname); + CommitTransactionCommand(); + subxactlist = list_truncate(subxactlist, i + 1); + } + + apply_bgworker_set_status(APPLY_BGWORKER_READY); + } + + reset_apply_error_context_info(); + } + else + { + ApplyBgworkerState *wstate = apply_bgworker_find_or_start(xid, false); + + /* + * This is the main apply worker. Check if we are processing this + * transaction in a apply background worker. + */ + if (wstate) + { + apply_bgworker_send_data(wstate, s->len, s->data); + + if (subxid == xid) + { + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_FINISHED); + apply_bgworker_free(wstate); + } + else + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_READY); + } + + /* + * We are in main apply worker and the transaction has been serialized + * to file. + */ + else + serialize_stream_abort(xid, subxid); + } } /* @@ -1462,40 +1718,6 @@ apply_spooled_messages(TransactionId xid, XLogRecPtr lsn) return; } -/* - * Handle STREAM COMMIT message. - */ -static void -apply_handle_stream_commit(StringInfo s) -{ - TransactionId xid; - LogicalRepCommitData commit_data; - - if (in_streamed_transaction) - ereport(ERROR, - (errcode(ERRCODE_PROTOCOL_VIOLATION), - errmsg_internal("STREAM COMMIT message without STREAM STOP"))); - - xid = logicalrep_read_stream_commit(s, &commit_data); - set_apply_error_context_xact(xid, commit_data.commit_lsn); - - elog(DEBUG1, "received commit for streamed transaction %u", xid); - - apply_spooled_messages(xid, commit_data.commit_lsn); - - apply_handle_commit_internal(&commit_data); - - /* unlink the files with serialized changes and subxact info */ - stream_cleanup_files(MyLogicalRepWorker->subid, xid); - - /* Process any tables that are being synchronized in parallel. */ - process_syncing_tables(commit_data.end_lsn); - - pgstat_report_activity(STATE_IDLE, NULL); - - reset_apply_error_context_info(); -} - /* * Helper function for apply_handle_commit and apply_handle_stream_commit. */ @@ -2445,11 +2667,105 @@ apply_handle_truncate(StringInfo s) end_replication_step(); } +/* + * Handle STREAM COMMIT message. + */ +static void +apply_handle_stream_commit(StringInfo s) +{ + LogicalRepCommitData commit_data; + TransactionId xid; + + if (in_streamed_transaction) + ereport(ERROR, + (errcode(ERRCODE_PROTOCOL_VIOLATION), + errmsg_internal("STREAM COMMIT message without STREAM STOP"))); + + xid = logicalrep_read_stream_commit(s, &commit_data); + set_apply_error_context_xact(xid, commit_data.commit_lsn); + + if (am_apply_bgworker()) + { + /* + * Update origin state so we can restart streaming from correct + * position in case of crash. + */ + replorigin_session_origin_lsn = commit_data.end_lsn; + replorigin_session_origin_timestamp = commit_data.committime; + + CommitTransactionCommand(); + EndTransactionBlock(false); + CommitTransactionCommand(); + + in_remote_transaction = false; + + pgstat_report_stat(false); + + list_free(subxactlist); + subxactlist = NIL; + + apply_bgworker_set_status(APPLY_BGWORKER_FINISHED); + } + else + { + /* + * This is the main apply worker. Check if we are processing this + * transaction in an apply background worker. + */ + ApplyBgworkerState *wstate = apply_bgworker_find_or_start(xid, false); + + elog(DEBUG1, "received commit for streamed transaction %u", xid); + + if (wstate) + { + /* Send commit message */ + apply_bgworker_send_data(wstate, s->len, s->data); + + /* Wait for apply background worker to finish */ + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_FINISHED); + + pgstat_report_stat(false); + store_flush_position(commit_data.end_lsn); + stop_skipping_changes(); + + apply_bgworker_free(wstate); + + /* + * The transaction is either non-empty or skipped, so we clear the + * subskiplsn. + */ + clear_subscription_skip_lsn(commit_data.commit_lsn); + } + else + { + /* + * This is the main apply worker and the transaction has been + * serialized to file, replay all the spooled operations. + */ + apply_spooled_messages(xid, commit_data.commit_lsn); + + apply_handle_commit_internal(&commit_data); + + /* unlink the files with serialized changes and subxact info */ + stream_cleanup_files(MyLogicalRepWorker->subid, xid); + } + } + + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); + + /* Process any tables that are being synchronized in parallel. */ + process_syncing_tables(commit_data.end_lsn); + + pgstat_report_activity(STATE_IDLE, NULL); + + reset_apply_error_context_info(); +} /* * Logical replication protocol message dispatcher. */ -static void +void apply_dispatch(StringInfo s) { LogicalRepMsgType action = pq_getmsgbyte(s); @@ -2618,6 +2934,10 @@ store_flush_position(XLogRecPtr remote_lsn) { FlushPosition *flushpos; + /* We only need to collect the LSN in main apply worker */ + if (am_apply_bgworker()) + return; + /* Need to do this in permanent context */ MemoryContextSwitchTo(ApplyContext); @@ -2632,7 +2952,7 @@ store_flush_position(XLogRecPtr remote_lsn) /* Update statistics of the worker. */ -static void +void UpdateWorkerStats(XLogRecPtr last_lsn, TimestampTz send_time, bool reply) { MyLogicalRepWorker->last_lsn = last_lsn; @@ -2794,6 +3114,9 @@ LogicalRepApplyLoop(XLogRecPtr last_received) AcceptInvalidationMessages(); maybe_reread_subscription(); + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); + /* Process any table synchronization changes. */ process_syncing_tables(last_received); } @@ -3095,7 +3418,7 @@ maybe_reread_subscription(void) /* * Callback from subscription syscache invalidation. */ -static void +void subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue) { MySubscriptionValid = false; @@ -3691,7 +4014,7 @@ ApplyWorkerMain(Datum main_arg) originid = replorigin_by_name(originname, true); if (!OidIsValid(originid)) originid = replorigin_create(originname); - replorigin_session_setup(originid); + replorigin_session_setup(originid, true); replorigin_session_origin = originid; origin_startpos = replorigin_session_get_progress(false); CommitTransactionCommand(); @@ -3738,7 +4061,7 @@ ApplyWorkerMain(Datum main_arg) options.proto.logical.publication_names = MySubscription->publications; options.proto.logical.binary = MySubscription->binary; - options.proto.logical.streaming = MySubscription->stream; + options.proto.logical.streaming = (MySubscription->stream != SUBSTREAM_OFF); options.proto.logical.twophase = false; if (!am_tablesync_worker()) @@ -3896,7 +4219,8 @@ clear_subscription_skip_lsn(XLogRecPtr finish_lsn) XLogRecPtr myskiplsn = MySubscription->skiplsn; bool started_tx = false; - if (likely(XLogRecPtrIsInvalid(myskiplsn))) + if (likely(XLogRecPtrIsInvalid(myskiplsn)) || + am_apply_bgworker()) return; if (!IsTransactionState()) @@ -3968,7 +4292,7 @@ clear_subscription_skip_lsn(XLogRecPtr finish_lsn) } /* Error callback to give more context info about the change being applied */ -static void +void apply_error_callback(void *arg) { ApplyErrorCallbackArg *errarg = &apply_error_callback_arg; diff --git a/src/backend/utils/activity/wait_event.c b/src/backend/utils/activity/wait_event.c index 87c15b9c6f..ba781e6f08 100644 --- a/src/backend/utils/activity/wait_event.c +++ b/src/backend/utils/activity/wait_event.c @@ -388,6 +388,9 @@ pgstat_get_wait_ipc(WaitEventIPC w) case WAIT_EVENT_HASH_GROW_BUCKETS_REINSERT: event_name = "HashGrowBucketsReinsert"; break; + case WAIT_EVENT_LOGICAL_APPLY_WORKER_STATE_CHANGE: + event_name = "LogicalApplyWorkerStateChange"; + break; case WAIT_EVENT_LOGICAL_SYNC_DATA: event_name = "LogicalSyncData"; break; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 7cc9c72e49..6f8b30abb0 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -4450,7 +4450,7 @@ getSubscriptions(Archive *fout) if (fout->remoteVersion >= 140000) appendPQExpBufferStr(query, " s.substream,\n"); else - appendPQExpBufferStr(query, " false AS substream,\n"); + appendPQExpBufferStr(query, " 'f' AS substream,\n"); if (fout->remoteVersion >= 150000) appendPQExpBufferStr(query, @@ -4580,8 +4580,10 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo) if (strcmp(subinfo->subbinary, "t") == 0) appendPQExpBufferStr(query, ", binary = true"); - if (strcmp(subinfo->substream, "f") != 0) + if (strcmp(subinfo->substream, "t") == 0) appendPQExpBufferStr(query, ", streaming = on"); + else if (strcmp(subinfo->substream, "a") == 0) + appendPQExpBufferStr(query, ", streaming = apply"); if (strcmp(subinfo->subtwophasestate, two_phase_disabled) != 0) appendPQExpBufferStr(query, ", two_phase = on"); diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h index d1260f590c..9b394a45fe 100644 --- a/src/include/catalog/pg_subscription.h +++ b/src/include/catalog/pg_subscription.h @@ -68,7 +68,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW bool subbinary; /* True if the subscription wants the * publisher to send data in binary */ - bool substream; /* Stream in-progress transactions. */ + char substream; /* Stream in-progress transactions. + * See SUBSTREAM_xxx constants. */ char subtwophasestate; /* Stream two-phase transactions */ @@ -109,7 +110,7 @@ typedef struct Subscription bool enabled; /* Indicates if the subscription is enabled */ bool binary; /* Indicates if the subscription wants data in * binary format */ - bool stream; /* Allow streaming in-progress transactions. */ + char stream; /* Allow streaming in-progress transactions. */ char twophasestate; /* Allow streaming two-phase transactions */ bool disableonerr; /* Indicates if the subscription should be * automatically disabled if a worker error @@ -120,6 +121,18 @@ typedef struct Subscription List *publications; /* List of publication names to subscribe to */ } Subscription; +/* Disallow streaming in-progress transactions */ +#define SUBSTREAM_OFF 'f' + +/* + * Streaming transactions are written to a temporary file and applied only + * after the transaction is committed on upstream. + */ +#define SUBSTREAM_ON 't' + +/* Streaming transactions are applied immediately via a background worker */ +#define SUBSTREAM_APPLY 'a' + extern Subscription *GetSubscription(Oid subid, bool missing_ok); extern void FreeSubscription(Subscription *sub); extern void DisableSubscription(Oid subid); diff --git a/src/include/replication/logicalworker.h b/src/include/replication/logicalworker.h index cd1b6e8afc..6a1af7f13c 100644 --- a/src/include/replication/logicalworker.h +++ b/src/include/replication/logicalworker.h @@ -13,6 +13,7 @@ #define LOGICALWORKER_H extern void ApplyWorkerMain(Datum main_arg); +extern void ApplyBgworkerMain(Datum main_arg); extern bool IsLogicalWorker(void); diff --git a/src/include/replication/origin.h b/src/include/replication/origin.h index 2d1b5e5c28..c7389b40a7 100644 --- a/src/include/replication/origin.h +++ b/src/include/replication/origin.h @@ -53,7 +53,7 @@ extern XLogRecPtr replorigin_get_progress(RepOriginId node, bool flush); extern void replorigin_session_advance(XLogRecPtr remote_commit, XLogRecPtr local_commit); -extern void replorigin_session_setup(RepOriginId node); +extern void replorigin_session_setup(RepOriginId node, bool must_acquire); extern void replorigin_session_reset(void); extern XLogRecPtr replorigin_session_get_progress(bool flush); diff --git a/src/include/replication/worker_internal.h b/src/include/replication/worker_internal.h index 901845abc2..e2d21444c0 100644 --- a/src/include/replication/worker_internal.h +++ b/src/include/replication/worker_internal.h @@ -17,8 +17,11 @@ #include "access/xlogdefs.h" #include "catalog/pg_subscription.h" #include "datatype/timestamp.h" +#include "replication/logicalrelation.h" #include "storage/fileset.h" #include "storage/lock.h" +#include "storage/shm_mq.h" +#include "storage/shm_toc.h" #include "storage/spin.h" @@ -60,6 +63,9 @@ typedef struct LogicalRepWorker */ FileSet *stream_fileset; + /* Indicates if this slot is used for an apply background worker. */ + bool subworker; + /* Stats. */ XLogRecPtr last_lsn; TimestampTz last_send_time; @@ -68,9 +74,67 @@ typedef struct LogicalRepWorker TimestampTz reply_time; } LogicalRepWorker; +/* Struct for saving and restoring apply errcontext information */ +typedef struct ApplyErrorCallbackArg +{ + LogicalRepMsgType command; /* 0 if invalid */ + LogicalRepRelMapEntry *rel; + + /* Remote node information */ + int remote_attnum; /* -1 if invalid */ + TransactionId remote_xid; + XLogRecPtr finish_lsn; + char *origin_name; +} ApplyErrorCallbackArg; + +/* + * Status for apply background worker. + */ +typedef enum ApplyBgworkerStatus +{ + APPLY_BGWORKER_ATTACHED = 0, + APPLY_BGWORKER_READY, + APPLY_BGWORKER_BUSY, + APPLY_BGWORKER_FINISHED, + APPLY_BGWORKER_EXIT +} ApplyBgworkerStatus; + +/* + * Shared information among apply workers. + */ +typedef struct ApplyBgworkerShared +{ + slock_t mutex; + + /* Status for apply background worker. */ + ApplyBgworkerStatus status; + + TransactionId stream_xid; + uint32 n; /* id of apply background worker */ +} ApplyBgworkerShared; + +/* + * Struct for maintaining an apply background worker. + */ +typedef struct ApplyBgworkerState +{ + shm_mq_handle *mq_handle; + dsm_segment *dsm_seg; + ApplyBgworkerShared volatile *pstate; +} ApplyBgworkerState; + /* Main memory context for apply worker. Permanent during worker lifetime. */ extern PGDLLIMPORT MemoryContext ApplyContext; +extern PGDLLIMPORT MemoryContext ApplyMessageContext; + +extern PGDLLIMPORT ApplyErrorCallbackArg apply_error_callback_arg; + +extern PGDLLIMPORT bool MySubscriptionValid; + +extern PGDLLIMPORT volatile ApplyBgworkerShared *MyParallelState; +extern PGDLLIMPORT List *subxactlist; + /* libpqreceiver connection */ extern PGDLLIMPORT struct WalReceiverConn *LogRepWorkerWalRcvConn; @@ -79,13 +143,16 @@ extern PGDLLIMPORT Subscription *MySubscription; extern PGDLLIMPORT LogicalRepWorker *MyLogicalRepWorker; extern PGDLLIMPORT bool in_remote_transaction; +extern PGDLLIMPORT bool in_streamed_transaction; +extern PGDLLIMPORT TransactionId stream_xid; extern void logicalrep_worker_attach(int slot); extern LogicalRepWorker *logicalrep_worker_find(Oid subid, Oid relid, bool only_running); extern List *logicalrep_workers_find(Oid subid, bool only_running); -extern void logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, - Oid userid, Oid relid); +extern bool logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, + Oid userid, Oid relid, + dsm_handle subworker_dsm); extern void logicalrep_worker_stop(Oid subid, Oid relid); extern void logicalrep_worker_wakeup(Oid subid, Oid relid); extern void logicalrep_worker_wakeup_ptr(LogicalRepWorker *worker); @@ -103,10 +170,39 @@ extern void process_syncing_tables(XLogRecPtr current_lsn); extern void invalidate_syncing_table_states(Datum arg, int cacheid, uint32 hashvalue); +extern void UpdateWorkerStats(XLogRecPtr last_lsn, TimestampTz send_time, + bool reply); + +/* prototype needed because of stream_commit */ +extern void apply_dispatch(StringInfo s); + +/* Function for apply error callback */ +extern void apply_error_callback(void *arg); + +extern void subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue); + +/* apply background worker setup and interactions */ +extern ApplyBgworkerState *apply_bgworker_find_or_start(TransactionId xid, + bool start); +extern void apply_bgworker_wait_for(ApplyBgworkerState *wstate, + ApplyBgworkerStatus wait_for_status); +extern void apply_bgworker_send_data(ApplyBgworkerState *wstate, Size nbytes, + const void *data); +extern void apply_bgworker_free(ApplyBgworkerState *wstate); +extern void apply_bgworker_check_status(void); +extern void apply_bgworker_set_status(ApplyBgworkerStatus status); +extern void apply_bgworker_subxact_info_add(TransactionId current_xid); + static inline bool am_tablesync_worker(void) { return OidIsValid(MyLogicalRepWorker->relid); } +static inline bool +am_apply_bgworker(void) +{ + return MyLogicalRepWorker->subworker; +} + #endif /* WORKER_INTERNAL_H */ diff --git a/src/include/utils/wait_event.h b/src/include/utils/wait_event.h index b578e2ec75..c2d2a114d7 100644 --- a/src/include/utils/wait_event.h +++ b/src/include/utils/wait_event.h @@ -105,6 +105,7 @@ typedef enum WAIT_EVENT_HASH_GROW_BUCKETS_ALLOCATE, WAIT_EVENT_HASH_GROW_BUCKETS_ELECT, WAIT_EVENT_HASH_GROW_BUCKETS_REINSERT, + WAIT_EVENT_LOGICAL_APPLY_WORKER_STATE_CHANGE, WAIT_EVENT_LOGICAL_SYNC_DATA, WAIT_EVENT_LOGICAL_SYNC_STATE_CHANGE, WAIT_EVENT_MQ_INTERNAL, diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out index 7fcfad1591..f769835af0 100644 --- a/src/test/regress/expected/subscription.out +++ b/src/test/regress/expected/subscription.out @@ -197,7 +197,7 @@ ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); DROP SUBSCRIPTION regress_testsub; -- fail - streaming must be boolean CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = foo); -ERROR: streaming requires a Boolean value +ERROR: streaming requires a Boolean value or "apply" -- now it works CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = true); WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 4fb746930a..12940d6f4f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -116,6 +116,10 @@ Append AppendPath AppendRelInfo AppendState +ApplyBgworkerEntry +ApplyBgworkerShared +ApplyBgworkerState +ApplyBgworkerStatus ApplyErrorCallbackArg ApplyExecutionData ApplySubXactData -- 2.23.0.windows.1 [application/octet-stream] v8-0002-Test-streaming-apply-option-in-tap-test.patch (68.9K, ../../OS3PR01MB62758A881FF3240171B7B21B9EDE9@OS3PR01MB6275.jpnprd01.prod.outlook.com/3-v8-0002-Test-streaming-apply-option-in-tap-test.patch) download | inline diff: From 5d49a3bc0a9dc0cea88668cc84c16d4f8341e474 Mon Sep 17 00:00:00 2001 From: "shiy.fnst" <[email protected]> Date: Fri, 13 May 2022 14:50:30 +0800 Subject: [PATCH v8 2/4] Test streaming apply option in tap test Change all TAP tests using the SUBSCRIPTION "streaming" option, so they now test both 'on' and 'apply' values. --- src/test/subscription/t/015_stream.pl | 199 ++++--- src/test/subscription/t/016_stream_subxact.pl | 119 +++-- src/test/subscription/t/017_stream_ddl.pl | 188 ++++--- .../t/018_stream_subxact_abort.pl | 195 ++++--- .../t/019_stream_subxact_ddl_abort.pl | 110 +++- .../subscription/t/022_twophase_cascade.pl | 363 +++++++------ .../subscription/t/023_twophase_stream.pl | 498 ++++++++++-------- 7 files changed, 1035 insertions(+), 637 deletions(-) diff --git a/src/test/subscription/t/015_stream.pl b/src/test/subscription/t/015_stream.pl index 6561b189de..b3d84b1c7c 100644 --- a/src/test/subscription/t/015_stream.pl +++ b/src/test/subscription/t/015_stream.pl @@ -8,6 +8,116 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" option +# so the same code can be run both for the streaming=on and streaming=apply +# cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_apply) = @_; + + # Interleave a pair of transactions, each exceeding the 64kB limit. + my $in = ''; + my $out = ''; + + my $timer = IPC::Run::timeout($PostgreSQL::Test::Utils::timeout_default); + + my $h = $node_publisher->background_psql('postgres', \$in, \$out, $timer, + on_error_stop => 0); + + # Check that a background worker starts if "streaming" option is + # specified as "apply". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_apply) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + $in .= q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + }; + $h->pump_nb; + + if ($is_apply) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001, 9999) s(i); + DELETE FROM test_tab WHERE a > 5000; + COMMIT; + }); + + $in .= q{ + COMMIT; + \q + }; + $h->finish; # errors make the next test fail, so ignore them here + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), + 'check extra columns contain local defaults'); + + # Test the streaming in binary mode + $node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET (binary = on)"); + + # Insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001, 10000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + COMMIT; + }); + + $node_publisher->wait_for_catchup($appname); + + $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(6667|6667|6667), + 'check extra columns contain local defaults'); + + # Change the local values of the extra columns on the subscriber, + # update publisher, and check that subscriber retains the expected + # values. This is to ensure that non-streaming transactions behave + # properly after a streaming transaction. + $node_subscriber->safe_psql('postgres', + "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'" + ); + $node_publisher->safe_psql('postgres', + "UPDATE test_tab SET b = md5(a::text)"); + + $node_publisher->wait_for_catchup($appname); + + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab" + ); + is($result, qq(6667|6667|6667), + 'check extra columns contain locally changed data'); + + # Cleanup the test data + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE (a > 2)"); + $node_publisher->wait_for_catchup($appname); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -37,6 +147,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -54,82 +168,25 @@ my $result = "SELECT count(*), count(c), count(d = 999) FROM test_tab"); is($result, qq(2|2|2), 'check initial data was copied to subscriber'); -# Interleave a pair of transactions, each exceeding the 64kB limit. -my $in = ''; -my $out = ''; - -my $timer = IPC::Run::timeout($PostgreSQL::Test::Utils::timeout_default); - -my $h = $node_publisher->background_psql('postgres', \$in, \$out, $timer, - on_error_stop => 0); - -$in .= q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -}; -$h->pump_nb; - -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001, 9999) s(i); -DELETE FROM test_tab WHERE a > 5000; -COMMIT; -}); - -$in .= q{ -COMMIT; -\q -}; -$h->finish; # errors make the next test fail, so ignore them here - -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), 'check extra columns contain local defaults'); - -# Test the streaming in binary mode -$node_subscriber->safe_psql('postgres', - "ALTER SUBSCRIPTION tap_sub SET (binary = on)"); - -# Insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001, 10000) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -COMMIT; -}); - -$node_publisher->wait_for_catchup($appname); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(6667|6667|6667), 'check extra columns contain local defaults'); +################################### +# Test using streaming mode 'apply' +################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" +); -# Change the local values of the extra columns on the subscriber, -# update publisher, and check that subscriber retains the expected -# values. This is to ensure that non-streaming transactions behave -# properly after a streaming transaction. $node_subscriber->safe_psql('postgres', - "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'" -); -$node_publisher->safe_psql('postgres', - "UPDATE test_tab SET b = md5(a::text)"); + "ALTER SUBSCRIPTION tap_sub SET(streaming = apply, binary = off)"); -$node_publisher->wait_for_catchup($appname); +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab" -); -is($result, qq(6667|6667|6667), - 'check extra columns contain locally changed data'); +test_streaming($node_publisher, $node_subscriber, $appname, 1); $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/016_stream_subxact.pl b/src/test/subscription/t/016_stream_subxact.pl index f27f1694f2..b5b6dc379f 100644 --- a/src/test/subscription/t/016_stream_subxact.pl +++ b/src/test/subscription/t/016_stream_subxact.pl @@ -8,6 +8,72 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" option +# so the same code can be run both for the streaming=on and streaming=apply +# cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_apply) = @_; + + # Check that a background worker starts if "streaming" option is + # specified as "apply". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_apply) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + # Insert, update and delete enough rows to exceed 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series( 3, 500) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501, 1000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001, 1500) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + SAVEPOINT s3; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501, 2000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + SAVEPOINT s4; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001, 2500) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + COMMIT; + }); + + if ($is_apply) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(1667|1667|1667), + 'check data was copied to subscriber in streaming mode and extra columns contain local defaults' + ); + + # Cleanup the test data + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE (a > 2)"); + $node_publisher->wait_for_catchup($appname); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -37,6 +103,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -54,41 +124,26 @@ my $result = "SELECT count(*), count(c), count(d = 999) FROM test_tab"); is($result, qq(2|2|2), 'check initial data was copied to subscriber'); -# Insert, update and delete enough rows to exceed 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series( 3, 500) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501, 1000) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001, 1500) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -SAVEPOINT s3; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501, 2000) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -SAVEPOINT s4; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001, 2500) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -COMMIT; -}); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(1667|1667|1667), - 'check data was copied to subscriber in streaming mode and extra columns contain local defaults' +################################### +# Test using streaming mode 'apply' +################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" ); +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = apply)"); + +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; + +test_streaming($node_publisher, $node_subscriber, $appname, 1); + $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/017_stream_ddl.pl b/src/test/subscription/t/017_stream_ddl.pl index 0bce63b716..7e5dc18cd2 100644 --- a/src/test/subscription/t/017_stream_ddl.pl +++ b/src/test/subscription/t/017_stream_ddl.pl @@ -8,6 +8,111 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" option +# so the same code can be run both for the streaming=on and streaming=apply +# cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_apply) = @_; + + # a small (non-streamed) transaction with DDL and DML + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab VALUES (3, md5(3::text)); + ALTER TABLE test_tab ADD COLUMN c INT; + SAVEPOINT s1; + INSERT INTO test_tab VALUES (4, md5(4::text), -4); + COMMIT; + }); + + # Check that a background worker starts if "streaming" option is + # specified as "apply". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_apply) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + # large (streamed) transaction with DDL and DML + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(5, 1000) s(i); + ALTER TABLE test_tab ADD COLUMN d INT; + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001, 2000) s(i); + COMMIT; + }); + + if ($is_apply) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + # a small (non-streamed) transaction with DDL and DML + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab VALUES (2001, md5(2001::text), -2001, 2*2001); + ALTER TABLE test_tab ADD COLUMN e INT; + SAVEPOINT s1; + INSERT INTO test_tab VALUES (2002, md5(2002::text), -2002, 2*2002, -3*2002); + COMMIT; + }); + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d), count(e) FROM test_tab"); + is($result, qq(2002|1999|1002|1), + 'check data was copied to subscriber in streaming mode and extra columns contain local defaults' + ); + + # A large (streamed) transaction with DDL and DML. One of the DDL is performed + # after DML to ensure that we invalidate the schema sent for test_tab so that + # the next transaction has to send the schema again. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(2003,5000) s(i); + ALTER TABLE test_tab ADD COLUMN f INT; + COMMIT; + }); + + # A small transaction that won't get streamed. This is just to ensure that we + # send the schema again to reflect the last column added in the previous test. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i, 4*i FROM generate_series(5001,5005) s(i); + COMMIT; + }); + + $node_publisher->wait_for_catchup($appname); + + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d), count(e), count(f) FROM test_tab" + ); + is($result, qq(5005|5002|4005|3004|5), + 'check data was copied to subscriber for both streaming and non-streaming transactions' + ); + + # Cleanup the test data + $node_publisher->safe_psql( + 'postgres', q{ + DELETE FROM test_tab WHERE (a > 2); + ALTER TABLE test_tab DROP COLUMN c, DROP COLUMN d, DROP COLUMN e, DROP COLUMN f; + }); + $node_publisher->wait_for_catchup($appname); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -37,6 +142,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -54,76 +163,25 @@ my $result = "SELECT count(*), count(c), count(d = 999) FROM test_tab"); is($result, qq(2|0|0), 'check initial data was copied to subscriber'); -# a small (non-streamed) transaction with DDL and DML -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab VALUES (3, md5(3::text)); -ALTER TABLE test_tab ADD COLUMN c INT; -SAVEPOINT s1; -INSERT INTO test_tab VALUES (4, md5(4::text), -4); -COMMIT; -}); - -# large (streamed) transaction with DDL and DML -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(5, 1000) s(i); -ALTER TABLE test_tab ADD COLUMN d INT; -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001, 2000) s(i); -COMMIT; -}); - -# a small (non-streamed) transaction with DDL and DML -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab VALUES (2001, md5(2001::text), -2001, 2*2001); -ALTER TABLE test_tab ADD COLUMN e INT; -SAVEPOINT s1; -INSERT INTO test_tab VALUES (2002, md5(2002::text), -2002, 2*2002, -3*2002); -COMMIT; -}); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d), count(e) FROM test_tab"); -is($result, qq(2002|1999|1002|1), - 'check data was copied to subscriber in streaming mode and extra columns contain local defaults' +################################### +# Test using streaming mode 'apply' +################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" ); -# A large (streamed) transaction with DDL and DML. One of the DDL is performed -# after DML to ensure that we invalidate the schema sent for test_tab so that -# the next transaction has to send the schema again. -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(2003,5000) s(i); -ALTER TABLE test_tab ADD COLUMN f INT; -COMMIT; -}); - -# A small transaction that won't get streamed. This is just to ensure that we -# send the schema again to reflect the last column added in the previous test. -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i, 4*i FROM generate_series(5001,5005) s(i); -COMMIT; -}); +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = apply)"); -$node_publisher->wait_for_catchup($appname); +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d), count(e), count(f) FROM test_tab"); -is($result, qq(5005|5002|4005|3004|5), - 'check data was copied to subscriber for both streaming and non-streaming transactions' -); +test_streaming($node_publisher, $node_subscriber, $appname, 1); $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/018_stream_subxact_abort.pl b/src/test/subscription/t/018_stream_subxact_abort.pl index 7155442e76..71f8c1dd05 100644 --- a/src/test/subscription/t/018_stream_subxact_abort.pl +++ b/src/test/subscription/t/018_stream_subxact_abort.pl @@ -8,6 +8,113 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" option +# so the same code can be run both for the streaming=on and streaming=apply +# cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_apply) = @_; + + # Check that a background worker starts if "streaming" option is + # specified as "apply". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_apply) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + # large (streamed) transaction with DDL, DML and ROLLBACKs + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i); + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501,1000) s(i); + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001,1500) s(i); + SAVEPOINT s3; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501,2000) s(i); + ROLLBACK TO s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001,2500) s(i); + ROLLBACK TO s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2501,3000) s(i); + SAVEPOINT s4; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3001,3500) s(i); + SAVEPOINT s5; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3501,4000) s(i); + COMMIT; + }); + + if ($is_apply) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c) FROM test_tab"); + is($result, qq(2000|0), + 'check rollback to savepoint was reflected on subscriber and extra columns contain local defaults' + ); + + # large (streamed) transaction with subscriber receiving out of order + # subtransaction ROLLBACKs + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(4001,4500) s(i); + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001,5500) s(i); + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(6001,6500) s(i); + SAVEPOINT s3; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(7001,7500) s(i); + RELEASE s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(8001,8500) s(i); + ROLLBACK TO s1; + COMMIT; + }); + + $node_publisher->wait_for_catchup($appname); + + $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c) FROM test_tab"); + is($result, qq(2500|0), + 'check rollback to savepoint was reflected on subscriber'); + + # large (streamed) transaction with subscriber receiving rollback + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(8501,9000) s(i); + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(9001,9500) s(i); + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(9501,10000) s(i); + ROLLBACK; + }); + + $node_publisher->wait_for_catchup($appname); + + $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c) FROM test_tab"); + is($result, qq(2500|0), 'check rollback was reflected on subscriber'); + + # Cleanup the test data + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE (a > 2)"); + $node_publisher->wait_for_catchup($appname); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -36,6 +143,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -53,81 +164,25 @@ my $result = "SELECT count(*), count(c) FROM test_tab"); is($result, qq(2|0), 'check initial data was copied to subscriber'); -# large (streamed) transaction with DDL, DML and ROLLBACKs -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i); -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501,1000) s(i); -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001,1500) s(i); -SAVEPOINT s3; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501,2000) s(i); -ROLLBACK TO s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001,2500) s(i); -ROLLBACK TO s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2501,3000) s(i); -SAVEPOINT s4; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3001,3500) s(i); -SAVEPOINT s5; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3501,4000) s(i); -COMMIT; -}); - -$node_publisher->wait_for_catchup($appname); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c) FROM test_tab"); -is($result, qq(2000|0), - 'check rollback to savepoint was reflected on subscriber and extra columns contain local defaults' +################################### +# Test using streaming mode 'apply' +################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" ); -# large (streamed) transaction with subscriber receiving out of order -# subtransaction ROLLBACKs -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(4001,4500) s(i); -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001,5500) s(i); -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(6001,6500) s(i); -SAVEPOINT s3; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(7001,7500) s(i); -RELEASE s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(8001,8500) s(i); -ROLLBACK TO s1; -COMMIT; -}); - -$node_publisher->wait_for_catchup($appname); +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = apply)"); -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c) FROM test_tab"); -is($result, qq(2500|0), - 'check rollback to savepoint was reflected on subscriber'); - -# large (streamed) transaction with subscriber receiving rollback -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(8501,9000) s(i); -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(9001,9500) s(i); -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(9501,10000) s(i); -ROLLBACK; -}); +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c) FROM test_tab"); -is($result, qq(2500|0), 'check rollback was reflected on subscriber'); +test_streaming($node_publisher, $node_subscriber, $appname, 1); $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/019_stream_subxact_ddl_abort.pl b/src/test/subscription/t/019_stream_subxact_ddl_abort.pl index dbd0fca4d1..d0cd869430 100644 --- a/src/test/subscription/t/019_stream_subxact_ddl_abort.pl +++ b/src/test/subscription/t/019_stream_subxact_ddl_abort.pl @@ -9,6 +9,69 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" option +# so the same code can be run both for the streaming=on and streaming=apply +# cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_apply) = @_; + + # Check that a background worker starts if "streaming" option is + # specified as "apply". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_apply) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + # large (streamed) transaction with DDL, DML and ROLLBACKs + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i); + ALTER TABLE test_tab ADD COLUMN c INT; + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(501,1000) s(i); + ALTER TABLE test_tab ADD COLUMN d INT; + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001,1500) s(i); + ALTER TABLE test_tab ADD COLUMN e INT; + SAVEPOINT s3; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(1501,2000) s(i); + ALTER TABLE test_tab DROP COLUMN c; + ROLLBACK TO s1; + INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(501,1000) s(i); + COMMIT; + }); + + if ($is_apply) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c) FROM test_tab"); + is($result, qq(1000|500), + 'check rollback to savepoint was reflected on subscriber and extra columns contain local defaults' + ); + + # Cleanup the test data + $node_publisher->safe_psql( + 'postgres', q{ + DELETE FROM test_tab WHERE (a > 2); + ALTER TABLE test_tab DROP COLUMN c; + }); + $node_publisher->wait_for_catchup($appname); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -37,6 +100,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -54,35 +121,26 @@ my $result = "SELECT count(*), count(c) FROM test_tab"); is($result, qq(2|0), 'check initial data was copied to subscriber'); -# large (streamed) transaction with DDL, DML and ROLLBACKs -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i); -ALTER TABLE test_tab ADD COLUMN c INT; -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(501,1000) s(i); -ALTER TABLE test_tab ADD COLUMN d INT; -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001,1500) s(i); -ALTER TABLE test_tab ADD COLUMN e INT; -SAVEPOINT s3; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(1501,2000) s(i); -ALTER TABLE test_tab DROP COLUMN c; -ROLLBACK TO s1; -INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(501,1000) s(i); -COMMIT; -}); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c) FROM test_tab"); -is($result, qq(1000|500), - 'check rollback to savepoint was reflected on subscriber and extra columns contain local defaults' +################################### +# Test using streaming mode 'apply' +################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" ); +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = apply)"); + +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; + +test_streaming($node_publisher, $node_subscriber, $appname, 1); + $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/022_twophase_cascade.pl b/src/test/subscription/t/022_twophase_cascade.pl index 7a797f37ba..b52af74e35 100644 --- a/src/test/subscription/t/022_twophase_cascade.pl +++ b/src/test/subscription/t/022_twophase_cascade.pl @@ -11,6 +11,208 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" option +# so the same code can be run both for the streaming=on and streaming=apply +# cases. +sub test_streaming +{ + my ($node_A, $node_B, $node_C, $appname_B, $appname_C, $streaming_mode) = + @_; + + my $oldpid_B = $node_A->safe_psql( + 'postgres', " + SELECT pid FROM pg_stat_replication + WHERE application_name = '$appname_B' AND state = 'streaming';"); + my $oldpid_C = $node_B->safe_psql( + 'postgres', " + SELECT pid FROM pg_stat_replication + WHERE application_name = '$appname_C' AND state = 'streaming';"); + + # Setup logical replication streaming mode + + $node_B->safe_psql( + 'postgres', " + ALTER SUBSCRIPTION tap_sub_B + SET (streaming = $streaming_mode);"); + $node_C->safe_psql( + 'postgres', " + ALTER SUBSCRIPTION tap_sub_C + SET (streaming = $streaming_mode)"); + + # Wait for subscribers to finish initialization + + $node_A->poll_query_until( + 'postgres', " + SELECT pid != $oldpid_B FROM pg_stat_replication + WHERE application_name = '$appname_B' AND state = 'streaming';" + ) or die "Timed out while waiting for apply to restart"; + $node_B->poll_query_until( + 'postgres', " + SELECT pid != $oldpid_C FROM pg_stat_replication + WHERE application_name = '$appname_C' AND state = 'streaming';" + ) or die "Timed out while waiting for apply to restart"; + + ############################### + # Test 2PC PREPARE / COMMIT PREPARED. + # 1. Data is streamed as a 2PC transaction. + # 2. Then do commit prepared. + # + # Expect all data is replicated on subscriber(s) after the commit. + ############################### + + # Check that a background worker starts if "streaming" option is + # specified as "apply". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($streaming_mode eq 'apply') + { + $node_B->append_conf('postgresql.conf', "log_min_messages = debug1"); + $node_B->reload; + + $node_C->append_conf('postgresql.conf', "log_min_messages = debug1"); + $node_C->reload; + } + + # Insert, update and delete enough rows to exceed the 64kB limit. + # Then 2PC PREPARE + $node_A->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + if ($streaming_mode eq 'apply') + { + $node_B->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_B->append_conf('postgresql.conf', "log_min_messages = warning"); + $node_B->reload; + + $node_C->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_C->append_conf('postgresql.conf', "log_min_messages = warning"); + $node_C->reload; + } + + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); + + # check the transaction state is prepared on subscriber(s) + my $result = + $node_B->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber B'); + $result = + $node_C->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber C'); + + # 2PC COMMIT + $node_A->safe_psql('postgres', "COMMIT PREPARED 'test_prepared_tab';"); + + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); + + # check that transaction was committed on subscriber(s) + $result = $node_B->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), + 'Rows inserted by 2PC have committed on subscriber B, and extra columns have local defaults' + ); + $result = $node_C->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), + 'Rows inserted by 2PC have committed on subscriber C, and extra columns have local defaults' + ); + + # check the transaction state is ended on subscriber(s) + $result = + $node_B->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is committed on subscriber B'); + $result = + $node_C->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is committed on subscriber C'); + + ############################### + # Test 2PC PREPARE with a nested ROLLBACK TO SAVEPOINT. + # 0. Cleanup from previous test leaving only 2 rows. + # 1. Insert one more row. + # 2. Record a SAVEPOINT. + # 3. Data is streamed using 2PC. + # 4. Do rollback to SAVEPOINT prior to the streamed inserts. + # 5. Then COMMIT PREPARED. + # + # Expect data after the SAVEPOINT is aborted leaving only 3 rows (= 2 original + 1 from step 1). + ############################### + + # First, delete the data except for 2 rows (delete will be replicated) + $node_A->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); + + # 2PC PREPARE with a nested ROLLBACK TO SAVEPOINT + $node_A->safe_psql( + 'postgres', " + BEGIN; + INSERT INTO test_tab VALUES (9999, 'foobar'); + SAVEPOINT sp_inner; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + ROLLBACK TO SAVEPOINT sp_inner; + PREPARE TRANSACTION 'outer'; + "); + + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); + + # check the transaction state prepared on subscriber(s) + $result = + $node_B->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber B'); + $result = + $node_C->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber C'); + + # 2PC COMMIT + $node_A->safe_psql('postgres', "COMMIT PREPARED 'outer';"); + + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); + + # check the transaction state is ended on subscriber + $result = + $node_B->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is ended on subscriber B'); + $result = + $node_C->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is ended on subscriber C'); + + # check inserts are visible at subscriber(s). + # All the streamed data (prior to the SAVEPOINT) should be rolled back. + # (9999, 'foobar') should be committed. + $result = $node_B->safe_psql('postgres', + "SELECT count(*) FROM test_tab where b = 'foobar';"); + is($result, qq(1), 'Rows committed are present on subscriber B'); + $result = + $node_B->safe_psql('postgres', "SELECT count(*) FROM test_tab;"); + is($result, qq(3), 'Rows committed are present on subscriber B'); + $result = $node_C->safe_psql('postgres', + "SELECT count(*) FROM test_tab where b = 'foobar';"); + is($result, qq(1), 'Rows committed are present on subscriber C'); + $result = + $node_C->safe_psql('postgres', "SELECT count(*) FROM test_tab;"); + is($result, qq(3), 'Rows committed are present on subscriber C'); + + # Cleanup the test data + $node_A->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); +} + ############################### # Setup a cascade of pub/sub nodes. # node_A -> node_B -> node_C @@ -260,160 +462,15 @@ is($result, qq(21), 'Rows committed are present on subscriber C'); # 2PC + STREAMING TESTS # --------------------- -my $oldpid_B = $node_A->safe_psql( - 'postgres', " - SELECT pid FROM pg_stat_replication - WHERE application_name = '$appname_B' AND state = 'streaming';"); -my $oldpid_C = $node_B->safe_psql( - 'postgres', " - SELECT pid FROM pg_stat_replication - WHERE application_name = '$appname_C' AND state = 'streaming';"); - -# Setup logical replication (streaming = on) - -$node_B->safe_psql( - 'postgres', " - ALTER SUBSCRIPTION tap_sub_B - SET (streaming = on);"); -$node_C->safe_psql( - 'postgres', " - ALTER SUBSCRIPTION tap_sub_C - SET (streaming = on)"); - -# Wait for subscribers to finish initialization - -$node_A->poll_query_until( - 'postgres', " - SELECT pid != $oldpid_B FROM pg_stat_replication - WHERE application_name = '$appname_B' AND state = 'streaming';" -) or die "Timed out while waiting for apply to restart"; -$node_B->poll_query_until( - 'postgres', " - SELECT pid != $oldpid_C FROM pg_stat_replication - WHERE application_name = '$appname_C' AND state = 'streaming';" -) or die "Timed out while waiting for apply to restart"; - -############################### -# Test 2PC PREPARE / COMMIT PREPARED. -# 1. Data is streamed as a 2PC transaction. -# 2. Then do commit prepared. -# -# Expect all data is replicated on subscriber(s) after the commit. -############################### - -# Insert, update and delete enough rows to exceed the 64kB limit. -# Then 2PC PREPARE -$node_A->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_A->wait_for_catchup($appname_B); -$node_B->wait_for_catchup($appname_C); +################################ +# Test using streaming mode 'on' +################################ +test_streaming($node_A, $node_B, $node_C, $appname_B, $appname_C, 'on'); -# check the transaction state is prepared on subscriber(s) -$result = - $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber B'); -$result = - $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber C'); - -# 2PC COMMIT -$node_A->safe_psql('postgres', "COMMIT PREPARED 'test_prepared_tab';"); - -$node_A->wait_for_catchup($appname_B); -$node_B->wait_for_catchup($appname_C); - -# check that transaction was committed on subscriber(s) -$result = $node_B->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), - 'Rows inserted by 2PC have committed on subscriber B, and extra columns have local defaults' -); -$result = $node_C->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), - 'Rows inserted by 2PC have committed on subscriber C, and extra columns have local defaults' -); - -# check the transaction state is ended on subscriber(s) -$result = - $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is committed on subscriber B'); -$result = - $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is committed on subscriber C'); - -############################### -# Test 2PC PREPARE with a nested ROLLBACK TO SAVEPOINT. -# 0. Cleanup from previous test leaving only 2 rows. -# 1. Insert one more row. -# 2. Record a SAVEPOINT. -# 3. Data is streamed using 2PC. -# 4. Do rollback to SAVEPOINT prior to the streamed inserts. -# 5. Then COMMIT PREPARED. -# -# Expect data after the SAVEPOINT is aborted leaving only 3 rows (= 2 original + 1 from step 1). -############################### - -# First, delete the data except for 2 rows (delete will be replicated) -$node_A->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); - -# 2PC PREPARE with a nested ROLLBACK TO SAVEPOINT -$node_A->safe_psql( - 'postgres', " - BEGIN; - INSERT INTO test_tab VALUES (9999, 'foobar'); - SAVEPOINT sp_inner; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - ROLLBACK TO SAVEPOINT sp_inner; - PREPARE TRANSACTION 'outer'; - "); - -$node_A->wait_for_catchup($appname_B); -$node_B->wait_for_catchup($appname_C); - -# check the transaction state prepared on subscriber(s) -$result = - $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber B'); -$result = - $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber C'); - -# 2PC COMMIT -$node_A->safe_psql('postgres', "COMMIT PREPARED 'outer';"); - -$node_A->wait_for_catchup($appname_B); -$node_B->wait_for_catchup($appname_C); - -# check the transaction state is ended on subscriber -$result = - $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is ended on subscriber B'); -$result = - $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is ended on subscriber C'); - -# check inserts are visible at subscriber(s). -# All the streamed data (prior to the SAVEPOINT) should be rolled back. -# (9999, 'foobar') should be committed. -$result = $node_B->safe_psql('postgres', - "SELECT count(*) FROM test_tab where b = 'foobar';"); -is($result, qq(1), 'Rows committed are present on subscriber B'); -$result = $node_B->safe_psql('postgres', "SELECT count(*) FROM test_tab;"); -is($result, qq(3), 'Rows committed are present on subscriber B'); -$result = $node_C->safe_psql('postgres', - "SELECT count(*) FROM test_tab where b = 'foobar';"); -is($result, qq(1), 'Rows committed are present on subscriber C'); -$result = $node_C->safe_psql('postgres', "SELECT count(*) FROM test_tab;"); -is($result, qq(3), 'Rows committed are present on subscriber C'); +################################### +# Test using streaming mode 'apply' +################################### +test_streaming($node_A, $node_B, $node_C, $appname_B, $appname_C, 'apply'); ############################### # check all the cleanup diff --git a/src/test/subscription/t/023_twophase_stream.pl b/src/test/subscription/t/023_twophase_stream.pl index d8475d25a4..f55c5c95e7 100644 --- a/src/test/subscription/t/023_twophase_stream.pl +++ b/src/test/subscription/t/023_twophase_stream.pl @@ -8,6 +8,266 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" option +# so the same code can be run both for the streaming=on and streaming=apply +# cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_apply) = @_; + + ############################### + # Test 2PC PREPARE / COMMIT PREPARED. + # 1. Data is streamed as a 2PC transaction. + # 2. Then do commit prepared. + # + # Expect all data is replicated on subscriber side after the commit. + ############################### + + # Check that a background worker starts if "streaming" option is + # specified as "apply". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_apply) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + # check that 2PC gets replicated to subscriber + # Insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + if ($is_apply) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is in prepared state on subscriber + my $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber'); + + # 2PC transaction gets committed + $node_publisher->safe_psql('postgres', + "COMMIT PREPARED 'test_prepared_tab';"); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is committed on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), + 'Rows inserted by 2PC have committed on subscriber, and extra columns contain local defaults' + ); + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is committed on subscriber'); + + ############################### + # Test 2PC PREPARE / ROLLBACK PREPARED. + # 1. Table is deleted back to 2 rows which are replicated on subscriber. + # 2. Data is streamed using 2PC. + # 3. Do rollback prepared. + # + # Expect data rolls back leaving only the original 2 rows. + ############################### + + # First, delete the data except for 2 rows (will be replicated) + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE a > 2;"); + + # Then insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is in prepared state on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber'); + + # 2PC transaction gets aborted + $node_publisher->safe_psql('postgres', + "ROLLBACK PREPARED 'test_prepared_tab';"); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is aborted on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(2|2|2), + 'Rows inserted by 2PC are rolled back, leaving only the original 2 rows' + ); + + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is aborted on subscriber'); + + ############################### + # Check that 2PC COMMIT PREPARED is decoded properly on crash restart. + # 1. insert, update and delete enough rows to exceed the 64kB limit. + # 2. Then server crashes before the 2PC transaction is committed. + # 3. After servers are restarted the pending transaction is committed. + # + # Expect all data is replicated on subscriber side after the commit. + # Note: both publisher and subscriber do crash/restart. + ############################### + + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + $node_subscriber->stop('immediate'); + $node_publisher->stop('immediate'); + + $node_publisher->start; + $node_subscriber->start; + + # commit post the restart + $node_publisher->safe_psql('postgres', + "COMMIT PREPARED 'test_prepared_tab';"); + $node_publisher->wait_for_catchup($appname); + + # check inserts are visible + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), + 'Rows inserted by 2PC have committed on subscriber, and extra columns contain local defaults' + ); + + ############################### + # Do INSERT after the PREPARE but before ROLLBACK PREPARED. + # 1. Table is deleted back to 2 rows which are replicated on subscriber. + # 2. Data is streamed using 2PC. + # 3. A single row INSERT is done which is after the PREPARE. + # 4. Then do a ROLLBACK PREPARED. + # + # Expect the 2PC data rolls back leaving only 3 rows on the subscriber + # (the original 2 + inserted 1). + ############################### + + # First, delete the data except for 2 rows (will be replicated) + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE a > 2;"); + + # Then insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is in prepared state on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber'); + + # Insert a different record (now we are outside of the 2PC transaction) + # Note: the 2PC transaction still holds row locks so make sure this insert is for a separate primary key + $node_publisher->safe_psql('postgres', + "INSERT INTO test_tab VALUES (99999, 'foobar')"); + + # 2PC transaction gets aborted + $node_publisher->safe_psql('postgres', + "ROLLBACK PREPARED 'test_prepared_tab';"); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is aborted on subscriber, + # but the extra INSERT outside of the 2PC still was replicated + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3|3|3), + 'check the outside insert was copied to subscriber'); + + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is aborted on subscriber'); + + ############################### + # Do INSERT after the PREPARE but before COMMIT PREPARED. + # 1. Table is deleted back to 2 rows which are replicated on subscriber. + # 2. Data is streamed using 2PC. + # 3. A single row INSERT is done which is after the PREPARE. + # 4. Then do a COMMIT PREPARED. + # + # Expect 2PC data + the extra row are on the subscriber + # (the 3334 + inserted 1 = 3335). + ############################### + + # First, delete the data except for 2 rows (will be replicated) + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE a > 2;"); + + # Then insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is in prepared state on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber'); + + # Insert a different record (now we are outside of the 2PC transaction) + # Note: the 2PC transaction still holds row locks so make sure this insert is for a separare primary key + $node_publisher->safe_psql('postgres', + "INSERT INTO test_tab VALUES (99999, 'foobar')"); + + # 2PC transaction gets committed + $node_publisher->safe_psql('postgres', + "COMMIT PREPARED 'test_prepared_tab';"); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is committed on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3335|3335|3335), + 'Rows inserted by 2PC (as well as outside insert) have committed on subscriber, and extra columns contain local defaults' + ); + + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is committed on subscriber'); + + # Cleanup the test data + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE a > 2;"); + $node_publisher->wait_for_catchup($appname); +} + ############################### # Setup ############################### @@ -48,6 +308,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql( 'postgres', " CREATE SUBSCRIPTION tap_sub @@ -70,236 +334,30 @@ my $twophase_query = $node_subscriber->poll_query_until('postgres', $twophase_query) or die "Timed out while waiting for subscriber to enable twophase"; -############################### # Check initial data was copied to subscriber -############################### my $result = $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab"); is($result, qq(2|2|2), 'check initial data was copied to subscriber'); -############################### -# Test 2PC PREPARE / COMMIT PREPARED. -# 1. Data is streamed as a 2PC transaction. -# 2. Then do commit prepared. -# -# Expect all data is replicated on subscriber side after the commit. -############################### - -# check that 2PC gets replicated to subscriber -# Insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is in prepared state on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber'); - -# 2PC transaction gets committed -$node_publisher->safe_psql('postgres', - "COMMIT PREPARED 'test_prepared_tab';"); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is committed on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), - 'Rows inserted by 2PC have committed on subscriber, and extra columns contain local defaults' -); -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is committed on subscriber'); - -############################### -# Test 2PC PREPARE / ROLLBACK PREPARED. -# 1. Table is deleted back to 2 rows which are replicated on subscriber. -# 2. Data is streamed using 2PC. -# 3. Do rollback prepared. -# -# Expect data rolls back leaving only the original 2 rows. -############################### - -# First, delete the data except for 2 rows (will be replicated) -$node_publisher->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); - -# Then insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is in prepared state on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber'); - -# 2PC transaction gets aborted -$node_publisher->safe_psql('postgres', - "ROLLBACK PREPARED 'test_prepared_tab';"); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is aborted on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(2|2|2), - 'Rows inserted by 2PC are rolled back, leaving only the original 2 rows'); - -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is aborted on subscriber'); - -############################### -# Check that 2PC COMMIT PREPARED is decoded properly on crash restart. -# 1. insert, update and delete enough rows to exceed the 64kB limit. -# 2. Then server crashes before the 2PC transaction is committed. -# 3. After servers are restarted the pending transaction is committed. -# -# Expect all data is replicated on subscriber side after the commit. -# Note: both publisher and subscriber do crash/restart. -############################### - -$node_publisher->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_subscriber->stop('immediate'); -$node_publisher->stop('immediate'); - -$node_publisher->start; -$node_subscriber->start; - -# commit post the restart -$node_publisher->safe_psql('postgres', - "COMMIT PREPARED 'test_prepared_tab';"); -$node_publisher->wait_for_catchup($appname); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -# check inserts are visible -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), - 'Rows inserted by 2PC have committed on subscriber, and extra columns contain local defaults' +################################### +# Test using streaming mode 'apply' +################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" ); -############################### -# Do INSERT after the PREPARE but before ROLLBACK PREPARED. -# 1. Table is deleted back to 2 rows which are replicated on subscriber. -# 2. Data is streamed using 2PC. -# 3. A single row INSERT is done which is after the PREPARE. -# 4. Then do a ROLLBACK PREPARED. -# -# Expect the 2PC data rolls back leaving only 3 rows on the subscriber -# (the original 2 + inserted 1). -############################### - -# First, delete the data except for 2 rows (will be replicated) -$node_publisher->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); - -# Then insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is in prepared state on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber'); - -# Insert a different record (now we are outside of the 2PC transaction) -# Note: the 2PC transaction still holds row locks so make sure this insert is for a separate primary key -$node_publisher->safe_psql('postgres', - "INSERT INTO test_tab VALUES (99999, 'foobar')"); - -# 2PC transaction gets aborted -$node_publisher->safe_psql('postgres', - "ROLLBACK PREPARED 'test_prepared_tab';"); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is aborted on subscriber, -# but the extra INSERT outside of the 2PC still was replicated -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3|3|3), 'check the outside insert was copied to subscriber'); - -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is aborted on subscriber'); - -############################### -# Do INSERT after the PREPARE but before COMMIT PREPARED. -# 1. Table is deleted back to 2 rows which are replicated on subscriber. -# 2. Data is streamed using 2PC. -# 3. A single row INSERT is done which is after the PREPARE. -# 4. Then do a COMMIT PREPARED. -# -# Expect 2PC data + the extra row are on the subscriber -# (the 3334 + inserted 1 = 3335). -############################### - -# First, delete the data except for 2 rows (will be replicated) -$node_publisher->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); - -# Then insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is in prepared state on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber'); - -# Insert a different record (now we are outside of the 2PC transaction) -# Note: the 2PC transaction still holds row locks so make sure this insert is for a separare primary key -$node_publisher->safe_psql('postgres', - "INSERT INTO test_tab VALUES (99999, 'foobar')"); - -# 2PC transaction gets committed -$node_publisher->safe_psql('postgres', - "COMMIT PREPARED 'test_prepared_tab';"); - -$node_publisher->wait_for_catchup($appname); +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = apply)"); -# check that transaction is committed on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3335|3335|3335), - 'Rows inserted by 2PC (as well as outside insert) have committed on subscriber, and extra columns contain local defaults' -); +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is committed on subscriber'); +test_streaming($node_publisher, $node_subscriber, $appname, 1); ############################### # check all the cleanup -- 2.23.0.windows.1 [application/octet-stream] v8-0003-Add-some-checks-before-using-apply-background-wor.patch (26.9K, ../../OS3PR01MB62758A881FF3240171B7B21B9EDE9@OS3PR01MB6275.jpnprd01.prod.outlook.com/4-v8-0003-Add-some-checks-before-using-apply-background-wor.patch) download | inline diff: From 4f0505955d603a537dcad9e399449ce6e8f126d3 Mon Sep 17 00:00:00 2001 From: wangw <[email protected]> Date: Thu, 2 Jun 2022 17:03:02 +0800 Subject: [PATCH v8 3/4] Add some checks before using apply background worker to apply changes. If any of the following checks are violated, an error will be reported. 1. The unique columns between publisher and subscriber are difference. 2. There is any non-immutable function present in expression in subscriber's relation. Check from the following 3 items: a. The function in triggers; b. Column default value expressions and domain constraints; c. Constraint expressions. --- doc/src/sgml/ref/create_subscription.sgml | 4 + .../replication/logical/applybgwroker.c | 46 ++++ src/backend/replication/logical/proto.c | 62 +++++- src/backend/replication/logical/relation.c | 183 +++++++++++++++- src/backend/replication/logical/tablesync.c | 1 + src/backend/replication/logical/worker.c | 6 + src/backend/utils/cache/typcache.c | 17 ++ src/include/replication/logicalproto.h | 1 + src/include/replication/logicalrelation.h | 15 ++ src/include/replication/worker_internal.h | 1 + src/include/utils/typcache.h | 2 + .../subscription/t/022_twophase_cascade.pl | 7 + .../subscription/t/032_streaming_apply.pl | 206 ++++++++++++++++++ 13 files changed, 546 insertions(+), 5 deletions(-) create mode 100644 src/test/subscription/t/032_streaming_apply.pl diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml index d4caae0222..8de1a23ce4 100644 --- a/doc/src/sgml/ref/create_subscription.sgml +++ b/doc/src/sgml/ref/create_subscription.sgml @@ -240,6 +240,10 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl the transaction is committed. Note that if an error happens when applying changes in a background worker, it might not report the finish LSN of the remote transaction in the server log. + To run in this mode, there are following two requirements. The first + is that the unique column should be the same between publisher and + subscriber; the second is that there should not be any non-immutable + function in subscriber-side replicated table. </para> </listitem> </varlistentry> diff --git a/src/backend/replication/logical/applybgwroker.c b/src/backend/replication/logical/applybgwroker.c index 9fcc600227..4184343f6c 100644 --- a/src/backend/replication/logical/applybgwroker.c +++ b/src/backend/replication/logical/applybgwroker.c @@ -741,3 +741,49 @@ apply_bgworker_subxact_info_add(TransactionId current_xid) MemoryContextSwitchTo(oldctx); } } + +/* + * Check if changes on this logical replication relation can be applied by + * apply background worker. + * + * Although we maintains the commit order by allowing only one process to + * commit at a time, our access order to the relation has changed. + * This could cause unexpected problems if the unique column on the replicated + * table is inconsistent with the publisher-side or contains non-immutable + * functions when applying transactions in the apply background worker. + */ +void +apply_bgworker_relation_check(LogicalRepRelMapEntry *rel) +{ + /* Check only we are in apply bgworker. */ + if (!am_apply_bgworker()) + return; + + /* + * FIXME: Checks on partitioned tables are currently not supported. Adding + * this check requires additional fixes for other issues. I will add this + * later, after the related issues are fixed. + */ + if (rel->localrel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + return; + + /* + * If any unique index exist, check that they are same as remoterel. + */ + if (!rel->sameunique) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot replicate relation with different unique index"), + errhint("Please change the streaming option to 'on' instead of 'apply'."))); + + /* + * Check if there is any non-immutable function present in expression in + * this relation. + */ + if (rel->volatility == FUNCTION_NONIMMUTABLE) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot replicate relation. There is at least one non-immutable function"), + errhint("Please change the streaming option to 'on' instead of 'apply'."))); + +} diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c index ff8513e2d2..20d18f2bed 100644 --- a/src/backend/replication/logical/proto.c +++ b/src/backend/replication/logical/proto.c @@ -23,7 +23,8 @@ /* * Protocol message flags. */ -#define LOGICALREP_IS_REPLICA_IDENTITY 1 +#define LOGICALREP_IS_REPLICA_IDENTITY 0x0001 +#define LOGICALREP_IS_UNIQUE 0x0002 #define MESSAGE_TRANSACTIONAL (1<<0) #define TRUNCATE_CASCADE (1<<0) @@ -933,11 +934,55 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns) TupleDesc desc; int i; uint16 nliveatts = 0; - Bitmapset *idattrs = NULL; + Bitmapset *idattrs = NULL, + *attunique = NULL; bool replidentfull; desc = RelationGetDescr(rel); + if (rel->rd_rel->relhasindex) + { + List *indexoidlist = RelationGetIndexList(rel); + ListCell *indexoidscan; + + foreach(indexoidscan, indexoidlist) + { + Oid indexoid = lfirst_oid(indexoidscan); + Relation indexRel; + + /* Look up the description for index */ + indexRel = RelationIdGetRelation(indexoid); + + if (!RelationIsValid(indexRel)) + elog(ERROR, "could not open relation with OID %u", indexoid); + + if (indexRel->rd_index->indisunique) + { + int i; + + /* Add referenced attributes to idindexattrs */ + for (i = 0; i < indexRel->rd_index->indnatts; i++) + { + int attrnum = indexRel->rd_index->indkey.values[i]; + + /* + * We don't include non-key columns into idindexattrs + * bitmaps. See RelationGetIndexAttrBitmap. + */ + if (attrnum != 0) + { + if (i < indexRel->rd_index->indnkeyatts && + !bms_is_member(attrnum - FirstLowInvalidHeapAttributeNumber, attunique)) + attunique = bms_add_member(attunique, + attrnum - FirstLowInvalidHeapAttributeNumber); + } + } + } + RelationClose(indexRel); + } + list_free(indexoidlist); + } + /* send number of live attributes */ for (i = 0; i < desc->natts; i++) { @@ -976,6 +1021,10 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns) idattrs)) flags |= LOGICALREP_IS_REPLICA_IDENTITY; + if (bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber, + attunique)) + flags |= LOGICALREP_IS_UNIQUE; + pq_sendbyte(out, flags); /* attribute name */ @@ -1001,7 +1050,8 @@ logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel) int natts; char **attnames; Oid *atttyps; - Bitmapset *attkeys = NULL; + Bitmapset *attkeys = NULL, + *attunique = NULL; natts = pq_getmsgint(in, 2); attnames = palloc(natts * sizeof(char *)); @@ -1012,11 +1062,14 @@ logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel) { uint8 flags; - /* Check for replica identity column */ + /* Check for replica identity and unique column */ flags = pq_getmsgbyte(in); if (flags & LOGICALREP_IS_REPLICA_IDENTITY) attkeys = bms_add_member(attkeys, i); + if (flags & LOGICALREP_IS_UNIQUE) + attunique = bms_add_member(attunique, i); + /* attribute name */ attnames[i] = pstrdup(pq_getmsgstring(in)); @@ -1030,6 +1083,7 @@ logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel) rel->attnames = attnames; rel->atttyps = atttyps; rel->attkeys = attkeys; + rel->attunique = attunique; rel->natts = natts; } diff --git a/src/backend/replication/logical/relation.c b/src/backend/replication/logical/relation.c index 80fb561a9a..2736ba1f4a 100644 --- a/src/backend/replication/logical/relation.c +++ b/src/backend/replication/logical/relation.c @@ -19,12 +19,18 @@ #include "access/table.h" #include "catalog/namespace.h" +#include "catalog/pg_proc.h" #include "catalog/pg_subscription_rel.h" #include "executor/executor.h" #include "nodes/makefuncs.h" +#include "optimizer/optimizer.h" #include "replication/logicalrelation.h" #include "replication/worker_internal.h" +#include "rewrite/rewriteHandler.h" #include "utils/inval.h" +#include "utils/lsyscache.h" +#include "utils/syscache.h" +#include "utils/typcache.h" static MemoryContext LogicalRepRelMapContext = NULL; @@ -91,6 +97,23 @@ logicalrep_relmap_invalidate_cb(Datum arg, Oid reloid) } } +/* + * Reset the flag volatility of all existing entry in the relation map cache. + */ +static void +logicalrep_relmap_reset_volatility_cb(Datum arg, int cacheid, uint32 hashvalue) +{ + HASH_SEQ_STATUS hash_seq; + LogicalRepRelMapEntry *entry; + + if (LogicalRepRelMap == NULL) + return; + + hash_seq_init(&hash_seq, LogicalRepRelMap); + while ((entry = hash_seq_search(&hash_seq)) != NULL) + entry->volatility = FUNCTION_UNKNOWN; +} + /* * Initialize the relation map cache. */ @@ -116,6 +139,9 @@ logicalrep_relmap_init(void) /* Watch for invalidation events. */ CacheRegisterRelcacheCallback(logicalrep_relmap_invalidate_cb, (Datum) 0); + CacheRegisterSyscacheCallback(PROCOID, + logicalrep_relmap_reset_volatility_cb, + (Datum) 0); } /* @@ -142,6 +168,7 @@ logicalrep_relmap_free_entry(LogicalRepRelMapEntry *entry) pfree(remoterel->atttyps); } bms_free(remoterel->attkeys); + bms_free(remoterel->attunique); if (entry->attrmap) pfree(entry->attrmap); @@ -190,6 +217,7 @@ logicalrep_relmap_update(LogicalRepRelation *remoterel) } entry->remoterel.replident = remoterel->replident; entry->remoterel.attkeys = bms_copy(remoterel->attkeys); + entry->remoterel.attunique = bms_copy(remoterel->attunique); MemoryContextSwitchTo(oldctx); } @@ -307,7 +335,8 @@ logicalrep_rel_open(LogicalRepRelId remoteid, LOCKMODE lockmode) if (!entry->localrelvalid) { Oid relid; - Bitmapset *idkey; + Bitmapset *idkey, + *ukey; TupleDesc desc; MemoryContext oldctx; int i; @@ -415,6 +444,157 @@ logicalrep_rel_open(LogicalRepRelId remoteid, LOCKMODE lockmode) } } + /* + * Check whether the unique index of publisher and subscriber are + * consistent. + */ + entry->sameunique = true; + ukey = RelationGetIndexAttrBitmap(entry->localrel, + INDEX_ATTR_BITMAP_KEY); + + if (ukey) + { + i = -1; + while ((i = bms_next_member(ukey, i)) >= 0) + { + int attnum = i + FirstLowInvalidHeapAttributeNumber; + + attnum = AttrNumberGetAttrOffset(attnum); + + if (entry->attrmap->attnums[attnum] < 0 || + !bms_is_member(entry->attrmap->attnums[attnum], remoterel->attunique)) + { + entry->sameunique = false; + break; + } + } + } + + if (entry->volatility == FUNCTION_UNKNOWN) + { + /* + * Check from the following points: + * + * a. The function in triggers; + * b. Column default value expressions and domain constraints; + * c. Constraint expressions. + */ + bool nonimmutable = false; + + /* Check the trigger functions. */ + if (!nonimmutable) + { + int i; + + if (entry->localrel->trigdesc != NULL) + for (i = 0; i < entry->localrel->trigdesc->numtriggers; i++) + { + Trigger *trig = entry->localrel->trigdesc->triggers + i; + + if (func_volatile(trig->tgfoid) != PROVOLATILE_IMMUTABLE) + { + nonimmutable = true; + break; + } + } + } + + /* Check the columns. */ + if (!nonimmutable) + { + TupleDesc tupdesc = RelationGetDescr(entry->localrel); + int attnum; + + for (attnum = 0; attnum < tupdesc->natts; attnum++) + { + Form_pg_attribute att = TupleDescAttr(tupdesc, attnum); + + /* break if we set nonimmutable in last check for domain. */ + if (nonimmutable) + break; + + /* We don't need info for dropped or generated attributes */ + if (att->attisdropped || att->attgenerated) + continue; + + /* + * We don't need to check columns that only exist on the + * subscriber + */ + if (entry->attrmap->attnums[attnum] < 0) + continue; + + if (att->atthasdef) + { + Node *defaultexpr; + + defaultexpr = build_column_default(entry->localrel, attnum + 1); + if (contain_mutable_functions(defaultexpr)) + { + nonimmutable = true; + break; + } + } + + /* + * If the column is of a DOMAIN type, determine whether + * that domain has any CHECK expressions that are not + * immutable. + */ + if (get_typtype(att->atttypid) == TYPTYPE_DOMAIN) + { + List *domain_constraints; + ListCell *lc; + + domain_constraints = GetDomainConstraints(att->atttypid); + + foreach(lc, domain_constraints) + { + DomainConstraintState *con = (DomainConstraintState *) lfirst(lc); + + if (con->check_expr && contain_mutable_functions((Node *) con->check_expr)) + { + nonimmutable = true; + break; + } + } + } + } + } + + /* Check the constraints. */ + if (!nonimmutable) + { + TupleDesc tupdesc = RelationGetDescr(entry->localrel); + + if (tupdesc->constr) + { + ConstrCheck *check = tupdesc->constr->check; + int i; + + /* + * Determine if there are any CHECK constraints which + * contains non-immutable function. + */ + for (i = 0; i < tupdesc->constr->num_check; i++) + { + Expr *check_expr = stringToNode(check[i].ccbin); + + if (contain_mutable_functions((Node *) check_expr)) + { + nonimmutable = true; + break; + } + } + } + } + + if (nonimmutable) + entry->volatility = FUNCTION_NONIMMUTABLE; + else + entry->volatility = FUNCTION_IMMUTABLE; + } + entry->localrelvalid = true; } @@ -570,6 +750,7 @@ logicalrep_partition_open(LogicalRepRelMapEntry *root, } entry->remoterel.replident = remoterel->replident; entry->remoterel.attkeys = bms_copy(remoterel->attkeys); + entry->remoterel.attunique = bms_copy(remoterel->attunique); entry->localrel = partrel; entry->localreloid = partOid; diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c index 8ffba7e2e5..3cdbf8b457 100644 --- a/src/backend/replication/logical/tablesync.c +++ b/src/backend/replication/logical/tablesync.c @@ -884,6 +884,7 @@ fetch_remote_table_info(char *nspname, char *relname, lrel->attnames = palloc0(MaxTupleAttributeNumber * sizeof(char *)); lrel->atttyps = palloc0(MaxTupleAttributeNumber * sizeof(Oid)); lrel->attkeys = NULL; + lrel->attunique = NULL; /* * Store the columns as a list of names. Ignore those that are not diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c index 20c7865275..7355b30bfb 100644 --- a/src/backend/replication/logical/worker.c +++ b/src/backend/replication/logical/worker.c @@ -1896,6 +1896,8 @@ apply_handle_insert(StringInfo s) /* Set relation for error callback */ apply_error_callback_arg.rel = rel; + apply_bgworker_relation_check(rel); + /* Initialize the executor state. */ edata = create_edata_for_relation(rel); estate = edata->estate; @@ -2032,6 +2034,8 @@ apply_handle_update(StringInfo s) /* Check if we can do the update. */ check_relation_updatable(rel); + apply_bgworker_relation_check(rel); + /* Initialize the executor state. */ edata = create_edata_for_relation(rel); estate = edata->estate; @@ -2200,6 +2204,8 @@ apply_handle_delete(StringInfo s) /* Check if we can do the delete. */ check_relation_updatable(rel); + apply_bgworker_relation_check(rel); + /* Initialize the executor state. */ edata = create_edata_for_relation(rel); estate = edata->estate; diff --git a/src/backend/utils/cache/typcache.c b/src/backend/utils/cache/typcache.c index 808f9ebd0d..b248899d82 100644 --- a/src/backend/utils/cache/typcache.c +++ b/src/backend/utils/cache/typcache.c @@ -2540,6 +2540,23 @@ compare_values_of_enum(TypeCacheEntry *tcache, Oid arg1, Oid arg2) return 0; } +/* + * GetDomainConstraints --- get DomainConstraintState list of specified domain type + */ +List * +GetDomainConstraints(Oid type_id) +{ + TypeCacheEntry *typentry; + List *constraints = NIL; + + typentry = lookup_type_cache(type_id, TYPECACHE_DOMAIN_CONSTR_INFO); + + if(typentry->domainData != NULL) + constraints = typentry->domainData->constraints; + + return constraints; +} + /* * Load (or re-load) the enumData member of the typcache entry. */ diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h index a771ab8ff3..a88535820b 100644 --- a/src/include/replication/logicalproto.h +++ b/src/include/replication/logicalproto.h @@ -108,6 +108,7 @@ typedef struct LogicalRepRelation char replident; /* replica identity */ char relkind; /* remote relation kind */ Bitmapset *attkeys; /* Bitmap of key columns */ + Bitmapset *attunique; /* Bitmap of unique columns */ } LogicalRepRelation; /* Type mapping info */ diff --git a/src/include/replication/logicalrelation.h b/src/include/replication/logicalrelation.h index 7bf8cd22bd..9f568e9c00 100644 --- a/src/include/replication/logicalrelation.h +++ b/src/include/replication/logicalrelation.h @@ -15,6 +15,17 @@ #include "access/attmap.h" #include "replication/logicalproto.h" +/* + * States to determine volatility of the function in expressions in one + * relation. + */ +typedef enum RelFuncVolatility +{ + FUNCTION_UNKNOWN = 0, /* initializing */ + FUNCTION_IMMUTABLE, /* all functions are immutable function */ + FUNCTION_NONIMMUTABLE /* at least one non-immutable function */ +} RelFuncVolatility; + typedef struct LogicalRepRelMapEntry { LogicalRepRelation remoterel; /* key is remoterel.remoteid */ @@ -31,6 +42,10 @@ typedef struct LogicalRepRelMapEntry Relation localrel; /* relcache entry (NULL when closed) */ AttrMap *attrmap; /* map of local attributes to remote ones */ bool updatable; /* Can apply updates/deletes? */ + bool sameunique; /* Is the unique column of local and remote + consistent? */ + RelFuncVolatility volatility; /* all functions in localrel are + immutable function? */ /* Sync state. */ char state; diff --git a/src/include/replication/worker_internal.h b/src/include/replication/worker_internal.h index e2d21444c0..1dc1a491e9 100644 --- a/src/include/replication/worker_internal.h +++ b/src/include/replication/worker_internal.h @@ -192,6 +192,7 @@ extern void apply_bgworker_free(ApplyBgworkerState *wstate); extern void apply_bgworker_check_status(void); extern void apply_bgworker_set_status(ApplyBgworkerStatus status); extern void apply_bgworker_subxact_info_add(TransactionId current_xid); +extern void apply_bgworker_relation_check(LogicalRepRelMapEntry *rel); static inline bool am_tablesync_worker(void) diff --git a/src/include/utils/typcache.h b/src/include/utils/typcache.h index 431ad7f1b3..ed7c2e7f48 100644 --- a/src/include/utils/typcache.h +++ b/src/include/utils/typcache.h @@ -199,6 +199,8 @@ extern uint64 assign_record_type_identifier(Oid type_id, int32 typmod); extern int compare_values_of_enum(TypeCacheEntry *tcache, Oid arg1, Oid arg2); +extern List *GetDomainConstraints(Oid type_id); + extern size_t SharedRecordTypmodRegistryEstimate(void); extern void SharedRecordTypmodRegistryInit(SharedRecordTypmodRegistry *, diff --git a/src/test/subscription/t/022_twophase_cascade.pl b/src/test/subscription/t/022_twophase_cascade.pl index b52af74e35..81791c8d3a 100644 --- a/src/test/subscription/t/022_twophase_cascade.pl +++ b/src/test/subscription/t/022_twophase_cascade.pl @@ -39,6 +39,13 @@ sub test_streaming ALTER SUBSCRIPTION tap_sub_C SET (streaming = $streaming_mode)"); + if ($streaming_mode eq 'apply') + { + $node_C->safe_psql( + 'postgres', " + ALTER TABLE test_tab ALTER c DROP DEFAULT"); + } + # Wait for subscribers to finish initialization $node_A->poll_query_until( diff --git a/src/test/subscription/t/032_streaming_apply.pl b/src/test/subscription/t/032_streaming_apply.pl new file mode 100644 index 0000000000..764ea2ed59 --- /dev/null +++ b/src/test/subscription/t/032_streaming_apply.pl @@ -0,0 +1,206 @@ +# Copyright (c) 2022, PostgreSQL Global Development Group + +# Test the restrictions of streaming mode "apply" in logical replication + +use strict; +use warnings; +use PostgreSQL::Test::Cluster; +use PostgreSQL::Test::Utils; +use Test::More; + +my $offset = 0; + +# Create publisher node +my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); +$node_publisher->init(allows_streaming => 'logical'); +$node_publisher->append_conf('postgresql.conf', + 'logical_decoding_work_mem = 64kB'); +$node_publisher->start; + +# Create subscriber node +my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber'); +$node_subscriber->init; +$node_subscriber->start; + +# Setup structure on publisher +$node_publisher->safe_psql('postgres', + "CREATE TABLE test_tab (a int primary key, b varchar)"); + +# Setup structure on subscriber +$node_subscriber->safe_psql('postgres', + "CREATE TABLE test_tab (a int primary key, b varchar)"); + +# Setup logical replication +my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres'; +$node_publisher->safe_psql('postgres', + "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); + +my $appname = 'tap_sub'; +$node_subscriber->safe_psql( + 'postgres', " + CREATE SUBSCRIPTION tap_sub + CONNECTION '$publisher_connstr application_name=$appname' + PUBLICATION tap_pub + WITH (streaming = apply, copy_data = false)"); + +$node_publisher->wait_for_catchup($appname); + +# It is not allowed that the unique index on the publisher and the subscriber +# is different. Check the error reported by background worker in this case. +$node_subscriber->safe_psql('postgres', + "CREATE UNIQUE INDEX test_tab_b_idx ON test_tab (b)"); + +# Check that a background worker starts if "streaming" option is +# specified as "apply". We have to look for the DEBUG1 log messages +# about that, so temporarily bump up the log verbosity. +$node_subscriber->append_conf('postgresql.conf', "log_min_messages = debug1"); +$node_subscriber->reload; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); +$node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); +$node_subscriber->reload; + +$node_subscriber->wait_for_log( + qr/ERROR: cannot replicate relation with different unique index/, + $offset); + +# Drop the unique index on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', "DROP INDEX test_tab_b_idx"); + +$node_publisher->wait_for_catchup($appname); + +my $result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); +is($result, qq(5000), 'data replicated to subscriber after dropping index'); + +# Triggers which execute non-immutable function are not allowed on the +# subscriber side. Check the error reported by background worker in this case. +$node_subscriber->safe_psql( + 'postgres', qq{ +CREATE FUNCTION trigger_func() RETURNS TRIGGER AS \$\$ + BEGIN + RETURN NULL; + END +\$\$ language plpgsql; +CREATE TRIGGER insert_trig +BEFORE INSERT ON test_tab +FOR EACH ROW EXECUTE PROCEDURE trigger_func(); +ALTER TABLE test_tab ENABLE REPLICA TRIGGER insert_trig; +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', "DELETE FROM test_tab"); + +$node_subscriber->wait_for_log( + qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, + $offset); + +# Drop the trigger on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "DROP TRIGGER insert_trig ON test_tab"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); +is($result, qq(0), 'data replicated to subscriber after dropping trigger'); + +# It is not allowed that column default value expression contains a +# non-immutable function on the subscriber side. Check the error reported by +# background worker in this case. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab ALTER COLUMN b SET DEFAULT random()"); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log( + qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, + $offset); + +# Drop default value on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab ALTER COLUMN b DROP DEFAULT"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); +is($result, qq(5000), + 'data replicated to subscriber after dropping default value expression'); + +# It is not allowed that domain constraint expression contains a non-immutable +# function on the subscriber side. Check the error reported by background +# worker in this case. +$node_subscriber->safe_psql( + 'postgres', qq{ +CREATE DOMAIN test_domain AS int CHECK(VALUE > random()); +ALTER TABLE test_tab ALTER COLUMN a TYPE test_domain; +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', "DELETE FROM test_tab"); + +$node_subscriber->wait_for_log( + qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, + $offset); + +# Drop domain constraint expression on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab ALTER COLUMN a TYPE int"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); +is($result, qq(0), + 'data replicated to subscriber after dropping domain constraint expression' +); + +# It is not allowed that constraint expression contains a non-immutable function +# on the subscriber side. Check the error reported by background worker in this +# case. +$node_subscriber->safe_psql( + 'postgres', qq{ +ALTER TABLE test_tab ADD CONSTRAINT test_tab_con check (a > random()); +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log( + qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, + $offset); + +# Drop default value on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab DROP CONSTRAINT test_tab_con"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); +is($result, qq(5000), + 'data replicated to subscriber after dropping constraint expression'); + +$node_subscriber->stop; +$node_publisher->stop; + +done_testing(); -- 2.23.0.windows.1 [application/octet-stream] v8-0004-Add-a-GUC-max_apply_bgworkers_per_subscription-to.patch (6.9K, ../../OS3PR01MB62758A881FF3240171B7B21B9EDE9@OS3PR01MB6275.jpnprd01.prod.outlook.com/5-v8-0004-Add-a-GUC-max_apply_bgworkers_per_subscription-to.patch) download | inline diff: From 0cf3079200abdbe428ce55b7997e2345c8a40471 Mon Sep 17 00:00:00 2001 From: wangw <[email protected]> Date: Wed, 1 Jun 2022 19:59:27 +0800 Subject: [PATCH v8 4/4] Add a GUC "max_apply_bgworkers_per_subscription" to control parallelism. This GUC controls how many apply background workers can be launched per subscription. --- doc/src/sgml/config.sgml | 25 +++++++++++++++++++ .../replication/logical/applybgwroker.c | 9 +++++++ src/backend/replication/logical/launcher.c | 25 +++++++++++++++++++ src/backend/utils/misc/guc.c | 12 +++++++++ src/backend/utils/misc/postgresql.conf.sample | 1 + src/include/replication/logicallauncher.h | 1 + src/include/replication/worker_internal.h | 1 + 7 files changed, 74 insertions(+) diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml index 5b7ce6531d..ce7f6827f2 100644 --- a/doc/src/sgml/config.sgml +++ b/doc/src/sgml/config.sgml @@ -4970,6 +4970,31 @@ ANY <replaceable class="parameter">num_sync</replaceable> ( <replaceable class=" </listitem> </varlistentry> + <varlistentry id="guc-max-apply-bgworkers-per-subscription" xreflabel="max_apply_bgworkers_per_subscription"> + <term><varname>max_apply_bgworkers_per_subscription</varname> (<type>integer</type>) + <indexterm> + <primary><varname>max_apply_bgworkers_per_subscription</varname> configuration parameter</primary> + </indexterm> + </term> + <listitem> + <para> + Maximum number of apply background workers per subscription. This + parameter controls the amount of parallelism of the streaming of + in-progress transactions if we set subscription option + <literal>streaming</literal> to <literal>apply</literal>. + </para> + <para> + The apply background workers are taken from the pool defined by + <varname>max_logical_replication_workers</varname>. + </para> + <para> + The default value is 3. This parameter can only be set in the + <filename>postgresql.conf</filename> file or on the server command + line. + </para> + </listitem> + </varlistentry> + </variablelist> </sect2> diff --git a/src/backend/replication/logical/applybgwroker.c b/src/backend/replication/logical/applybgwroker.c index 4184343f6c..82861b0bfd 100644 --- a/src/backend/replication/logical/applybgwroker.c +++ b/src/backend/replication/logical/applybgwroker.c @@ -21,6 +21,7 @@ #include "mb/pg_wchar.h" #include "pgstat.h" #include "postmaster/interrupt.h" +#include "replication/logicallauncher.h" #include "replication/logicalworker.h" #include "replication/origin.h" #include "replication/worker_internal.h" @@ -558,9 +559,17 @@ apply_bgworker_setup(void) MemoryContext oldcontext; bool launched; ApplyBgworkerState *wstate; + int napplyworkers; elog(DEBUG1, "setting up apply worker #%u", list_length(ApplyWorkersList) + 1); + /* Check If there are free worker slot(s) */ + LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); + napplyworkers = logicalrep_apply_background_worker_count(MyLogicalRepWorker->subid); + LWLockRelease(LogicalRepWorkerLock); + if (napplyworkers >= max_apply_bgworkers_per_subscription) + return NULL; + oldcontext = MemoryContextSwitchTo(ApplyContext); wstate = (ApplyBgworkerState *) palloc0(sizeof(ApplyBgworkerState)); diff --git a/src/backend/replication/logical/launcher.c b/src/backend/replication/logical/launcher.c index e810875ddd..67dd40598c 100644 --- a/src/backend/replication/logical/launcher.c +++ b/src/backend/replication/logical/launcher.c @@ -54,6 +54,7 @@ int max_logical_replication_workers = 4; int max_sync_workers_per_subscription = 2; +int max_apply_bgworkers_per_subscription = 3; LogicalRepWorker *MyLogicalRepWorker = NULL; @@ -736,6 +737,30 @@ logicalrep_sync_worker_count(Oid subid) return res; } +/* + * Count the number of registered (not necessarily running) apply background + * worker for a subscription. + */ +int +logicalrep_apply_background_worker_count(Oid subid) +{ + int i; + int res = 0; + + Assert(LWLockHeldByMe(LogicalRepWorkerLock)); + + /* Search for attached worker for a given subscription id. */ + for (i = 0; i < max_logical_replication_workers; i++) + { + LogicalRepWorker *w = &LogicalRepCtx->workers[i]; + + if (w->subid == subid && w->subworker) + res++; + } + + return res; +} + /* * ApplyLauncherShmemSize * Compute space needed for replication launcher shared memory diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c index 55d41ae7d6..539ec07ff5 100644 --- a/src/backend/utils/misc/guc.c +++ b/src/backend/utils/misc/guc.c @@ -3220,6 +3220,18 @@ static struct config_int ConfigureNamesInt[] = NULL, NULL, NULL }, + { + {"max_apply_bgworkers_per_subscription", + PGC_SIGHUP, + REPLICATION_SUBSCRIBERS, + gettext_noop("Maximum number of apply backgrand workers per subscription."), + NULL, + }, + &max_apply_bgworkers_per_subscription, + 3, 0, MAX_BACKENDS, + NULL, NULL, NULL + }, + { {"log_rotation_age", PGC_SIGHUP, LOGGING_WHERE, gettext_noop("Sets the amount of time to wait before forcing " diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample index 48ad80cf2e..b71340549d 100644 --- a/src/backend/utils/misc/postgresql.conf.sample +++ b/src/backend/utils/misc/postgresql.conf.sample @@ -360,6 +360,7 @@ #max_logical_replication_workers = 4 # taken from max_worker_processes # (change requires restart) #max_sync_workers_per_subscription = 2 # taken from max_logical_replication_workers +#max_apply_bgworkers_per_subscription = 3 # taken from max_logical_replication_workers #------------------------------------------------------------------------------ diff --git a/src/include/replication/logicallauncher.h b/src/include/replication/logicallauncher.h index f1e2821e25..ac8ef94381 100644 --- a/src/include/replication/logicallauncher.h +++ b/src/include/replication/logicallauncher.h @@ -14,6 +14,7 @@ extern PGDLLIMPORT int max_logical_replication_workers; extern PGDLLIMPORT int max_sync_workers_per_subscription; +extern PGDLLIMPORT int max_apply_bgworkers_per_subscription; extern void ApplyLauncherRegister(void); extern void ApplyLauncherMain(Datum main_arg); diff --git a/src/include/replication/worker_internal.h b/src/include/replication/worker_internal.h index 1dc1a491e9..3da852fed4 100644 --- a/src/include/replication/worker_internal.h +++ b/src/include/replication/worker_internal.h @@ -158,6 +158,7 @@ extern void logicalrep_worker_wakeup(Oid subid, Oid relid); extern void logicalrep_worker_wakeup_ptr(LogicalRepWorker *worker); extern int logicalrep_sync_worker_count(Oid subid); +extern int logicalrep_apply_background_worker_count(Oid subid); extern void ReplicationOriginNameForTablesync(Oid suboid, Oid relid, char *originname, int szorgname); -- 2.23.0.windows.1 ^ permalink raw reply [nested|flat] 66+ messages in thread
* RE: Perform streaming logical transactions by background workers and parallel apply @ 2022-06-02 10:03 [email protected] <[email protected]> parent: Amit Kapila <[email protected]> 1 sibling, 0 replies; 66+ messages in thread From: [email protected] @ 2022-06-02 10:03 UTC (permalink / raw) To: Amit Kapila <[email protected]>; +Cc: Peter Smith <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; PostgreSQL Hackers <[email protected]> On Mon, May 30, 2022 7:38 PM Amit Kapila <[email protected]> wrote: > Few comments/suggestions for 0001 and 0003 > ===================================== > 0001 > -------- Thanks for your comments. > 1. > + else > + snprintf(bgw.bgw_name, BGW_MAXLEN, > + "logical replication apply worker for subscription %u", subid); > > Can we slightly change the message to: "logical replication background > apply worker for subscription %u"? Improve the message as suggested. > 2. Can we think of separating the new logic for applying the xact by > bgworker into a new file like applybgwroker or applyparallel? We have > previously done the same in the case of vacuum (see vacuumparallel.c). Improve the patch as suggested. I separated the new logic related to apply background worker to new file src/backend/replication/logical/applybgwroker.c. > 3. > + /* > + * XXX The publisher side doesn't always send relation update messages > + * after the streaming transaction, so update the relation in main > + * apply worker here. > + */ > + if (action == LOGICAL_REP_MSG_RELATION) > + { > + LogicalRepRelation *rel = logicalrep_read_rel(s); > + logicalrep_relmap_update(rel); > + } > > I think the publisher side won't send the relation update message > after streaming transaction only if it has already been sent for a > non-streaming transaction in which case we don't need to update the > local cache here. This is as per my understanding of > maybe_send_schema(), do let me know if I am missing something? If my > understanding is correct then we don't need this change. I think we need this change because the publisher will invoke function cleanup_rel_sync_cache when committing a streaming transaction, then it will set "schema_sent" to true for related entry. Later, publisher may not send this schema in function maybe_send_schema because we already sent this schema (schema_sent = true). If we do not have this change, It would cause an error in the following case: Suppose that after walsender worker starts, first we commit a streaming transaction. Walsender sends relation update message, and only apply background worker can update relation map cache by this message. After this, if we commit a non-streamed transaction that contains same replicated table, walsender will not send relation update message, so main apply worker would not get relation update message. I think we need this change to update relation map cache not only in apply background worker but also in apply main worker. In addition, we should also handle the LOGICAL_REP_MSG_TYPE message just like LOGICAL_REP_MSG_RELATION. So improve the code you mentioned. BTW, I simplify the function handle_streamed_transaction(). > 4. > + * For the main apply worker, if in streaming mode (receiving a block of > + * streamed transaction), we send the data to the apply background worker. > * > - * If in streaming mode (receiving a block of streamed transaction), we > - * simply redirect it to a file for the proper toplevel transaction. > > This comment is slightly confusing. Can we change it to something > like: "In streaming case (receiving a block of streamed transaction), > for SUBSTREAM_ON mode, we simply redirect it to a file for the proper > toplevel transaction, and for SUBSTREAM_APPLY mode, we send the > changes to background apply worker."? Improve the comments as suggested. > 5. > +apply_handle_stream_abort(StringInfo s) > { > ... > ... > + /* > + * If the two XIDs are the same, it's in fact abort of toplevel xact, > + * so just free the subxactlist. > + */ > + if (subxid == xid) > + { > + set_apply_error_context_xact(subxid, InvalidXLogRecPtr); > > - fd = BufFileOpenFileSet(MyLogicalRepWorker->stream_fileset, path, > O_RDONLY, > - false); > + AbortCurrentTransaction(); > > - buffer = palloc(BLCKSZ); > + EndTransactionBlock(false); > + CommitTransactionCommand(); > + > + in_remote_transaction = false; > ... > ... > } > > Here, can we update the replication origin as we are doing in > apply_handle_rollback_prepared? Currently, we don't do it because we > are just cleaning up temporary files for which we don't even have a > transaction. Also, we don't have the required infrastructure to > advance origins for aborts as we have for abort prepared. See commits > [1eb6d6527a][8a812e5106]. If we think it is a good idea then I think > we need to send abort_lsn and abort_time from the publisher and we > need to be careful to make it work with lower subscriber versions that > don't have the facility to process these additional values. I think it is a good idea. I will consider this and add this part in next version. > 0003 > -------- > 6. > + /* > + * If any unique index exist, check that they are same as remoterel. > + */ > + if (!rel->sameunique) > + ereport(ERROR, > + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), > + errmsg("cannot replicate relation with different unique index"), > + errhint("Please change the streaming option to 'on' instead of 'apply'."))); > > I think we can do better here. Instead of simply erroring out and > asking the user to change streaming mode, we can remember this in the > system catalog probably in pg_subscription, and then on restart, we > can change the streaming mode to 'on', perform the transaction, and > again change the streaming mode to apply. I am not sure whether we > want to do it in the first version or not, so if you agree with this, > developing it as a separate patch would be a good idea. > > Also, please update comments here as to why we don't handle such cases. Yes, I think it is a good idea. I will develop it as a separate patch later. And I added the comments atop function apply_bgworker_relation_check as below: ``` * Although we maintains the commit order by allowing only one process to * commit at a time, our access order to the relation has changed. * This could cause unexpected problems if the unique column on the replicated * table is inconsistent with the publisher-side or contains non-immutable * functions when applying transactions in the apply background worker. ``` I also made some other changes. The new patches and the modification details were attached in [1]. [1] - https://www.postgresql.org/message-id/OS3PR01MB62758A881FF3240171B7B21B9EDE9%40OS3PR01MB6275.jpnprd0... Regards, Wang wei ^ permalink raw reply [nested|flat] 66+ messages in thread
* RE: Perform streaming logical transactions by background workers and parallel apply @ 2022-06-08 07:12 [email protected] <[email protected]> parent: [email protected] <[email protected]> 0 siblings, 1 reply; 66+ messages in thread From: [email protected] @ 2022-06-08 07:12 UTC (permalink / raw) To: Amit Kapila <[email protected]>; Masahiko Sawada <[email protected]>; +Cc: Peter Smith <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; PostgreSQL Hackers <[email protected]> On Thur, Jun 2, 2022 6:02 PM I wrote: > Attach the new patches. I tried to improve the patches by following 2 points: 1. Improved the patch as suggested by Amit-san that I mentioned in [1]. When publisher sends a "STREAM ABORT" message to subscriber, add the lsn and time of this abort to this message.(see function logicalrep_write_stream_abort) When subscriber receives this message, it will update the replication origin. (see function apply_handle_stream_abort and function RecordTransactionAbort) 2. Fixed missing settings for two GUCs (session_replication_role and search_path) in apply background worker in patch 0001 and improved checking of trigger functions in patch 0003. Thanks to Hou Zhi Jie for adding the aborts message related infrastructure for the first point. Thanks to Shi Yu for pointing out the second point. Attach the new patches.(only changed 0001 and 0003) [1] - https://www.postgresql.org/message-id/OS3PR01MB6275FBD9359F8ED0EDE7E5459EDE9%40OS3PR01MB6275.jpnprd0... Regards, Wang wei Attachments: [application/octet-stream] v9-0001-Perform-streaming-logical-transactions-by-backgro.patch (93.3K, ../../OS3PR01MB6275208A2F8ED832710F65E09EA49@OS3PR01MB6275.jpnprd01.prod.outlook.com/2-v9-0001-Perform-streaming-logical-transactions-by-backgro.patch) download | inline diff: From f5319845a26b07ce988736516a474d909a9a168a Mon Sep 17 00:00:00 2001 From: "houzj.fnst" <[email protected]> Date: Wed, 20 Apr 2022 16:45:07 +0800 Subject: [PATCH v9 1/4] Perform streaming logical transactions by background workers Currently, for large transactions, the publisher sends the data in multiple streams (changes divided into chunks depending upon logical_decoding_work_mem), and then on the subscriber-side, the apply worker writes the changes into temporary files and once it receives the commit, it reads from the file and applies the entire transaction. To improve the performance of such transactions, we can instead allow them to be applied via background workers. In this approach, we assign a new apply background worker (if available) as soon as the xact's first stream is received and the main apply worker will send changes to this new worker via shared memory. The apply background worker will directly apply the change instead of writing it to temporary files. We keep this worker assigned till the transaction commit is received and also wait for the worker to finish at commit. This preserves commit ordering and avoids writing to and reading from file in most cases. We still need to spill if there is no worker available. We also need to allow stream_stop to complete by the apply background worker to avoid deadlocks because T-1's current stream of changes can update rows in conflicting order with T-2's next stream of changes. This patch also extends the SUBSCRIPTION 'streaming' option so that the user can control whether to apply the streaming transaction in an apply background worker or spill the change to disk. User can set the streaming option to 'on/off', 'apply'. For now, 'apply' means the streaming will be applied via a apply background worker if available. 'on' means the streaming transaction will be spilled to disk. --- doc/src/sgml/catalogs.sgml | 10 +- doc/src/sgml/logical-replication.sgml | 7 + doc/src/sgml/protocol.sgml | 19 + doc/src/sgml/ref/create_subscription.sgml | 24 +- src/backend/access/transam/xact.c | 13 + src/backend/commands/subscriptioncmds.c | 66 +- src/backend/postmaster/bgworker.c | 3 + src/backend/replication/logical/Makefile | 3 +- .../replication/logical/applybgwroker.c | 756 ++++++++++++++++++ src/backend/replication/logical/decode.c | 10 +- src/backend/replication/logical/launcher.c | 87 +- src/backend/replication/logical/origin.c | 26 +- src/backend/replication/logical/proto.c | 35 +- .../replication/logical/reorderbuffer.c | 10 +- src/backend/replication/logical/tablesync.c | 10 +- src/backend/replication/logical/worker.c | 645 +++++++++++---- src/backend/replication/pgoutput/pgoutput.c | 2 +- src/backend/utils/activity/wait_event.c | 3 + src/bin/pg_dump/pg_dump.c | 6 +- src/include/catalog/pg_subscription.h | 17 +- src/include/replication/logicalproto.h | 19 +- src/include/replication/logicalworker.h | 1 + src/include/replication/origin.h | 2 +- src/include/replication/reorderbuffer.h | 7 +- src/include/replication/worker_internal.h | 103 ++- src/include/utils/wait_event.h | 1 + src/test/regress/expected/subscription.out | 2 +- src/tools/pgindent/typedefs.list | 5 + 28 files changed, 1677 insertions(+), 215 deletions(-) create mode 100644 src/backend/replication/logical/applybgwroker.c diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index c00c93dd7b..5cb39b18bd 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -7873,11 +7873,15 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l <row> <entry role="catalog_table_entry"><para role="column_definition"> - <structfield>substream</structfield> <type>bool</type> + <structfield>substream</structfield> <type>char</type> </para> <para> - If true, the subscription will allow streaming of in-progress - transactions + Controls how to handle the streaming of in-progress transactions: + <literal>f</literal> = disallow streaming of in-progress transactions, + <literal>t</literal> = spill the changes of in-progress transactions to + disk and apply at once after the transaction is committed on the + publisher, + <literal>a</literal> = apply changes directly using a background worker </para></entry> </row> diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml index 145ea71d61..cca6c97d79 100644 --- a/doc/src/sgml/logical-replication.sgml +++ b/doc/src/sgml/logical-replication.sgml @@ -948,6 +948,13 @@ CONTEXT: processing remote data for replication origin "pg_16395" during "INSER might not violate any constraint. This can easily make the subscriber inconsistent. </para> + + <para> + Setting streaming mode to <literal>apply</literal> could export invalid LSN + as finish LSN of failed transaction. Changing the streaming mode and making + the same conflict writes the finish LSN of the failed transaction in the + server log if required. + </para> </sect1> <sect1 id="logical-replication-restrictions"> diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml index bb30340892..34be1b2698 100644 --- a/doc/src/sgml/protocol.sgml +++ b/doc/src/sgml/protocol.sgml @@ -6809,6 +6809,25 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;" </listitem> </varlistentry> + <varlistentry> + <term>Int64 (XLogRecPtr)</term> + <listitem> + <para> + The LSN of the abort. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Int64 (TimestampTz)</term> + <listitem> + <para> + Abort timestamp of the transaction. The value is in number + of microseconds since PostgreSQL epoch (2000-01-01). + </para> + </listitem> + </varlistentry> + <varlistentry> <term>Int32 (TransactionId)</term> <listitem> diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml index 35b39c28da..d4caae0222 100644 --- a/doc/src/sgml/ref/create_subscription.sgml +++ b/doc/src/sgml/ref/create_subscription.sgml @@ -217,13 +217,29 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl </varlistentry> <varlistentry> - <term><literal>streaming</literal> (<type>boolean</type>)</term> + <term><literal>streaming</literal> (<type>enum</type>)</term> <listitem> <para> Specifies whether to enable streaming of in-progress transactions - for this subscription. By default, all transactions - are fully decoded on the publisher and only then sent to the - subscriber as a whole. + for this subscription. The default value is <literal>off</literal>, + all transactions are fully decoded on the publisher and only then + sent to the subscriber as a whole. + </para> + + <para> + If set to <literal>on</literal>, the changes of transaction are + written to temporary files and then applied at once after the + transaction is committed on the publisher. + </para> + + <para> + If set to <literal>apply</literal> incoming + changes are directly applied via one of the background workers, if + available. If no background worker is free to handle streaming + transaction then the changes are written to a file and applied after + the transaction is committed. Note that if an error happens when + applying changes in a background worker, it might not report the + finish LSN of the remote transaction in the server log. </para> </listitem> </varlistentry> diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c index 47d80b0d25..678540ba25 100644 --- a/src/backend/access/transam/xact.c +++ b/src/backend/access/transam/xact.c @@ -1711,6 +1711,7 @@ RecordTransactionAbort(bool isSubXact) int nchildren; TransactionId *children; TimestampTz xact_time; + bool replorigin; /* * If we haven't been assigned an XID, nobody will care whether we aborted @@ -1741,6 +1742,13 @@ RecordTransactionAbort(bool isSubXact) elog(PANIC, "cannot abort transaction %u, it was already committed", xid); + /* + * Are we using the replication origins feature? Or, in other words, + * are we replaying remote actions? + */ + replorigin = (replorigin_session_origin != InvalidRepOriginId && + replorigin_session_origin != DoNotReplicateId); + /* Fetch the data we need for the abort record */ nrels = smgrGetPendingDeletes(false, &rels); nchildren = xactGetCommittedChildren(&children); @@ -1765,6 +1773,11 @@ RecordTransactionAbort(bool isSubXact) MyXactFlags, InvalidTransactionId, NULL); + if (replorigin) + /* Move LSNs forward for this replication origin */ + replorigin_session_advance(replorigin_session_origin_lsn, + XactLastRecEnd); + /* * Report the latest async abort LSN, so that the WAL writer knows to * flush this abort. There's nothing to be gained by delaying this, since diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c index 83e6eae855..4080bba987 100644 --- a/src/backend/commands/subscriptioncmds.c +++ b/src/backend/commands/subscriptioncmds.c @@ -83,7 +83,7 @@ typedef struct SubOpts bool copy_data; bool refresh; bool binary; - bool streaming; + char streaming; bool twophase; bool disableonerr; XLogRecPtr lsn; @@ -95,6 +95,62 @@ static List *merge_publications(List *oldpublist, List *newpublist, bool addpub, static void ReportSlotConnectionError(List *rstates, Oid subid, char *slotname, char *err); +/* + * Extract the streaming mode value from a DefElem. This is like + * defGetBoolean() but also accepts the special value of "apply". + */ +static char +defGetStreamingMode(DefElem *def) +{ + /* + * If no parameter given, assume "true" is meant. + */ + if (def->arg == NULL) + return SUBSTREAM_ON; + + /* + * Allow 0, 1, "true", "false", "on", "off" or "apply". + */ + switch (nodeTag(def->arg)) + { + case T_Integer: + switch (intVal(def->arg)) + { + case 0: + return SUBSTREAM_OFF; + case 1: + return SUBSTREAM_ON; + default: + /* otherwise, error out below */ + break; + } + break; + default: + { + char *sval = defGetString(def); + + /* + * The set of strings accepted here should match up with the + * grammar's opt_boolean_or_string production. + */ + if (pg_strcasecmp(sval, "false") == 0 || + pg_strcasecmp(sval, "off") == 0) + return SUBSTREAM_OFF; + if (pg_strcasecmp(sval, "true") == 0 || + pg_strcasecmp(sval, "on") == 0) + return SUBSTREAM_ON; + if (pg_strcasecmp(sval, "apply") == 0) + return SUBSTREAM_APPLY; + } + break; + } + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s requires a Boolean value or \"apply\"", + def->defname))); + return SUBSTREAM_OFF; /* keep compiler quiet */ +} + /* * Common option parsing function for CREATE and ALTER SUBSCRIPTION commands. * @@ -132,7 +188,7 @@ parse_subscription_options(ParseState *pstate, List *stmt_options, if (IsSet(supported_opts, SUBOPT_BINARY)) opts->binary = false; if (IsSet(supported_opts, SUBOPT_STREAMING)) - opts->streaming = false; + opts->streaming = SUBSTREAM_OFF; if (IsSet(supported_opts, SUBOPT_TWOPHASE_COMMIT)) opts->twophase = false; if (IsSet(supported_opts, SUBOPT_DISABLE_ON_ERR)) @@ -233,7 +289,7 @@ parse_subscription_options(ParseState *pstate, List *stmt_options, errorConflictingDefElem(defel, pstate); opts->specified_opts |= SUBOPT_STREAMING; - opts->streaming = defGetBoolean(defel); + opts->streaming = defGetStreamingMode(defel); } else if (strcmp(defel->defname, "two_phase") == 0) { @@ -601,7 +657,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt, values[Anum_pg_subscription_subowner - 1] = ObjectIdGetDatum(owner); values[Anum_pg_subscription_subenabled - 1] = BoolGetDatum(opts.enabled); values[Anum_pg_subscription_subbinary - 1] = BoolGetDatum(opts.binary); - values[Anum_pg_subscription_substream - 1] = BoolGetDatum(opts.streaming); + values[Anum_pg_subscription_substream - 1] = CharGetDatum(opts.streaming); values[Anum_pg_subscription_subtwophasestate - 1] = CharGetDatum(opts.twophase ? LOGICALREP_TWOPHASE_STATE_PENDING : @@ -1060,7 +1116,7 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt, if (IsSet(opts.specified_opts, SUBOPT_STREAMING)) { values[Anum_pg_subscription_substream - 1] = - BoolGetDatum(opts.streaming); + CharGetDatum(opts.streaming); replaces[Anum_pg_subscription_substream - 1] = true; } diff --git a/src/backend/postmaster/bgworker.c b/src/backend/postmaster/bgworker.c index 40601aefd9..40ccb8993c 100644 --- a/src/backend/postmaster/bgworker.c +++ b/src/backend/postmaster/bgworker.c @@ -128,6 +128,9 @@ static const struct }, { "ApplyWorkerMain", ApplyWorkerMain + }, + { + "ApplyBgworkerMain", ApplyBgworkerMain } }; diff --git a/src/backend/replication/logical/Makefile b/src/backend/replication/logical/Makefile index c4e2fdeb71..a24a41969e 100644 --- a/src/backend/replication/logical/Makefile +++ b/src/backend/replication/logical/Makefile @@ -26,6 +26,7 @@ OBJS = \ reorderbuffer.o \ snapbuild.o \ tablesync.o \ - worker.o + worker.o \ + applybgwroker.o include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/replication/logical/applybgwroker.c b/src/backend/replication/logical/applybgwroker.c new file mode 100644 index 0000000000..8af64974f2 --- /dev/null +++ b/src/backend/replication/logical/applybgwroker.c @@ -0,0 +1,756 @@ +/*------------------------------------------------------------------------- + * applybgwroker.c + * Support routines for applying xact by apply background worker + * + * Copyright (c) 2016-2022, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/replication/logical/applybgwroker.c + * + * This file contains routines that are intended to support setting up, using, + * and tearing down a ApplyBgworkerState. + * Refer to the comments in file header of logical/worker.c to see more + * informations about apply background worker. + * + *------------------------------------------------------------------------- + */ + +#include "postgres.h" + +#include "libpq/pqformat.h" +#include "mb/pg_wchar.h" +#include "pgstat.h" +#include "postmaster/interrupt.h" +#include "replication/logicalworker.h" +#include "replication/origin.h" +#include "replication/walreceiver.h" +#include "replication/worker_internal.h" +#include "storage/ipc.h" +#include "storage/procarray.h" +#include "tcop/tcopprot.h" +#include "utils/inval.h" +#include "utils/memutils.h" +#include "utils/resowner.h" +#include "utils/syscache.h" + +#define PG_LOGICAL_APPLY_SHM_MAGIC 0x79fb2447 // TODO Consider change + +/* + * DSM keys for apply background worker. Unlike other parallel execution code, + * since we don't need to worry about DSM keys conflicting with plan_node_id we + * can use small integers. + */ +#define APPLY_BGWORKER_KEY_SHARED 1 +#define APPLY_BGWORKER_KEY_MQ 2 + +/* + * entry for a hash table we use to map from xid to our apply background worker + * state. + */ +typedef struct ApplyBgworkerEntry +{ + TransactionId xid; + ApplyBgworkerState *wstate; +} ApplyBgworkerEntry; + +/* Apply background workers hash table (initialized on first use) */ +static HTAB *ApplyWorkersHash = NULL; +static List *ApplyWorkersFreeList = NIL; +static List *ApplyWorkersList = NIL; + +/* + * Fields to record the share informations between main apply worker and apply + * background worker. + */ +volatile ApplyBgworkerShared *MyParallelState = NULL; + +List *subxactlist = NIL; + +/* apply background worker setup */ +static ApplyBgworkerState *apply_bgworker_setup(void); +static void apply_bgworker_setup_dsm(ApplyBgworkerState *wstate); + +/* + * Look up worker inside ApplyWorkersHash for requested xid. + * + * If start flag is true, try to start a new worker if not found in hash table. + */ +ApplyBgworkerState * +apply_bgworker_find_or_start(TransactionId xid, bool start) +{ + bool found; + ApplyBgworkerState *wstate; + ApplyBgworkerEntry *entry = NULL; + + if (!TransactionIdIsValid(xid)) + return NULL; + + /* + * We don't start new background worker if we are not in streaming apply + * mode. + */ + if (MySubscription->stream != SUBSTREAM_APPLY) + return NULL; + + /* + * We don't start new background worker if user has set skiplsn as it's + * possible that user want to skip the streaming transaction. For + * streaming transaction, we need to spill the transaction to disk so that + * we can get the last LSN of the transaction to judge whether to skip + * before starting to apply the change. + */ + if (start && !XLogRecPtrIsInvalid(MySubscription->skiplsn)) + return NULL; + + /* + * For streaming transactions that are being applied in apply background + * worker, we cannot decide whether to apply the change for a relation + * that is not in the READY state (see should_apply_changes_for_rel) as we + * won't know remote_final_lsn by that time. So, we don't start new apply + * background worker in this case. + */ + if (start && !AllTablesyncsReady()) + return NULL; + + /* First time through, initialize apply workers hashtable */ + if (ApplyWorkersHash == NULL) + { + HASHCTL ctl; + + MemSet(&ctl, 0, sizeof(ctl)); + ctl.keysize = sizeof(TransactionId); + ctl.entrysize = sizeof(ApplyBgworkerEntry); + ctl.hcxt = ApplyContext; + + ApplyWorkersHash = hash_create("logical apply workers hash", 8, &ctl, + HASH_ELEM | HASH_BLOBS | HASH_CONTEXT); + } + + /* + * Find entry for requested transaction. + */ + entry = hash_search(ApplyWorkersHash, &xid, start ? HASH_ENTER : HASH_FIND, + &found); + if (found) + { + entry->wstate->pstate->status = APPLY_BGWORKER_BUSY; + return entry->wstate; + } + else if (!start) + return NULL; + + /* + * Now, we try to get a apply background worker. If there is at least one + * worker in the idle list, then take one. Otherwise, we try to start a + * new apply background worker. + */ + if (list_length(ApplyWorkersFreeList) > 0) + { + wstate = (ApplyBgworkerState *) llast(ApplyWorkersFreeList); + ApplyWorkersFreeList = list_delete_last(ApplyWorkersFreeList); + } + else + { + wstate = apply_bgworker_setup(); + + if (wstate == NULL) + { + /* + * If the apply background worker cannot be launched, remove entry + * in hash table. + */ + hash_search(ApplyWorkersHash, &xid, HASH_REMOVE, &found); + return NULL; + } + } + + /* Fill up the hash entry */ + wstate->pstate->status = APPLY_BGWORKER_BUSY; + wstate->pstate->server_version = walrcv_server_version(LogRepWorkerWalRcvConn); + wstate->pstate->stream_xid = xid; + entry->wstate = wstate; + entry->xid = xid; + + return wstate; +} + +/* + * Add the worker to the free list and remove the entry from hash table. + */ +void +apply_bgworker_free(ApplyBgworkerState *wstate) +{ + MemoryContext oldctx; + TransactionId xid = wstate->pstate->stream_xid; + + Assert(wstate->pstate->status == APPLY_BGWORKER_FINISHED); + + oldctx = MemoryContextSwitchTo(ApplyContext); + + hash_search(ApplyWorkersHash, &xid, HASH_REMOVE, NULL); + + elog(DEBUG1, "adding finished apply worker #%u for xid %u to the idle list", + wstate->pstate->n, wstate->pstate->stream_xid); + + ApplyWorkersFreeList = lappend(ApplyWorkersFreeList, wstate); + + MemoryContextSwitchTo(oldctx); +} + +/* Apply Background Worker main loop */ +static void +LogicalApplyBgwLoop(shm_mq_handle *mqh, volatile ApplyBgworkerShared *pst) +{ + shm_mq_result shmq_res; + PGPROC *registrant; + ErrorContextCallback errcallback; + XLogRecPtr last_received = InvalidXLogRecPtr; + + registrant = BackendPidGetProc(MyBgworkerEntry->bgw_notify_pid); + SetLatch(®istrant->procLatch); + + /* + * Push apply error context callback. Fields will be filled during + * applying a change. + */ + errcallback.callback = apply_error_callback; + errcallback.previous = error_context_stack; + error_context_stack = &errcallback; + + for (;;) + { + void *data; + Size len; + int c; + StringInfoData s; + MemoryContext oldctx; + XLogRecPtr start_lsn; + XLogRecPtr end_lsn; + TimestampTz send_time; + + CHECK_FOR_INTERRUPTS(); + + /* Ensure we are reading the data into our memory context. */ + oldctx = MemoryContextSwitchTo(ApplyMessageContext); + + shmq_res = shm_mq_receive(mqh, &len, &data, false); + + if (shmq_res != SHM_MQ_SUCCESS) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("lost connection to the main apply worker"))); + + if (len == 0) + break; + + s.cursor = 0; + s.maxlen = -1; + s.data = (char *) data; + s.len = len; + + /* + * We use first byte of message for additional communication between + * main Logical replication worker and apply bgworkers, so if it + * differs from 'w', then process it first. + */ + c = pq_getmsgbyte(&s); + switch (c) + { + /* End message of streaming chunk */ + case LOGICAL_REP_MSG_STREAM_STOP: + elog(DEBUG1, "[Apply BGW #%u] ended processing streaming chunk," + "waiting on shm_mq_receive", pst->n); + + apply_bgworker_set_status(APPLY_BGWORKER_READY); + + SetLatch(®istrant->procLatch); + + in_streamed_transaction = false; + pgstat_report_activity(STATE_IDLE, NULL); + continue; + case 'w': + break; + default: + elog(ERROR, "[Apply BGW #%u] unexpected message \"%c\"", + pst->n, c); + break; + } + + start_lsn = pq_getmsgint64(&s); + end_lsn = pq_getmsgint64(&s); + send_time = pq_getmsgint64(&s); + + if (last_received < start_lsn) + last_received = start_lsn; + + if (last_received < end_lsn) + last_received = end_lsn; + + /* + * TO IMPROVE: Do we need to display the apply background worker's + * information in pg_stat_replication ? + */ + UpdateWorkerStats(last_received, send_time, false); + + apply_dispatch(&s); + + if (ConfigReloadPending) + { + ConfigReloadPending = false; + ProcessConfigFile(PGC_SIGHUP); + } + + MemoryContextSwitchTo(oldctx); + MemoryContextReset(ApplyMessageContext); + } + + MemoryContextSwitchTo(TopMemoryContext); + MemoryContextReset(ApplyContext); + + /* Pop the error context stack */ + error_context_stack = errcallback.previous; + + elog(DEBUG1, "[Apply BGW #%u] exiting", pst->n); + + /* Signal main process that we are done. */ + SetLatch(®istrant->procLatch); +} + +/* + * Set the exit status so that the main apply worker can realize we have + * shutdown. + */ +static void +ApplyBgwShutdown(int code, Datum arg) +{ + SpinLockAcquire(&MyParallelState->mutex); + MyParallelState->status = APPLY_BGWORKER_EXIT; + SpinLockRelease(&MyParallelState->mutex); + + dsm_detach((dsm_segment *) DatumGetPointer(arg)); +} + +/* + * Apply Background Worker entry point + */ +void +ApplyBgworkerMain(Datum main_arg) +{ + volatile ApplyBgworkerShared *pst; + + dsm_handle handle; + dsm_segment *seg; + shm_toc *toc; + shm_mq *mq; + shm_mq_handle *mqh; + MemoryContext oldcontext; + RepOriginId originid; + int worker_slot = DatumGetInt32(main_arg); + char originname[NAMEDATALEN]; + + /* Attach to slot */ + logicalrep_worker_attach(worker_slot); + + MemoryContextSwitchTo(TopMemoryContext); + + /* Load the subscription into persistent memory context. */ + ApplyContext = AllocSetContextCreate(TopMemoryContext, + "ApplyContext", + ALLOCSET_DEFAULT_SIZES); + + /* + * Init the ApplyMessageContext which we clean up after each replication + * protocol message. + */ + ApplyMessageContext = AllocSetContextCreate(ApplyContext, + "ApplyMessageContext", + ALLOCSET_DEFAULT_SIZES); + + /* Setup signal handling */ + pqsignal(SIGHUP, SignalHandlerForConfigReload); + pqsignal(SIGTERM, die); + BackgroundWorkerUnblockSignals(); + + /* + * Connect to the dynamic shared memory segment. + * + * The backend that registered this worker passed us the ID of a shared + * memory segment to which we must attach for further instructions. In + * order to attach to dynamic shared memory, we need a resource owner. + * Once we've mapped the segment in our address space, attach to the table + * of contents so we can locate the various data structures we'll need to + * find within the segment. + */ + CurrentResourceOwner = ResourceOwnerCreate(NULL, "Logical apply worker"); + memcpy(&handle, MyBgworkerEntry->bgw_extra, sizeof(dsm_handle)); + seg = dsm_attach(handle); + if (seg == NULL) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("unable to map dynamic shared memory segment"))); + toc = shm_toc_attach(PG_LOGICAL_APPLY_SHM_MAGIC, dsm_segment_address(seg)); + if (toc == NULL) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("bad magic number in dynamic shared memory segment"))); + + before_shmem_exit(ApplyBgwShutdown, PointerGetDatum(seg)); + + /* Look up the parallel state. */ + pst = shm_toc_lookup(toc, APPLY_BGWORKER_KEY_SHARED, false); + MyParallelState = pst; + + /* + * Attach to the message queue. + */ + mq = shm_toc_lookup(toc, APPLY_BGWORKER_KEY_MQ, false); + shm_mq_set_receiver(mq, MyProc); + mqh = shm_mq_attach(mq, seg, NULL); + + /* Run as replica session replication role. */ + SetConfigOption("session_replication_role", "replica", + PGC_SUSET, PGC_S_OVERRIDE); + + /* Connect to our database. */ + BackgroundWorkerInitializeConnectionByOid(MyLogicalRepWorker->dbid, + MyLogicalRepWorker->userid, + 0); + + /* + * Set always-secure search path, so malicious users can't redirect user + * code (e.g. pg_index.indexprs). + */ + SetConfigOption("search_path", "", PGC_SUSET, PGC_S_OVERRIDE); + + /* + * Set the client encoding to the database encoding, since that is what + * the leader will expect. + */ + SetClientEncoding(GetDatabaseEncoding()); + + stream_xid = pst->stream_xid; + + StartTransactionCommand(); + oldcontext = MemoryContextSwitchTo(ApplyContext); + + MySubscription = GetSubscription(MyLogicalRepWorker->subid, true); + if (!MySubscription) + { + ereport(LOG, + (errmsg("logical replication apply worker for subscription %u will not " + "start because the subscription was removed during startup", + MyLogicalRepWorker->subid))); + proc_exit(0); + } + + MySubscriptionValid = true; + MemoryContextSwitchTo(oldcontext); + + /* Setup synchronous commit according to the user's wishes */ + SetConfigOption("synchronous_commit", MySubscription->synccommit, + PGC_BACKEND, PGC_S_OVERRIDE); + + /* Keep us informed about subscription changes. */ + CacheRegisterSyscacheCallback(SUBSCRIPTIONOID, + subscription_change_cb, + (Datum) 0); + + CommitTransactionCommand(); + + /* Setup replication origin tracking. */ + StartTransactionCommand(); + snprintf(originname, sizeof(originname), "pg_%u", MySubscription->oid); + originid = replorigin_by_name(originname, true); + if (!OidIsValid(originid)) + originid = replorigin_create(originname); + + /* + * The apply background worker doesn't need to monopolize this replication + * origin which was already acquired by its leader process. + */ + replorigin_session_setup(originid, false); + replorigin_session_origin = originid; + CommitTransactionCommand(); + + /* + * Allocate the origin name in long-lived context for error context + * message. + */ + apply_error_callback_arg.origin_name = MemoryContextStrdup(ApplyContext, + originname); + + /* + * Indicate that we're fully initialized and ready to begin the main part + * of the apply operation. + */ + apply_bgworker_set_status(APPLY_BGWORKER_ATTACHED); + + elog(DEBUG1, "[Apply BGW #%u] started", pst->n); + + LogicalApplyBgwLoop(mqh, pst); + + /* + * We're done. Explicitly detach the shared memory segment so that we + * don't get a resource leak warning at commit time. This will fire any + * on_dsm_detach callbacks we've registered, as well. Once that's done, + * we can go ahead and exit. + */ + dsm_detach(seg); + proc_exit(0); +} + +/* + * Set up a dynamic shared memory segment. + * + * We set up a control region that contains a ApplyBgworkerShared, + * plus one region per message queue. There are as many message queues as + * the number of workers. + */ +static void +apply_bgworker_setup_dsm(ApplyBgworkerState *wstate) +{ + shm_toc_estimator e; + Size segsize; + dsm_segment *seg; + shm_toc *toc; + ApplyBgworkerShared *pst; + shm_mq *mq; + int64 queue_size = 160000000; /* 16 MB for now */ + + /* + * Estimate how much shared memory we need. + * + * Because the TOC machinery may choose to insert padding of oddly-sized + * requests, we must estimate each chunk separately. + * + * We need one key to register the location of the header, and we need + * another key to track the location of the message queue. + */ + shm_toc_initialize_estimator(&e); + shm_toc_estimate_chunk(&e, sizeof(ApplyBgworkerShared)); + shm_toc_estimate_chunk(&e, (Size) queue_size); + + shm_toc_estimate_keys(&e, 1 + 1); + segsize = shm_toc_estimate(&e); + + /* Create the shared memory segment and establish a table of contents. */ + seg = dsm_create(shm_toc_estimate(&e), 0); + toc = shm_toc_create(PG_LOGICAL_APPLY_SHM_MAGIC, dsm_segment_address(seg), + segsize); + + /* Set up the header region. */ + pst = shm_toc_allocate(toc, sizeof(ApplyBgworkerShared)); + SpinLockInit(&pst->mutex); + pst->status = APPLY_BGWORKER_BUSY; + pst->server_version = walrcv_server_version(LogRepWorkerWalRcvConn); + pst->stream_xid = stream_xid; + pst->n = list_length(ApplyWorkersList) + 1; + + shm_toc_insert(toc, APPLY_BGWORKER_KEY_SHARED, pst); + + /* Set up one message queue per worker, plus one. */ + mq = shm_mq_create(shm_toc_allocate(toc, (Size) queue_size), + (Size) queue_size); + shm_toc_insert(toc, APPLY_BGWORKER_KEY_MQ, mq); + shm_mq_set_sender(mq, MyProc); + + /* Attach the queues. */ + wstate->mq_handle = shm_mq_attach(mq, seg, NULL); + + /* Return results to caller. */ + wstate->dsm_seg = seg; + wstate->pstate = pst; +} + +/* + * Start apply background worker process and allocate shared memory for it. + */ +static ApplyBgworkerState * +apply_bgworker_setup(void) +{ + MemoryContext oldcontext; + bool launched; + ApplyBgworkerState *wstate; + + elog(DEBUG1, "setting up apply worker #%u", list_length(ApplyWorkersList) + 1); + + oldcontext = MemoryContextSwitchTo(ApplyContext); + + wstate = (ApplyBgworkerState *) palloc0(sizeof(ApplyBgworkerState)); + + /* Setup shared memory */ + apply_bgworker_setup_dsm(wstate); + + launched = logicalrep_worker_launch(MyLogicalRepWorker->dbid, + MySubscription->oid, + MySubscription->name, + MyLogicalRepWorker->userid, + InvalidOid, + dsm_segment_handle(wstate->dsm_seg)); + + if (launched) + { + /* Wait for worker to attach. */ + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_ATTACHED); + + ApplyWorkersList = lappend(ApplyWorkersList, wstate); + } + else + { + pfree(wstate); + wstate = NULL; + } + + MemoryContextSwitchTo(oldcontext); + + return wstate; +} + +/* + * Send the data to the specified apply background worker via shared-memory queue. + */ +void +apply_bgworker_send_data(ApplyBgworkerState *wstate, Size nbytes, const void *data) +{ + shm_mq_result result; + + result = shm_mq_send(wstate->mq_handle, nbytes, data, false, true); + + if (result != SHM_MQ_SUCCESS) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("could not send tuples to shared-memory queue"))); +} + +/* + * Wait until the status of apply background worker reaches the + * 'wait_for_status' + */ +void +apply_bgworker_wait_for(ApplyBgworkerState *wstate, + ApplyBgworkerStatus wait_for_status) +{ + for (;;) + { + char status; + + SpinLockAcquire(&wstate->pstate->mutex); + status = wstate->pstate->status; + SpinLockRelease(&wstate->pstate->mutex); + + /* Done if already in correct status. */ + if (status == wait_for_status) + break; + + /* If any workers (or the postmaster) have died, we have failed. */ + if (status == APPLY_BGWORKER_EXIT) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("background worker %u failed to apply transaction %u", + wstate->pstate->n, wstate->pstate->stream_xid))); + + /* Wait to be signalled. */ + WaitLatch(MyLatch, WL_LATCH_SET | WL_EXIT_ON_PM_DEATH, 0, + WAIT_EVENT_LOGICAL_APPLY_WORKER_STATE_CHANGE); + + /* Reset the latch so we don't spin. */ + ResetLatch(MyLatch); + + /* An interrupt may have occurred while we were waiting. */ + CHECK_FOR_INTERRUPTS(); + } +} + +/* + * Check the status of workers and report an error if any apply background + * worker has exited unexpectedly. + * + * Exit if any relation is not in the READY state and if any worker is handling + * the streaming transaction at the same time. Because for streaming + * transactions that is being applied in apply background worker, we cannot + * decide whether to apply the change for a relation that is not in the READY + * state (see should_apply_changes_for_rel) as we won't know remote_final_lsn + * by that time. + */ +void +apply_bgworker_check_status(void) +{ + ListCell *lc; + + if (am_apply_bgworker() || MySubscription->stream != SUBSTREAM_APPLY) + return; + + foreach(lc, ApplyWorkersList) + { + ApplyBgworkerState *wstate = (ApplyBgworkerState *) lfirst(lc); + + /* + * We don't lock here as in the worst case we will just detect the + * failure of worker a bit later. + */ + if (wstate->pstate->status == APPLY_BGWORKER_EXIT) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("background worker %u exited unexpectedly", + wstate->pstate->n))); + } + + if (list_length(ApplyWorkersFreeList) != list_length(ApplyWorkersList) && + !AllTablesyncsReady()) + { + ereport(LOG, + (errmsg("logical replication apply workers for subscription \"%s\" will restart", + MySubscription->name), + errdetail("Cannot handle streamed replication transaction by apply " + "bgworkers until all tables are synchronized"))); + + proc_exit(0); + } +} + +/* Set the apply background worker status */ +void +apply_bgworker_set_status(ApplyBgworkerStatus status) +{ + if (!am_apply_bgworker()) + return; + + elog(DEBUG1, "[Apply BGW #%u] set status to %d", MyParallelState->n, status); + + SpinLockAcquire(&MyParallelState->mutex); + MyParallelState->status = status; + SpinLockRelease(&MyParallelState->mutex); +} + +/* + * Define a savepoint for a subxact in apply background worker if needed. + * + * Inside apply background worker we can figure out that new subtransaction was + * started if new change arrived with different xid. In that case we can define + * named savepoint, so that we were able to commit/rollback it separately + * later. + * Special case is if the first change comes from subtransaction, then + * we check that current_xid differs from stream_xid. + */ +void +apply_bgworker_subxact_info_add(TransactionId current_xid) +{ + if (current_xid != stream_xid && + !list_member_int(subxactlist, (int) current_xid)) + { + MemoryContext oldctx; + char spname[MAXPGPATH]; + + snprintf(spname, MAXPGPATH, "savepoint_for_xid_%u", current_xid); + + elog(DEBUG1, "[Apply BGW #%u] defining savepoint %s", + MyParallelState->n, spname); + + DefineSavepoint(spname); + CommitTransactionCommand(); + + oldctx = MemoryContextSwitchTo(ApplyContext); + subxactlist = lappend_int(subxactlist, (int) current_xid); + MemoryContextSwitchTo(oldctx); + } +} diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c index aa2427ba73..00fdde0830 100644 --- a/src/backend/replication/logical/decode.c +++ b/src/backend/replication/logical/decode.c @@ -651,9 +651,10 @@ DecodeCommit(LogicalDecodingContext *ctx, XLogRecordBuffer *buf, { for (i = 0; i < parsed->nsubxacts; i++) { - ReorderBufferForget(ctx->reorder, parsed->subxacts[i], buf->origptr); + ReorderBufferForget(ctx->reorder, parsed->subxacts[i], buf->origptr, + commit_time); } - ReorderBufferForget(ctx->reorder, xid, buf->origptr); + ReorderBufferForget(ctx->reorder, xid, buf->origptr, commit_time); return; } @@ -821,10 +822,11 @@ DecodeAbort(LogicalDecodingContext *ctx, XLogRecordBuffer *buf, for (i = 0; i < parsed->nsubxacts; i++) { ReorderBufferAbort(ctx->reorder, parsed->subxacts[i], - buf->record->EndRecPtr); + buf->record->EndRecPtr, abort_time); } - ReorderBufferAbort(ctx->reorder, xid, buf->record->EndRecPtr); + ReorderBufferAbort(ctx->reorder, xid, buf->record->EndRecPtr, + abort_time); } /* update the decoding stats */ diff --git a/src/backend/replication/logical/launcher.c b/src/backend/replication/logical/launcher.c index 2bdab53e19..201f51232e 100644 --- a/src/backend/replication/logical/launcher.c +++ b/src/backend/replication/logical/launcher.c @@ -73,6 +73,7 @@ static void logicalrep_launcher_onexit(int code, Datum arg); static void logicalrep_worker_onexit(int code, Datum arg); static void logicalrep_worker_detach(void); static void logicalrep_worker_cleanup(LogicalRepWorker *worker); +static void logicalrep_worker_stop_internal(LogicalRepWorker *worker); static bool on_commit_launcher_wakeup = false; @@ -223,6 +224,10 @@ logicalrep_worker_find(Oid subid, Oid relid, bool only_running) { LogicalRepWorker *w = &LogicalRepCtx->workers[i]; + /* We only need main apply worker or table sync worker here */ + if (w->subworker) + continue; + if (w->in_use && w->subid == subid && w->relid == relid && (!only_running || w->proc)) { @@ -259,11 +264,11 @@ logicalrep_workers_find(Oid subid, bool only_running) } /* - * Start new apply background worker, if possible. + * Start new background worker, if possible. */ -void +bool logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, Oid userid, - Oid relid) + Oid relid, dsm_handle subworker_dsm) { BackgroundWorker bgw; BackgroundWorkerHandle *bgw_handle; @@ -273,6 +278,10 @@ logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, Oid userid, LogicalRepWorker *worker = NULL; int nsyncworkers; TimestampTz now; + bool is_subworker = (subworker_dsm != DSM_HANDLE_INVALID); + + /* We don't support table sync in subworker */ + Assert(!(is_subworker && OidIsValid(relid))); ereport(DEBUG1, (errmsg_internal("starting logical replication worker for subscription \"%s\"", @@ -350,7 +359,7 @@ retry: if (OidIsValid(relid) && nsyncworkers >= max_sync_workers_per_subscription) { LWLockRelease(LogicalRepWorkerLock); - return; + return false; } /* @@ -364,7 +373,7 @@ retry: (errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED), errmsg("out of logical replication worker slots"), errhint("You might need to increase max_logical_replication_workers."))); - return; + return false; } /* Prepare the worker slot. */ @@ -379,6 +388,7 @@ retry: worker->relstate = SUBREL_STATE_UNKNOWN; worker->relstate_lsn = InvalidXLogRecPtr; worker->stream_fileset = NULL; + worker->subworker = is_subworker; worker->last_lsn = InvalidXLogRecPtr; TIMESTAMP_NOBEGIN(worker->last_send_time); TIMESTAMP_NOBEGIN(worker->last_recv_time); @@ -396,19 +406,30 @@ retry: BGWORKER_BACKEND_DATABASE_CONNECTION; bgw.bgw_start_time = BgWorkerStart_RecoveryFinished; snprintf(bgw.bgw_library_name, BGW_MAXLEN, "postgres"); - snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyWorkerMain"); + + if (!is_subworker) + snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyWorkerMain"); + else + snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyBgworkerMain"); + if (OidIsValid(relid)) snprintf(bgw.bgw_name, BGW_MAXLEN, "logical replication worker for subscription %u sync %u", subid, relid); - else + else if (!is_subworker) snprintf(bgw.bgw_name, BGW_MAXLEN, "logical replication worker for subscription %u", subid); + else + snprintf(bgw.bgw_name, BGW_MAXLEN, + "logical replication background apply worker for subscription %u ", subid); snprintf(bgw.bgw_type, BGW_MAXLEN, "logical replication worker"); bgw.bgw_restart_time = BGW_NEVER_RESTART; bgw.bgw_notify_pid = MyProcPid; bgw.bgw_main_arg = Int32GetDatum(slot); + if (is_subworker) + memcpy(bgw.bgw_extra, &subworker_dsm, sizeof(dsm_handle)); + if (!RegisterDynamicBackgroundWorker(&bgw, &bgw_handle)) { /* Failed to start worker, so clean up the worker slot. */ @@ -421,11 +442,13 @@ retry: (errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED), errmsg("out of background worker slots"), errhint("You might need to increase max_worker_processes."))); - return; + return false; } /* Now wait until it attaches. */ WaitForReplicationWorkerAttach(worker, generation, bgw_handle); + + return true; } /* @@ -436,7 +459,6 @@ void logicalrep_worker_stop(Oid subid, Oid relid) { LogicalRepWorker *worker; - uint16 generation; LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); @@ -449,6 +471,22 @@ logicalrep_worker_stop(Oid subid, Oid relid) return; } + logicalrep_worker_stop_internal(worker); + + LWLockRelease(LogicalRepWorkerLock); +} + +/* + * Workhorse for logicalrep_worker_stop() and logicalrep_worker_detach(). Stop + * the worker and wait for it to die. + */ +static void +logicalrep_worker_stop_internal(LogicalRepWorker *worker) +{ + uint16 generation; + + Assert(LWLockHeldByMe(LogicalRepWorkerLock)); + /* * Remember which generation was our worker so we can check if what we see * is still the same one. @@ -485,10 +523,7 @@ logicalrep_worker_stop(Oid subid, Oid relid) * different, meaning that a different worker has taken the slot. */ if (!worker->in_use || worker->generation != generation) - { - LWLockRelease(LogicalRepWorkerLock); return; - } /* Worker has assigned proc, so it has started. */ if (worker->proc) @@ -522,8 +557,6 @@ logicalrep_worker_stop(Oid subid, Oid relid) LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); } - - LWLockRelease(LogicalRepWorkerLock); } /* @@ -599,6 +632,29 @@ logicalrep_worker_attach(int slot) static void logicalrep_worker_detach(void) { + /* + * If we are the main apply worker, stop all the apply background workers + * we started before. + */ + if (!MyLogicalRepWorker->subworker) + { + List *workers; + ListCell *lc; + + LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); + + workers = logicalrep_workers_find(MyLogicalRepWorker->subid, true); + foreach(lc, workers) + { + LogicalRepWorker *w = (LogicalRepWorker *) lfirst(lc); + + if (w->subworker) + logicalrep_worker_stop_internal(w); + } + + LWLockRelease(LogicalRepWorkerLock); + } + /* Block concurrent access. */ LWLockAcquire(LogicalRepWorkerLock, LW_EXCLUSIVE); @@ -621,6 +677,7 @@ logicalrep_worker_cleanup(LogicalRepWorker *worker) worker->userid = InvalidOid; worker->subid = InvalidOid; worker->relid = InvalidOid; + worker->subworker = false; } /* @@ -868,7 +925,7 @@ ApplyLauncherMain(Datum main_arg) wait_time = wal_retrieve_retry_interval; logicalrep_worker_launch(sub->dbid, sub->oid, sub->name, - sub->owner, InvalidOid); + sub->owner, InvalidOid, DSM_HANDLE_INVALID); } } diff --git a/src/backend/replication/logical/origin.c b/src/backend/replication/logical/origin.c index 21937ab2d3..9273011b0a 100644 --- a/src/backend/replication/logical/origin.c +++ b/src/backend/replication/logical/origin.c @@ -1063,12 +1063,21 @@ ReplicationOriginExitCleanup(int code, Datum arg) * array doesn't have to be searched when calling * replorigin_session_advance(). * - * Obviously only one such cached origin can exist per process and the current + * Normally only one such cached origin can exist per process and the current * cached value can only be set again after the previous value is torn down * with replorigin_session_reset(). + * + * However, If must_acquire is false, we allow process to get the slot which is + * already acquired by other process. It's safe because 1) The only caller + * (apply background workers) will maintain the commit order by allowing only + * one process to commit at a time, so no two workers will be operating on the + * same origin at the same time (see comments in logical/worker.c). 2) Even + * though we try to advance the session's origin concurrently, it's safe to do + * so as we change/advance the session_origin LSNs under replicate_state + * LWLock. */ void -replorigin_session_setup(RepOriginId node) +replorigin_session_setup(RepOriginId node, bool must_acquire) { static bool registered_cleanup; int i; @@ -1110,7 +1119,7 @@ replorigin_session_setup(RepOriginId node) if (curstate->roident != node) continue; - else if (curstate->acquired_by != 0) + else if (curstate->acquired_by != 0 && must_acquire) { ereport(ERROR, (errcode(ERRCODE_OBJECT_IN_USE), @@ -1141,7 +1150,14 @@ replorigin_session_setup(RepOriginId node) Assert(session_replication_state->roident != InvalidRepOriginId); - session_replication_state->acquired_by = MyProcPid; + if (must_acquire) + session_replication_state->acquired_by = MyProcPid; + else if (session_replication_state->acquired_by == 0) + ereport(ERROR, + (errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED), + errmsg("could not find correct replication state slot for replication origin with OID %u for apply background worker", + node), + errhint("There is no replication state slot set by its main apply worker."))); LWLockRelease(ReplicationOriginLock); @@ -1321,7 +1337,7 @@ pg_replication_origin_session_setup(PG_FUNCTION_ARGS) name = text_to_cstring((text *) DatumGetPointer(PG_GETARG_DATUM(0))); origin = replorigin_by_name(name, false); - replorigin_session_setup(origin); + replorigin_session_setup(origin, true); replorigin_session_origin = origin; diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c index ff8513e2d2..a888038eb4 100644 --- a/src/backend/replication/logical/proto.c +++ b/src/backend/replication/logical/proto.c @@ -1166,28 +1166,47 @@ logicalrep_read_stream_commit(StringInfo in, LogicalRepCommitData *commit_data) */ void logicalrep_write_stream_abort(StringInfo out, TransactionId xid, - TransactionId subxid) + ReorderBufferTXN *txn, XLogRecPtr abort_lsn) { pq_sendbyte(out, LOGICAL_REP_MSG_STREAM_ABORT); - Assert(TransactionIdIsValid(xid) && TransactionIdIsValid(subxid)); + Assert(TransactionIdIsValid(xid) && TransactionIdIsValid(txn->xid)); /* transaction ID */ pq_sendint32(out, xid); - pq_sendint32(out, subxid); + pq_sendint32(out, txn->xid); + pq_sendint64(out, abort_lsn); + pq_sendint64(out, txn->xact_time.abort_time); } /* * Read STREAM ABORT from the output stream. */ void -logicalrep_read_stream_abort(StringInfo in, TransactionId *xid, - TransactionId *subxid) +logicalrep_read_stream_abort(StringInfo in, + LogicalRepStreamAbortData *abort_data, + bool include_abort_lsn) { - Assert(xid && subxid); + Assert(abort_data); + + abort_data->xid = pq_getmsgint(in, 4); + abort_data->subxid = pq_getmsgint(in, 4); - *xid = pq_getmsgint(in, 4); - *subxid = pq_getmsgint(in, 4); + /* + * If the version of the publisher is lower than the version of the + * subscriber, it may not support sending these two fields, so only take + * these fields when include_abort_lsn is true. + */ + if (include_abort_lsn) + { + abort_data->abort_lsn = pq_getmsgint64(in); + abort_data->abort_time = pq_getmsgint64(in); + } + else + { + abort_data->abort_lsn = InvalidXLogRecPtr; + abort_data->abort_time = 0; + } } /* diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c index 8da5f9089c..3d2bbdb38d 100644 --- a/src/backend/replication/logical/reorderbuffer.c +++ b/src/backend/replication/logical/reorderbuffer.c @@ -2826,7 +2826,8 @@ ReorderBufferFinishPrepared(ReorderBuffer *rb, TransactionId xid, * disk. */ void -ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn) +ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn, + TimestampTz abort_time) { ReorderBufferTXN *txn; @@ -2837,6 +2838,8 @@ ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn) if (txn == NULL) return; + txn->xact_time.abort_time = abort_time; + /* For streamed transactions notify the remote node about the abort. */ if (rbtxn_is_streamed(txn)) { @@ -2911,7 +2914,8 @@ ReorderBufferAbortOld(ReorderBuffer *rb, TransactionId oldestRunningXid) * to this xid might re-create the transaction incompletely. */ void -ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn) +ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn, + TimestampTz abort_time) { ReorderBufferTXN *txn; @@ -2922,6 +2926,8 @@ ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn) if (txn == NULL) return; + txn->xact_time.abort_time = abort_time; + /* For streamed transactions notify the remote node about the abort. */ if (rbtxn_is_streamed(txn)) rb->stream_abort(rb, txn, lsn); diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c index 670c6fcada..8ffba7e2e5 100644 --- a/src/backend/replication/logical/tablesync.c +++ b/src/backend/replication/logical/tablesync.c @@ -568,7 +568,8 @@ process_syncing_tables_for_apply(XLogRecPtr current_lsn) MySubscription->oid, MySubscription->name, MyLogicalRepWorker->userid, - rstate->relid); + rstate->relid, + DSM_HANDLE_INVALID); hentry->last_start_time = now; } } @@ -589,6 +590,9 @@ process_syncing_tables_for_apply(XLogRecPtr current_lsn) void process_syncing_tables(XLogRecPtr current_lsn) { + if (MyLogicalRepWorker->subworker) + return; + if (am_tablesync_worker()) process_syncing_tables_for_sync(current_lsn); else @@ -1273,7 +1277,7 @@ LogicalRepSyncTableStart(XLogRecPtr *origin_startpos) * time this tablesync was launched. */ originid = replorigin_by_name(originname, false); - replorigin_session_setup(originid); + replorigin_session_setup(originid, true); replorigin_session_origin = originid; *origin_startpos = replorigin_session_get_progress(false); @@ -1384,7 +1388,7 @@ LogicalRepSyncTableStart(XLogRecPtr *origin_startpos) true /* go backward */ , true /* WAL log */ ); UnlockRelationOid(ReplicationOriginRelationId, RowExclusiveLock); - replorigin_session_setup(originid); + replorigin_session_setup(originid, true); replorigin_session_origin = originid; } else diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c index fc210a9e7b..32bf516ebf 100644 --- a/src/backend/replication/logical/worker.c +++ b/src/backend/replication/logical/worker.c @@ -22,8 +22,25 @@ * STREAMED TRANSACTIONS * --------------------- * Streamed transactions (large transactions exceeding a memory limit on the - * upstream) are not applied immediately, but instead, the data is written - * to temporary files and then applied at once when the final commit arrives. + * upstream) are applied via one of two approaches. + * + * 1) Separate background workers + * + * Assign a new apply background worker (if available) as soon as the xact's + * first stream is received and the main apply worker will send changes to this + * new worker via shared memory. We keep this worker assigned till the + * transaction commit is received and also wait for the worker to finish at + * commit. This preserves commit ordering and avoids writing to and reading + * from file in most cases. We still need to spill if there is no worker + * available. We also need to allow stream_stop to complete by the background + * worker to avoid deadlocks because T-1's current stream of changes can update + * rows in conflicting order with T-2's next stream of changes. + * + * 2) Write to temporary files and apply when the final commit arrives + * + * If no worker is available to handle streamed transaction, the data is + * written to temporary files and then applied at once when the final commit + * arrives. * * Unlike the regular (non-streamed) case, handling streamed transactions has * to handle aborts of both the toplevel transaction and subtransactions. This @@ -219,20 +236,7 @@ typedef struct ApplyExecutionData PartitionTupleRouting *proute; /* partition routing info */ } ApplyExecutionData; -/* Struct for saving and restoring apply errcontext information */ -typedef struct ApplyErrorCallbackArg -{ - LogicalRepMsgType command; /* 0 if invalid */ - LogicalRepRelMapEntry *rel; - - /* Remote node information */ - int remote_attnum; /* -1 if invalid */ - TransactionId remote_xid; - XLogRecPtr finish_lsn; - char *origin_name; -} ApplyErrorCallbackArg; - -static ApplyErrorCallbackArg apply_error_callback_arg = +ApplyErrorCallbackArg apply_error_callback_arg = { .command = 0, .rel = NULL, @@ -242,7 +246,7 @@ static ApplyErrorCallbackArg apply_error_callback_arg = .origin_name = NULL, }; -static MemoryContext ApplyMessageContext = NULL; +MemoryContext ApplyMessageContext = NULL; MemoryContext ApplyContext = NULL; /* per stream context for streaming transactions */ @@ -251,27 +255,38 @@ static MemoryContext LogicalStreamingContext = NULL; WalReceiverConn *LogRepWorkerWalRcvConn = NULL; Subscription *MySubscription = NULL; -static bool MySubscriptionValid = false; +bool MySubscriptionValid = false; bool in_remote_transaction = false; static XLogRecPtr remote_final_lsn = InvalidXLogRecPtr; /* fields valid only when processing streamed transaction */ -static bool in_streamed_transaction = false; +bool in_streamed_transaction = false; + +TransactionId stream_xid = InvalidTransactionId; +static ApplyBgworkerState *stream_apply_worker = NULL; + +/* check if we are applying the transaction in apply background worker */ +#define apply_bgworker_active() (in_streamed_transaction && stream_apply_worker != NULL) -static TransactionId stream_xid = InvalidTransactionId; +/* + * The number of changes during one streaming block (only for apply background + * workers) + */ +static uint32 nchanges = 0; /* * We enable skipping all data modification changes (INSERT, UPDATE, etc.) for * the subscription if the remote transaction's finish LSN matches the subskiplsn. * Once we start skipping changes, we don't stop it until we skip all changes of * the transaction even if pg_subscription is updated and MySubscription->skiplsn - * gets changed or reset during that. Also, in streaming transaction cases, we - * don't skip receiving and spooling the changes since we decide whether or not + * gets changed or reset during that. Also, in streaming transaction cases (streaming = on), + * we don't skip receiving and spooling the changes since we decide whether or not * to skip applying the changes when starting to apply changes. The subskiplsn is * cleared after successfully skipping the transaction or applying non-empty * transaction. The latter prevents the mistakenly specified subskiplsn from - * being left. + * being left. Note that we cannot skip the streaming transaction in apply mode, + * because we cannot get the finish LSN before applying the changes. */ static XLogRecPtr skip_xact_finish_lsn = InvalidXLogRecPtr; #define is_skipping_changes() (unlikely(!XLogRecPtrIsInvalid(skip_xact_finish_lsn))) @@ -324,9 +339,6 @@ static void maybe_reread_subscription(void); static void DisableSubscriptionAndExit(void); -/* prototype needed because of stream_commit */ -static void apply_dispatch(StringInfo s); - static void apply_handle_commit_internal(LogicalRepCommitData *commit_data); static void apply_handle_insert_internal(ApplyExecutionData *edata, ResultRelInfo *relinfo, @@ -359,7 +371,6 @@ static void stop_skipping_changes(void); static void clear_subscription_skip_lsn(XLogRecPtr finish_lsn); /* Functions for apply error callback */ -static void apply_error_callback(void *arg); static inline void set_apply_error_context_xact(TransactionId xid, XLogRecPtr lsn); static inline void reset_apply_error_context_info(void); @@ -426,41 +437,76 @@ end_replication_step(void) } /* - * Handle streamed transactions. + * Handle streamed transactions for both main apply worker and apply background + * worker. * - * If in streaming mode (receiving a block of streamed transaction), we - * simply redirect it to a file for the proper toplevel transaction. + * In streaming case (receiving a block of streamed transaction), for + * SUBSTREAM_ON mode, we simply redirect it to a file for the proper toplevel + * transaction, and for SUBSTREAM_APPLY mode, we send the changes to background + * apply worker (LOGICAL_REP_MSG_RELATION or LOGICAL_REP_MSG_TYPE changes will + * also be applied in main apply worker). * - * Returns true for streamed transactions, false otherwise (regular mode). + * For non-streamed transactions, returns false; + * For streamed transactions, returns true if in main apply worker (except we + * apply streamed transaction in "apply" mode and address + * LOGICAL_REP_MSG_RELATION or LOGICAL_REP_MSG_TYPE changes), false otherwise. */ static bool handle_streamed_transaction(LogicalRepMsgType action, StringInfo s) { - TransactionId xid; + TransactionId current_xid = InvalidTransactionId; /* not in streaming mode */ - if (!in_streamed_transaction) + if (!(in_streamed_transaction || am_apply_bgworker())) return false; - Assert(stream_fd != NULL); Assert(TransactionIdIsValid(stream_xid)); /* * We should have received XID of the subxact as the first part of the * message, so extract it. */ - xid = pq_getmsgint(s, 4); + current_xid = pq_getmsgint(s, 4); - if (!TransactionIdIsValid(xid)) + if (!TransactionIdIsValid(current_xid)) ereport(ERROR, (errcode(ERRCODE_PROTOCOL_VIOLATION), errmsg_internal("invalid transaction ID in streamed replication transaction"))); - /* Add the new subxact to the array (unless already there). */ - subxact_info_add(xid); + if (am_apply_bgworker()) + { + /* define a savepoint for a subxact if needed. */ + apply_bgworker_subxact_info_add(current_xid); - /* write the change to the current file */ - stream_write_change(action, s); + return false; + } + else if (apply_bgworker_active()) + { + /* + * If we decided to apply the changes of this transaction in an apply + * background worker, pass the data to the worker. + */ + apply_bgworker_send_data(stream_apply_worker, s->len, s->data); + nchanges += 1; + + /* + * XXX The publisher side doesn't always send relation/type update + * messages after the streaming transaction, so also update the + * relation/type in main apply worker here. See function + * cleanup_rel_sync_cache. + */ + if (action == LOGICAL_REP_MSG_RELATION || + action == LOGICAL_REP_MSG_TYPE) + return false; + } + else + { + /* Add the new subxact to the array (unless already there). */ + subxact_info_add(current_xid); + + /* write the change to the current file */ + stream_write_change(action, s); + } return true; } @@ -844,6 +890,9 @@ apply_handle_commit(StringInfo s) apply_handle_commit_internal(&commit_data); + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); + /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(commit_data.end_lsn); @@ -898,7 +947,9 @@ apply_handle_prepare_internal(LogicalRepPreparedTxnData *prepare_data) * BeginTransactionBlock is necessary to balance the EndTransactionBlock * called within the PrepareTransactionBlock below. */ - BeginTransactionBlock(); + if (!IsTransactionBlock()) + BeginTransactionBlock(); + CommitTransactionCommand(); /* Completes the preceding Begin command. */ /* @@ -950,6 +1001,9 @@ apply_handle_prepare(StringInfo s) in_remote_transaction = false; + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); + /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(prepare_data.end_lsn); @@ -1064,10 +1118,6 @@ apply_handle_rollback_prepared(StringInfo s) /* * Handle STREAM PREPARE. - * - * Logic is in two parts: - * 1. Replay all the spooled operations - * 2. Mark the transaction as prepared */ static void apply_handle_stream_prepare(StringInfo s) @@ -1088,24 +1138,70 @@ apply_handle_stream_prepare(StringInfo s) logicalrep_read_stream_prepare(s, &prepare_data); set_apply_error_context_xact(prepare_data.xid, prepare_data.prepare_lsn); - elog(DEBUG1, "received prepare for streamed transaction %u", prepare_data.xid); + if (am_apply_bgworker()) + { + /* Mark the transaction as prepared. */ + apply_handle_prepare_internal(&prepare_data); - /* Replay all the spooled operations. */ - apply_spooled_messages(prepare_data.xid, prepare_data.prepare_lsn); + CommitTransactionCommand(); - /* Mark the transaction as prepared. */ - apply_handle_prepare_internal(&prepare_data); + pgstat_report_stat(false); - CommitTransactionCommand(); + list_free(subxactlist); + subxactlist = NIL; - pgstat_report_stat(false); + apply_bgworker_set_status(APPLY_BGWORKER_FINISHED); + } + else + { + ApplyBgworkerState *wstate = apply_bgworker_find_or_start(prepare_data.xid, false); - store_flush_position(prepare_data.end_lsn); + elog(DEBUG1, "received prepare for streamed transaction %u", + prepare_data.xid); + + /* + * This is the main apply worker. Check if we are processing this + * transaction in a apply background worker. + */ + if (wstate) + { + apply_bgworker_send_data(wstate, s->len, s->data); + + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_FINISHED); + apply_bgworker_free(wstate); + + pgstat_report_stat(false); + store_flush_position(prepare_data.end_lsn); + } + else + { + /* + * This is the main apply worker and the transaction has been + * serialized to file, replay all the spooled operations. + */ + apply_spooled_messages(prepare_data.xid, prepare_data.prepare_lsn); + + /* Mark the transaction as prepared. */ + apply_handle_prepare_internal(&prepare_data); + + CommitTransactionCommand(); + + pgstat_report_stat(false); + + store_flush_position(prepare_data.end_lsn); + + in_remote_transaction = false; + + /* unlink the files with serialized changes and subxact info. */ + stream_cleanup_files(MyLogicalRepWorker->subid, prepare_data.xid); + } + } in_remote_transaction = false; + stream_apply_worker = NULL; - /* unlink the files with serialized changes and subxact info. */ - stream_cleanup_files(MyLogicalRepWorker->subid, prepare_data.xid); + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(prepare_data.end_lsn); @@ -1155,15 +1251,6 @@ apply_handle_stream_start(StringInfo s) (errcode(ERRCODE_PROTOCOL_VIOLATION), errmsg_internal("duplicate STREAM START message"))); - /* - * Start a transaction on stream start, this transaction will be committed - * on the stream stop unless it is a tablesync worker in which case it - * will be committed after processing all the messages. We need the - * transaction for handling the buffile, used for serializing the - * streaming data and subxact info. - */ - begin_replication_step(); - /* notify handle methods we're processing a remote transaction */ in_streamed_transaction = true; @@ -1177,36 +1264,92 @@ apply_handle_stream_start(StringInfo s) set_apply_error_context_xact(stream_xid, InvalidXLogRecPtr); + if (am_apply_bgworker()) + { + /* + * Make sure the handle apply_dispatch methods are aware we're in a + * remote transaction. + */ + in_remote_transaction = true; + + /* If we are in a apply background worker, begin the transaction */ + AcceptInvalidationMessages(); + maybe_reread_subscription(); + + StartTransactionCommand(); + BeginTransactionBlock(); + CommitTransactionCommand(); + + return; + } + /* - * Initialize the worker's stream_fileset if we haven't yet. This will be - * used for the entire duration of the worker so create it in a permanent - * context. We create this on the very first streaming message from any - * transaction and then use it for this and other streaming transactions. - * Now, we could create a fileset at the start of the worker as well but - * then we won't be sure that it will ever be used. + * This is the main apply worker. Check if there is any free apply + * background worker we can use to process this transaction. */ - if (MyLogicalRepWorker->stream_fileset == NULL) + stream_apply_worker = apply_bgworker_find_or_start(stream_xid, first_segment); + + if (stream_apply_worker) { - MemoryContext oldctx; + /* + * If we have found a free worker or if we are already applying this + * transaction in an apply background worker, then we pass the data to + * that worker. + */ + if (first_segment) + apply_bgworker_send_data(stream_apply_worker, s->len, s->data); - oldctx = MemoryContextSwitchTo(ApplyContext); + nchanges = 0; + elog(DEBUG1, "starting streaming of xid %u", stream_xid); + } - MyLogicalRepWorker->stream_fileset = palloc(sizeof(FileSet)); - FileSetInit(MyLogicalRepWorker->stream_fileset); + /* + * If no worker is available for the first stream start, we start to + * serialize all the changes of the transaction. + */ + else + { + /* + * Start a transaction on stream start, this transaction will be + * committed on the stream stop unless it is a tablesync worker in + * which case it will be committed after processing all the messages. + * We need the transaction for handling the buffile, used for + * serializing the streaming data and subxact info. + */ + begin_replication_step(); - MemoryContextSwitchTo(oldctx); - } + /* + * Initialize the worker's stream_fileset if we haven't yet. This will + * be used for the entire duration of the worker so create it in a + * permanent context. We create this on the very first streaming + * message from any transaction and then use it for this and other + * streaming transactions. Now, we could create a fileset at the start + * of the worker as well but then we won't be sure that it will ever + * be used. + */ + if (MyLogicalRepWorker->stream_fileset == NULL) + { + MemoryContext oldctx; - /* open the spool file for this transaction */ - stream_open_file(MyLogicalRepWorker->subid, stream_xid, first_segment); + oldctx = MemoryContextSwitchTo(ApplyContext); - /* if this is not the first segment, open existing subxact file */ - if (!first_segment) - subxact_info_read(MyLogicalRepWorker->subid, stream_xid); + MyLogicalRepWorker->stream_fileset = palloc(sizeof(FileSet)); + FileSetInit(MyLogicalRepWorker->stream_fileset); - pgstat_report_activity(STATE_RUNNING, NULL); + MemoryContextSwitchTo(oldctx); + } - end_replication_step(); + /* open the spool file for this transaction */ + stream_open_file(MyLogicalRepWorker->subid, stream_xid, first_segment); + + /* if this is not the first segment, open existing subxact file */ + if (!first_segment) + subxact_info_read(MyLogicalRepWorker->subid, stream_xid); + + end_replication_step(); + } + + pgstat_report_activity(STATE_RUNNING, NULL); } /* @@ -1220,44 +1363,47 @@ apply_handle_stream_stop(StringInfo s) (errcode(ERRCODE_PROTOCOL_VIOLATION), errmsg_internal("STREAM STOP message without STREAM START"))); - /* - * Close the file with serialized changes, and serialize information about - * subxacts for the toplevel transaction. - */ - subxact_info_write(MyLogicalRepWorker->subid, stream_xid); - stream_close_file(); + if (apply_bgworker_active()) + { + char action = LOGICAL_REP_MSG_STREAM_STOP; - /* We must be in a valid transaction state */ - Assert(IsTransactionState()); + apply_bgworker_send_data(stream_apply_worker, 1, &action); + apply_bgworker_wait_for(stream_apply_worker, APPLY_BGWORKER_READY); - /* Commit the per-stream transaction */ - CommitTransactionCommand(); + elog(DEBUG1, "stopped streaming of xid %u, %u changes streamed", stream_xid, nchanges); + } + else + { + /* + * Close the file with serialized changes, and serialize information + * about subxacts for the toplevel transaction. + */ + subxact_info_write(MyLogicalRepWorker->subid, stream_xid); + stream_close_file(); - in_streamed_transaction = false; + /* We must be in a valid transaction state */ + Assert(IsTransactionState()); + + /* Commit the per-stream transaction */ + CommitTransactionCommand(); + + /* Reset per-stream context */ + MemoryContextReset(LogicalStreamingContext); + } - /* Reset per-stream context */ - MemoryContextReset(LogicalStreamingContext); + in_streamed_transaction = false; + stream_apply_worker = NULL; pgstat_report_activity(STATE_IDLE, NULL); reset_apply_error_context_info(); } /* - * Handle STREAM abort message. + * Handle STREAM ABORT message when the transaction was spilled to disk. */ static void -apply_handle_stream_abort(StringInfo s) +serialize_stream_abort(TransactionId xid, TransactionId subxid) { - TransactionId xid; - TransactionId subxid; - - if (in_streamed_transaction) - ereport(ERROR, - (errcode(ERRCODE_PROTOCOL_VIOLATION), - errmsg_internal("STREAM ABORT message without STREAM STOP"))); - - logicalrep_read_stream_abort(s, &xid, &subxid); - /* * If the two XIDs are the same, it's in fact abort of toplevel xact, so * just delete the files with serialized info. @@ -1339,8 +1485,137 @@ apply_handle_stream_abort(StringInfo s) end_replication_step(); CommitTransactionCommand(); } +} - reset_apply_error_context_info(); +/* + * Handle STREAM ABORT message. + */ +static void +apply_handle_stream_abort(StringInfo s) +{ + TransactionId xid; + TransactionId subxid; + LogicalRepStreamAbortData abort_data; + bool include_abort_lsn; + + if (in_streamed_transaction) + ereport(ERROR, + (errcode(ERRCODE_PROTOCOL_VIOLATION), + errmsg_internal("STREAM ABORT message without STREAM STOP"))); + + /* Check whether the publisher sends abort_lsn and abort_time. */ + if (am_apply_bgworker()) + include_abort_lsn = MyParallelState->server_version >= 150000; + + logicalrep_read_stream_abort(s, &abort_data, include_abort_lsn); + + xid = abort_data.xid; + subxid = abort_data.subxid; + + if (am_apply_bgworker()) + { + elog(DEBUG1, "[Apply BGW #%u] aborting current transaction xid=%u, subxid=%u", + MyParallelState->n, GetCurrentTransactionIdIfAny(), + GetCurrentSubTransactionId()); + + /* + * Update origin state so we can restart streaming from correct + * position in case of crash. + */ + if (include_abort_lsn) + { + replorigin_session_origin_lsn = abort_data.abort_lsn; + replorigin_session_origin_timestamp = abort_data.abort_time; + } + + /* + * If the two XIDs are the same, it's in fact abort of toplevel xact, + * so just free the subxactlist. + */ + if (subxid == xid) + { + set_apply_error_context_xact(subxid, InvalidXLogRecPtr); + + AbortCurrentTransaction(); + + EndTransactionBlock(false); + CommitTransactionCommand(); + + in_remote_transaction = false; + + list_free(subxactlist); + subxactlist = NIL; + + apply_bgworker_set_status(APPLY_BGWORKER_FINISHED); + } + else + { + /* + * OK, so it's a subxact. Rollback to the savepoint. + * + * We also need to read the subxactlist, determine the offset + * tracked for the subxact, and truncate the list. + */ + int i; + bool found = false; + char spname[MAXPGPATH]; + + set_apply_error_context_xact(subxid, InvalidXLogRecPtr); + snprintf(spname, MAXPGPATH, "savepoint_for_xid_%u", subxid); + + elog(DEBUG1, "[Apply BGW #%u] rolling back to savepoint %s", + MyParallelState->n, spname); + + for (i = list_length(subxactlist) - 1; i >= 0; i--) + { + xid = (TransactionId) list_nth_int(subxactlist, i); + if (xid == subxid) + { + found = true; + break; + } + } + + if (found) + { + RollbackToSavepoint(spname); + CommitTransactionCommand(); + subxactlist = list_truncate(subxactlist, i + 1); + } + + apply_bgworker_set_status(APPLY_BGWORKER_READY); + } + + reset_apply_error_context_info(); + } + else + { + ApplyBgworkerState *wstate = apply_bgworker_find_or_start(xid, false); + + /* + * This is the main apply worker. Check if we are processing this + * transaction in a apply background worker. + */ + if (wstate) + { + apply_bgworker_send_data(wstate, s->len, s->data); + + if (subxid == xid) + { + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_FINISHED); + apply_bgworker_free(wstate); + } + else + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_READY); + } + + /* + * We are in main apply worker and the transaction has been serialized + * to file. + */ + else + serialize_stream_abort(xid, subxid); + } } /* @@ -1462,40 +1737,6 @@ apply_spooled_messages(TransactionId xid, XLogRecPtr lsn) return; } -/* - * Handle STREAM COMMIT message. - */ -static void -apply_handle_stream_commit(StringInfo s) -{ - TransactionId xid; - LogicalRepCommitData commit_data; - - if (in_streamed_transaction) - ereport(ERROR, - (errcode(ERRCODE_PROTOCOL_VIOLATION), - errmsg_internal("STREAM COMMIT message without STREAM STOP"))); - - xid = logicalrep_read_stream_commit(s, &commit_data); - set_apply_error_context_xact(xid, commit_data.commit_lsn); - - elog(DEBUG1, "received commit for streamed transaction %u", xid); - - apply_spooled_messages(xid, commit_data.commit_lsn); - - apply_handle_commit_internal(&commit_data); - - /* unlink the files with serialized changes and subxact info */ - stream_cleanup_files(MyLogicalRepWorker->subid, xid); - - /* Process any tables that are being synchronized in parallel. */ - process_syncing_tables(commit_data.end_lsn); - - pgstat_report_activity(STATE_IDLE, NULL); - - reset_apply_error_context_info(); -} - /* * Helper function for apply_handle_commit and apply_handle_stream_commit. */ @@ -2445,11 +2686,105 @@ apply_handle_truncate(StringInfo s) end_replication_step(); } +/* + * Handle STREAM COMMIT message. + */ +static void +apply_handle_stream_commit(StringInfo s) +{ + LogicalRepCommitData commit_data; + TransactionId xid; + + if (in_streamed_transaction) + ereport(ERROR, + (errcode(ERRCODE_PROTOCOL_VIOLATION), + errmsg_internal("STREAM COMMIT message without STREAM STOP"))); + + xid = logicalrep_read_stream_commit(s, &commit_data); + set_apply_error_context_xact(xid, commit_data.commit_lsn); + + if (am_apply_bgworker()) + { + /* + * Update origin state so we can restart streaming from correct + * position in case of crash. + */ + replorigin_session_origin_lsn = commit_data.end_lsn; + replorigin_session_origin_timestamp = commit_data.committime; + + CommitTransactionCommand(); + EndTransactionBlock(false); + CommitTransactionCommand(); + + in_remote_transaction = false; + + pgstat_report_stat(false); + + list_free(subxactlist); + subxactlist = NIL; + + apply_bgworker_set_status(APPLY_BGWORKER_FINISHED); + } + else + { + /* + * This is the main apply worker. Check if we are processing this + * transaction in an apply background worker. + */ + ApplyBgworkerState *wstate = apply_bgworker_find_or_start(xid, false); + + elog(DEBUG1, "received commit for streamed transaction %u", xid); + + if (wstate) + { + /* Send commit message */ + apply_bgworker_send_data(wstate, s->len, s->data); + + /* Wait for apply background worker to finish */ + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_FINISHED); + + pgstat_report_stat(false); + store_flush_position(commit_data.end_lsn); + stop_skipping_changes(); + + apply_bgworker_free(wstate); + + /* + * The transaction is either non-empty or skipped, so we clear the + * subskiplsn. + */ + clear_subscription_skip_lsn(commit_data.commit_lsn); + } + else + { + /* + * This is the main apply worker and the transaction has been + * serialized to file, replay all the spooled operations. + */ + apply_spooled_messages(xid, commit_data.commit_lsn); + + apply_handle_commit_internal(&commit_data); + + /* unlink the files with serialized changes and subxact info */ + stream_cleanup_files(MyLogicalRepWorker->subid, xid); + } + } + + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); + + /* Process any tables that are being synchronized in parallel. */ + process_syncing_tables(commit_data.end_lsn); + + pgstat_report_activity(STATE_IDLE, NULL); + + reset_apply_error_context_info(); +} /* * Logical replication protocol message dispatcher. */ -static void +void apply_dispatch(StringInfo s) { LogicalRepMsgType action = pq_getmsgbyte(s); @@ -2618,6 +2953,10 @@ store_flush_position(XLogRecPtr remote_lsn) { FlushPosition *flushpos; + /* We only need to collect the LSN in main apply worker */ + if (am_apply_bgworker()) + return; + /* Need to do this in permanent context */ MemoryContextSwitchTo(ApplyContext); @@ -2632,7 +2971,7 @@ store_flush_position(XLogRecPtr remote_lsn) /* Update statistics of the worker. */ -static void +void UpdateWorkerStats(XLogRecPtr last_lsn, TimestampTz send_time, bool reply) { MyLogicalRepWorker->last_lsn = last_lsn; @@ -2794,6 +3133,9 @@ LogicalRepApplyLoop(XLogRecPtr last_received) AcceptInvalidationMessages(); maybe_reread_subscription(); + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); + /* Process any table synchronization changes. */ process_syncing_tables(last_received); } @@ -3095,7 +3437,7 @@ maybe_reread_subscription(void) /* * Callback from subscription syscache invalidation. */ -static void +void subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue) { MySubscriptionValid = false; @@ -3691,7 +4033,7 @@ ApplyWorkerMain(Datum main_arg) originid = replorigin_by_name(originname, true); if (!OidIsValid(originid)) originid = replorigin_create(originname); - replorigin_session_setup(originid); + replorigin_session_setup(originid, true); replorigin_session_origin = originid; origin_startpos = replorigin_session_get_progress(false); CommitTransactionCommand(); @@ -3738,7 +4080,7 @@ ApplyWorkerMain(Datum main_arg) options.proto.logical.publication_names = MySubscription->publications; options.proto.logical.binary = MySubscription->binary; - options.proto.logical.streaming = MySubscription->stream; + options.proto.logical.streaming = (MySubscription->stream != SUBSTREAM_OFF); options.proto.logical.twophase = false; if (!am_tablesync_worker()) @@ -3896,7 +4238,8 @@ clear_subscription_skip_lsn(XLogRecPtr finish_lsn) XLogRecPtr myskiplsn = MySubscription->skiplsn; bool started_tx = false; - if (likely(XLogRecPtrIsInvalid(myskiplsn))) + if (likely(XLogRecPtrIsInvalid(myskiplsn)) || + am_apply_bgworker()) return; if (!IsTransactionState()) @@ -3968,7 +4311,7 @@ clear_subscription_skip_lsn(XLogRecPtr finish_lsn) } /* Error callback to give more context info about the change being applied */ -static void +void apply_error_callback(void *arg) { ApplyErrorCallbackArg *errarg = &apply_error_callback_arg; diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c index 8deae57143..6ebfa24baf 100644 --- a/src/backend/replication/pgoutput/pgoutput.c +++ b/src/backend/replication/pgoutput/pgoutput.c @@ -1833,7 +1833,7 @@ pgoutput_stream_abort(struct LogicalDecodingContext *ctx, Assert(rbtxn_is_streamed(toptxn)); OutputPluginPrepareWrite(ctx, true); - logicalrep_write_stream_abort(ctx->out, toptxn->xid, txn->xid); + logicalrep_write_stream_abort(ctx->out, toptxn->xid, txn, abort_lsn); OutputPluginWrite(ctx, true); cleanup_rel_sync_cache(toptxn->xid, false); diff --git a/src/backend/utils/activity/wait_event.c b/src/backend/utils/activity/wait_event.c index 87c15b9c6f..ba781e6f08 100644 --- a/src/backend/utils/activity/wait_event.c +++ b/src/backend/utils/activity/wait_event.c @@ -388,6 +388,9 @@ pgstat_get_wait_ipc(WaitEventIPC w) case WAIT_EVENT_HASH_GROW_BUCKETS_REINSERT: event_name = "HashGrowBucketsReinsert"; break; + case WAIT_EVENT_LOGICAL_APPLY_WORKER_STATE_CHANGE: + event_name = "LogicalApplyWorkerStateChange"; + break; case WAIT_EVENT_LOGICAL_SYNC_DATA: event_name = "LogicalSyncData"; break; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 7cc9c72e49..6f8b30abb0 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -4450,7 +4450,7 @@ getSubscriptions(Archive *fout) if (fout->remoteVersion >= 140000) appendPQExpBufferStr(query, " s.substream,\n"); else - appendPQExpBufferStr(query, " false AS substream,\n"); + appendPQExpBufferStr(query, " 'f' AS substream,\n"); if (fout->remoteVersion >= 150000) appendPQExpBufferStr(query, @@ -4580,8 +4580,10 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo) if (strcmp(subinfo->subbinary, "t") == 0) appendPQExpBufferStr(query, ", binary = true"); - if (strcmp(subinfo->substream, "f") != 0) + if (strcmp(subinfo->substream, "t") == 0) appendPQExpBufferStr(query, ", streaming = on"); + else if (strcmp(subinfo->substream, "a") == 0) + appendPQExpBufferStr(query, ", streaming = apply"); if (strcmp(subinfo->subtwophasestate, two_phase_disabled) != 0) appendPQExpBufferStr(query, ", two_phase = on"); diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h index d1260f590c..9b394a45fe 100644 --- a/src/include/catalog/pg_subscription.h +++ b/src/include/catalog/pg_subscription.h @@ -68,7 +68,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW bool subbinary; /* True if the subscription wants the * publisher to send data in binary */ - bool substream; /* Stream in-progress transactions. */ + char substream; /* Stream in-progress transactions. + * See SUBSTREAM_xxx constants. */ char subtwophasestate; /* Stream two-phase transactions */ @@ -109,7 +110,7 @@ typedef struct Subscription bool enabled; /* Indicates if the subscription is enabled */ bool binary; /* Indicates if the subscription wants data in * binary format */ - bool stream; /* Allow streaming in-progress transactions. */ + char stream; /* Allow streaming in-progress transactions. */ char twophasestate; /* Allow streaming two-phase transactions */ bool disableonerr; /* Indicates if the subscription should be * automatically disabled if a worker error @@ -120,6 +121,18 @@ typedef struct Subscription List *publications; /* List of publication names to subscribe to */ } Subscription; +/* Disallow streaming in-progress transactions */ +#define SUBSTREAM_OFF 'f' + +/* + * Streaming transactions are written to a temporary file and applied only + * after the transaction is committed on upstream. + */ +#define SUBSTREAM_ON 't' + +/* Streaming transactions are applied immediately via a background worker */ +#define SUBSTREAM_APPLY 'a' + extern Subscription *GetSubscription(Oid subid, bool missing_ok); extern void FreeSubscription(Subscription *sub); extern void DisableSubscription(Oid subid); diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h index a771ab8ff3..7bba77c9e7 100644 --- a/src/include/replication/logicalproto.h +++ b/src/include/replication/logicalproto.h @@ -175,6 +175,17 @@ typedef struct LogicalRepRollbackPreparedTxnData char gid[GIDSIZE]; } LogicalRepRollbackPreparedTxnData; +/* + * Transaction protocol information for stream abort. + */ +typedef struct LogicalRepStreamAbortData +{ + TransactionId xid; + TransactionId subxid; + XLogRecPtr abort_lsn; + TimestampTz abort_time; +} LogicalRepStreamAbortData; + extern void logicalrep_write_begin(StringInfo out, ReorderBufferTXN *txn); extern void logicalrep_read_begin(StringInfo in, LogicalRepBeginData *begin_data); @@ -246,9 +257,11 @@ extern void logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn extern TransactionId logicalrep_read_stream_commit(StringInfo out, LogicalRepCommitData *commit_data); extern void logicalrep_write_stream_abort(StringInfo out, TransactionId xid, - TransactionId subxid); -extern void logicalrep_read_stream_abort(StringInfo in, TransactionId *xid, - TransactionId *subxid); + ReorderBufferTXN *txn, + XLogRecPtr abort_lsn); +extern void logicalrep_read_stream_abort(StringInfo in, + LogicalRepStreamAbortData *abort_data, + bool include_abort_lsn); extern char *logicalrep_message_type(LogicalRepMsgType action); #endif /* LOGICAL_PROTO_H */ diff --git a/src/include/replication/logicalworker.h b/src/include/replication/logicalworker.h index cd1b6e8afc..6a1af7f13c 100644 --- a/src/include/replication/logicalworker.h +++ b/src/include/replication/logicalworker.h @@ -13,6 +13,7 @@ #define LOGICALWORKER_H extern void ApplyWorkerMain(Datum main_arg); +extern void ApplyBgworkerMain(Datum main_arg); extern bool IsLogicalWorker(void); diff --git a/src/include/replication/origin.h b/src/include/replication/origin.h index 2d1b5e5c28..c7389b40a7 100644 --- a/src/include/replication/origin.h +++ b/src/include/replication/origin.h @@ -53,7 +53,7 @@ extern XLogRecPtr replorigin_get_progress(RepOriginId node, bool flush); extern void replorigin_session_advance(XLogRecPtr remote_commit, XLogRecPtr local_commit); -extern void replorigin_session_setup(RepOriginId node); +extern void replorigin_session_setup(RepOriginId node, bool must_acquire); extern void replorigin_session_reset(void); extern XLogRecPtr replorigin_session_get_progress(bool flush); diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h index 4a01f877e5..aba1640f4d 100644 --- a/src/include/replication/reorderbuffer.h +++ b/src/include/replication/reorderbuffer.h @@ -301,6 +301,7 @@ typedef struct ReorderBufferTXN { TimestampTz commit_time; TimestampTz prepare_time; + TimestampTz abort_time; } xact_time; /* @@ -647,9 +648,11 @@ extern void ReorderBufferFinishPrepared(ReorderBuffer *rb, TransactionId xid, extern void ReorderBufferAssignChild(ReorderBuffer *, TransactionId, TransactionId, XLogRecPtr commit_lsn); extern void ReorderBufferCommitChild(ReorderBuffer *, TransactionId, TransactionId, XLogRecPtr commit_lsn, XLogRecPtr end_lsn); -extern void ReorderBufferAbort(ReorderBuffer *, TransactionId, XLogRecPtr lsn); +extern void ReorderBufferAbort(ReorderBuffer *, TransactionId, XLogRecPtr lsn, + TimestampTz abort_time); extern void ReorderBufferAbortOld(ReorderBuffer *, TransactionId xid); -extern void ReorderBufferForget(ReorderBuffer *, TransactionId, XLogRecPtr lsn); +extern void ReorderBufferForget(ReorderBuffer *, TransactionId, XLogRecPtr lsn, + TimestampTz abort_time); extern void ReorderBufferInvalidate(ReorderBuffer *, TransactionId, XLogRecPtr lsn); extern void ReorderBufferSetBaseSnapshot(ReorderBuffer *, TransactionId, XLogRecPtr lsn, struct SnapshotData *snap); diff --git a/src/include/replication/worker_internal.h b/src/include/replication/worker_internal.h index 901845abc2..5b9fea2c8c 100644 --- a/src/include/replication/worker_internal.h +++ b/src/include/replication/worker_internal.h @@ -17,8 +17,11 @@ #include "access/xlogdefs.h" #include "catalog/pg_subscription.h" #include "datatype/timestamp.h" +#include "replication/logicalrelation.h" #include "storage/fileset.h" #include "storage/lock.h" +#include "storage/shm_mq.h" +#include "storage/shm_toc.h" #include "storage/spin.h" @@ -60,6 +63,9 @@ typedef struct LogicalRepWorker */ FileSet *stream_fileset; + /* Indicates if this slot is used for an apply background worker. */ + bool subworker; + /* Stats. */ XLogRecPtr last_lsn; TimestampTz last_send_time; @@ -68,9 +74,70 @@ typedef struct LogicalRepWorker TimestampTz reply_time; } LogicalRepWorker; +/* Struct for saving and restoring apply errcontext information */ +typedef struct ApplyErrorCallbackArg +{ + LogicalRepMsgType command; /* 0 if invalid */ + LogicalRepRelMapEntry *rel; + + /* Remote node information */ + int remote_attnum; /* -1 if invalid */ + TransactionId remote_xid; + XLogRecPtr finish_lsn; + char *origin_name; +} ApplyErrorCallbackArg; + +/* + * Status for apply background worker. + */ +typedef enum ApplyBgworkerStatus +{ + APPLY_BGWORKER_ATTACHED = 0, + APPLY_BGWORKER_READY, + APPLY_BGWORKER_BUSY, + APPLY_BGWORKER_FINISHED, + APPLY_BGWORKER_EXIT +} ApplyBgworkerStatus; + +/* + * Shared information among apply workers. + */ +typedef struct ApplyBgworkerShared +{ + slock_t mutex; + + /* Status for apply background worker. */ + ApplyBgworkerStatus status; + + /* server version of publisher. */ + int server_version; + + TransactionId stream_xid; + uint32 n; /* id of apply background worker */ +} ApplyBgworkerShared; + +/* + * Struct for maintaining an apply background worker. + */ +typedef struct ApplyBgworkerState +{ + shm_mq_handle *mq_handle; + dsm_segment *dsm_seg; + ApplyBgworkerShared volatile *pstate; +} ApplyBgworkerState; + /* Main memory context for apply worker. Permanent during worker lifetime. */ extern PGDLLIMPORT MemoryContext ApplyContext; +extern PGDLLIMPORT MemoryContext ApplyMessageContext; + +extern PGDLLIMPORT ApplyErrorCallbackArg apply_error_callback_arg; + +extern PGDLLIMPORT bool MySubscriptionValid; + +extern PGDLLIMPORT volatile ApplyBgworkerShared *MyParallelState; +extern PGDLLIMPORT List *subxactlist; + /* libpqreceiver connection */ extern PGDLLIMPORT struct WalReceiverConn *LogRepWorkerWalRcvConn; @@ -79,13 +146,16 @@ extern PGDLLIMPORT Subscription *MySubscription; extern PGDLLIMPORT LogicalRepWorker *MyLogicalRepWorker; extern PGDLLIMPORT bool in_remote_transaction; +extern PGDLLIMPORT bool in_streamed_transaction; +extern PGDLLIMPORT TransactionId stream_xid; extern void logicalrep_worker_attach(int slot); extern LogicalRepWorker *logicalrep_worker_find(Oid subid, Oid relid, bool only_running); extern List *logicalrep_workers_find(Oid subid, bool only_running); -extern void logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, - Oid userid, Oid relid); +extern bool logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, + Oid userid, Oid relid, + dsm_handle subworker_dsm); extern void logicalrep_worker_stop(Oid subid, Oid relid); extern void logicalrep_worker_wakeup(Oid subid, Oid relid); extern void logicalrep_worker_wakeup_ptr(LogicalRepWorker *worker); @@ -103,10 +173,39 @@ extern void process_syncing_tables(XLogRecPtr current_lsn); extern void invalidate_syncing_table_states(Datum arg, int cacheid, uint32 hashvalue); +extern void UpdateWorkerStats(XLogRecPtr last_lsn, TimestampTz send_time, + bool reply); + +/* prototype needed because of stream_commit */ +extern void apply_dispatch(StringInfo s); + +/* Function for apply error callback */ +extern void apply_error_callback(void *arg); + +extern void subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue); + +/* apply background worker setup and interactions */ +extern ApplyBgworkerState *apply_bgworker_find_or_start(TransactionId xid, + bool start); +extern void apply_bgworker_wait_for(ApplyBgworkerState *wstate, + ApplyBgworkerStatus wait_for_status); +extern void apply_bgworker_send_data(ApplyBgworkerState *wstate, Size nbytes, + const void *data); +extern void apply_bgworker_free(ApplyBgworkerState *wstate); +extern void apply_bgworker_check_status(void); +extern void apply_bgworker_set_status(ApplyBgworkerStatus status); +extern void apply_bgworker_subxact_info_add(TransactionId current_xid); + static inline bool am_tablesync_worker(void) { return OidIsValid(MyLogicalRepWorker->relid); } +static inline bool +am_apply_bgworker(void) +{ + return MyLogicalRepWorker->subworker; +} + #endif /* WORKER_INTERNAL_H */ diff --git a/src/include/utils/wait_event.h b/src/include/utils/wait_event.h index b578e2ec75..c2d2a114d7 100644 --- a/src/include/utils/wait_event.h +++ b/src/include/utils/wait_event.h @@ -105,6 +105,7 @@ typedef enum WAIT_EVENT_HASH_GROW_BUCKETS_ALLOCATE, WAIT_EVENT_HASH_GROW_BUCKETS_ELECT, WAIT_EVENT_HASH_GROW_BUCKETS_REINSERT, + WAIT_EVENT_LOGICAL_APPLY_WORKER_STATE_CHANGE, WAIT_EVENT_LOGICAL_SYNC_DATA, WAIT_EVENT_LOGICAL_SYNC_STATE_CHANGE, WAIT_EVENT_MQ_INTERNAL, diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out index 7fcfad1591..f769835af0 100644 --- a/src/test/regress/expected/subscription.out +++ b/src/test/regress/expected/subscription.out @@ -197,7 +197,7 @@ ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); DROP SUBSCRIPTION regress_testsub; -- fail - streaming must be boolean CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = foo); -ERROR: streaming requires a Boolean value +ERROR: streaming requires a Boolean value or "apply" -- now it works CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = true); WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 4fb746930a..664d4b8b1a 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -116,6 +116,10 @@ Append AppendPath AppendRelInfo AppendState +ApplyBgworkerEntry +ApplyBgworkerShared +ApplyBgworkerState +ApplyBgworkerStatus ApplyErrorCallbackArg ApplyExecutionData ApplySubXactData @@ -1483,6 +1487,7 @@ LogicalRepRelId LogicalRepRelMapEntry LogicalRepRelation LogicalRepRollbackPreparedTxnData +LogicalRepStreamAbortData LogicalRepTupleData LogicalRepTyp LogicalRepWorker -- 2.23.0.windows.1 [application/octet-stream] v9-0002-Test-streaming-apply-option-in-tap-test.patch (68.9K, ../../OS3PR01MB6275208A2F8ED832710F65E09EA49@OS3PR01MB6275.jpnprd01.prod.outlook.com/3-v9-0002-Test-streaming-apply-option-in-tap-test.patch) download | inline diff: From 173558e34a3b61d92a44d13c53732a4325454b70 Mon Sep 17 00:00:00 2001 From: "shiy.fnst" <[email protected]> Date: Fri, 13 May 2022 14:50:30 +0800 Subject: [PATCH v9 2/4] Test streaming apply option in tap test Change all TAP tests using the SUBSCRIPTION "streaming" option, so they now test both 'on' and 'apply' values. --- src/test/subscription/t/015_stream.pl | 199 ++++--- src/test/subscription/t/016_stream_subxact.pl | 119 +++-- src/test/subscription/t/017_stream_ddl.pl | 188 ++++--- .../t/018_stream_subxact_abort.pl | 195 ++++--- .../t/019_stream_subxact_ddl_abort.pl | 110 +++- .../subscription/t/022_twophase_cascade.pl | 363 +++++++------ .../subscription/t/023_twophase_stream.pl | 498 ++++++++++-------- 7 files changed, 1035 insertions(+), 637 deletions(-) diff --git a/src/test/subscription/t/015_stream.pl b/src/test/subscription/t/015_stream.pl index 6561b189de..b3d84b1c7c 100644 --- a/src/test/subscription/t/015_stream.pl +++ b/src/test/subscription/t/015_stream.pl @@ -8,6 +8,116 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" option +# so the same code can be run both for the streaming=on and streaming=apply +# cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_apply) = @_; + + # Interleave a pair of transactions, each exceeding the 64kB limit. + my $in = ''; + my $out = ''; + + my $timer = IPC::Run::timeout($PostgreSQL::Test::Utils::timeout_default); + + my $h = $node_publisher->background_psql('postgres', \$in, \$out, $timer, + on_error_stop => 0); + + # Check that a background worker starts if "streaming" option is + # specified as "apply". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_apply) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + $in .= q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + }; + $h->pump_nb; + + if ($is_apply) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001, 9999) s(i); + DELETE FROM test_tab WHERE a > 5000; + COMMIT; + }); + + $in .= q{ + COMMIT; + \q + }; + $h->finish; # errors make the next test fail, so ignore them here + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), + 'check extra columns contain local defaults'); + + # Test the streaming in binary mode + $node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET (binary = on)"); + + # Insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001, 10000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + COMMIT; + }); + + $node_publisher->wait_for_catchup($appname); + + $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(6667|6667|6667), + 'check extra columns contain local defaults'); + + # Change the local values of the extra columns on the subscriber, + # update publisher, and check that subscriber retains the expected + # values. This is to ensure that non-streaming transactions behave + # properly after a streaming transaction. + $node_subscriber->safe_psql('postgres', + "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'" + ); + $node_publisher->safe_psql('postgres', + "UPDATE test_tab SET b = md5(a::text)"); + + $node_publisher->wait_for_catchup($appname); + + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab" + ); + is($result, qq(6667|6667|6667), + 'check extra columns contain locally changed data'); + + # Cleanup the test data + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE (a > 2)"); + $node_publisher->wait_for_catchup($appname); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -37,6 +147,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -54,82 +168,25 @@ my $result = "SELECT count(*), count(c), count(d = 999) FROM test_tab"); is($result, qq(2|2|2), 'check initial data was copied to subscriber'); -# Interleave a pair of transactions, each exceeding the 64kB limit. -my $in = ''; -my $out = ''; - -my $timer = IPC::Run::timeout($PostgreSQL::Test::Utils::timeout_default); - -my $h = $node_publisher->background_psql('postgres', \$in, \$out, $timer, - on_error_stop => 0); - -$in .= q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -}; -$h->pump_nb; - -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001, 9999) s(i); -DELETE FROM test_tab WHERE a > 5000; -COMMIT; -}); - -$in .= q{ -COMMIT; -\q -}; -$h->finish; # errors make the next test fail, so ignore them here - -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), 'check extra columns contain local defaults'); - -# Test the streaming in binary mode -$node_subscriber->safe_psql('postgres', - "ALTER SUBSCRIPTION tap_sub SET (binary = on)"); - -# Insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001, 10000) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -COMMIT; -}); - -$node_publisher->wait_for_catchup($appname); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(6667|6667|6667), 'check extra columns contain local defaults'); +################################### +# Test using streaming mode 'apply' +################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" +); -# Change the local values of the extra columns on the subscriber, -# update publisher, and check that subscriber retains the expected -# values. This is to ensure that non-streaming transactions behave -# properly after a streaming transaction. $node_subscriber->safe_psql('postgres', - "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'" -); -$node_publisher->safe_psql('postgres', - "UPDATE test_tab SET b = md5(a::text)"); + "ALTER SUBSCRIPTION tap_sub SET(streaming = apply, binary = off)"); -$node_publisher->wait_for_catchup($appname); +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab" -); -is($result, qq(6667|6667|6667), - 'check extra columns contain locally changed data'); +test_streaming($node_publisher, $node_subscriber, $appname, 1); $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/016_stream_subxact.pl b/src/test/subscription/t/016_stream_subxact.pl index f27f1694f2..b5b6dc379f 100644 --- a/src/test/subscription/t/016_stream_subxact.pl +++ b/src/test/subscription/t/016_stream_subxact.pl @@ -8,6 +8,72 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" option +# so the same code can be run both for the streaming=on and streaming=apply +# cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_apply) = @_; + + # Check that a background worker starts if "streaming" option is + # specified as "apply". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_apply) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + # Insert, update and delete enough rows to exceed 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series( 3, 500) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501, 1000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001, 1500) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + SAVEPOINT s3; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501, 2000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + SAVEPOINT s4; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001, 2500) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + COMMIT; + }); + + if ($is_apply) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(1667|1667|1667), + 'check data was copied to subscriber in streaming mode and extra columns contain local defaults' + ); + + # Cleanup the test data + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE (a > 2)"); + $node_publisher->wait_for_catchup($appname); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -37,6 +103,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -54,41 +124,26 @@ my $result = "SELECT count(*), count(c), count(d = 999) FROM test_tab"); is($result, qq(2|2|2), 'check initial data was copied to subscriber'); -# Insert, update and delete enough rows to exceed 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series( 3, 500) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501, 1000) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001, 1500) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -SAVEPOINT s3; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501, 2000) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -SAVEPOINT s4; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001, 2500) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -COMMIT; -}); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(1667|1667|1667), - 'check data was copied to subscriber in streaming mode and extra columns contain local defaults' +################################### +# Test using streaming mode 'apply' +################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" ); +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = apply)"); + +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; + +test_streaming($node_publisher, $node_subscriber, $appname, 1); + $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/017_stream_ddl.pl b/src/test/subscription/t/017_stream_ddl.pl index 0bce63b716..7e5dc18cd2 100644 --- a/src/test/subscription/t/017_stream_ddl.pl +++ b/src/test/subscription/t/017_stream_ddl.pl @@ -8,6 +8,111 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" option +# so the same code can be run both for the streaming=on and streaming=apply +# cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_apply) = @_; + + # a small (non-streamed) transaction with DDL and DML + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab VALUES (3, md5(3::text)); + ALTER TABLE test_tab ADD COLUMN c INT; + SAVEPOINT s1; + INSERT INTO test_tab VALUES (4, md5(4::text), -4); + COMMIT; + }); + + # Check that a background worker starts if "streaming" option is + # specified as "apply". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_apply) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + # large (streamed) transaction with DDL and DML + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(5, 1000) s(i); + ALTER TABLE test_tab ADD COLUMN d INT; + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001, 2000) s(i); + COMMIT; + }); + + if ($is_apply) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + # a small (non-streamed) transaction with DDL and DML + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab VALUES (2001, md5(2001::text), -2001, 2*2001); + ALTER TABLE test_tab ADD COLUMN e INT; + SAVEPOINT s1; + INSERT INTO test_tab VALUES (2002, md5(2002::text), -2002, 2*2002, -3*2002); + COMMIT; + }); + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d), count(e) FROM test_tab"); + is($result, qq(2002|1999|1002|1), + 'check data was copied to subscriber in streaming mode and extra columns contain local defaults' + ); + + # A large (streamed) transaction with DDL and DML. One of the DDL is performed + # after DML to ensure that we invalidate the schema sent for test_tab so that + # the next transaction has to send the schema again. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(2003,5000) s(i); + ALTER TABLE test_tab ADD COLUMN f INT; + COMMIT; + }); + + # A small transaction that won't get streamed. This is just to ensure that we + # send the schema again to reflect the last column added in the previous test. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i, 4*i FROM generate_series(5001,5005) s(i); + COMMIT; + }); + + $node_publisher->wait_for_catchup($appname); + + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d), count(e), count(f) FROM test_tab" + ); + is($result, qq(5005|5002|4005|3004|5), + 'check data was copied to subscriber for both streaming and non-streaming transactions' + ); + + # Cleanup the test data + $node_publisher->safe_psql( + 'postgres', q{ + DELETE FROM test_tab WHERE (a > 2); + ALTER TABLE test_tab DROP COLUMN c, DROP COLUMN d, DROP COLUMN e, DROP COLUMN f; + }); + $node_publisher->wait_for_catchup($appname); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -37,6 +142,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -54,76 +163,25 @@ my $result = "SELECT count(*), count(c), count(d = 999) FROM test_tab"); is($result, qq(2|0|0), 'check initial data was copied to subscriber'); -# a small (non-streamed) transaction with DDL and DML -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab VALUES (3, md5(3::text)); -ALTER TABLE test_tab ADD COLUMN c INT; -SAVEPOINT s1; -INSERT INTO test_tab VALUES (4, md5(4::text), -4); -COMMIT; -}); - -# large (streamed) transaction with DDL and DML -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(5, 1000) s(i); -ALTER TABLE test_tab ADD COLUMN d INT; -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001, 2000) s(i); -COMMIT; -}); - -# a small (non-streamed) transaction with DDL and DML -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab VALUES (2001, md5(2001::text), -2001, 2*2001); -ALTER TABLE test_tab ADD COLUMN e INT; -SAVEPOINT s1; -INSERT INTO test_tab VALUES (2002, md5(2002::text), -2002, 2*2002, -3*2002); -COMMIT; -}); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d), count(e) FROM test_tab"); -is($result, qq(2002|1999|1002|1), - 'check data was copied to subscriber in streaming mode and extra columns contain local defaults' +################################### +# Test using streaming mode 'apply' +################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" ); -# A large (streamed) transaction with DDL and DML. One of the DDL is performed -# after DML to ensure that we invalidate the schema sent for test_tab so that -# the next transaction has to send the schema again. -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(2003,5000) s(i); -ALTER TABLE test_tab ADD COLUMN f INT; -COMMIT; -}); - -# A small transaction that won't get streamed. This is just to ensure that we -# send the schema again to reflect the last column added in the previous test. -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i, 4*i FROM generate_series(5001,5005) s(i); -COMMIT; -}); +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = apply)"); -$node_publisher->wait_for_catchup($appname); +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d), count(e), count(f) FROM test_tab"); -is($result, qq(5005|5002|4005|3004|5), - 'check data was copied to subscriber for both streaming and non-streaming transactions' -); +test_streaming($node_publisher, $node_subscriber, $appname, 1); $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/018_stream_subxact_abort.pl b/src/test/subscription/t/018_stream_subxact_abort.pl index 7155442e76..71f8c1dd05 100644 --- a/src/test/subscription/t/018_stream_subxact_abort.pl +++ b/src/test/subscription/t/018_stream_subxact_abort.pl @@ -8,6 +8,113 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" option +# so the same code can be run both for the streaming=on and streaming=apply +# cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_apply) = @_; + + # Check that a background worker starts if "streaming" option is + # specified as "apply". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_apply) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + # large (streamed) transaction with DDL, DML and ROLLBACKs + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i); + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501,1000) s(i); + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001,1500) s(i); + SAVEPOINT s3; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501,2000) s(i); + ROLLBACK TO s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001,2500) s(i); + ROLLBACK TO s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2501,3000) s(i); + SAVEPOINT s4; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3001,3500) s(i); + SAVEPOINT s5; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3501,4000) s(i); + COMMIT; + }); + + if ($is_apply) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c) FROM test_tab"); + is($result, qq(2000|0), + 'check rollback to savepoint was reflected on subscriber and extra columns contain local defaults' + ); + + # large (streamed) transaction with subscriber receiving out of order + # subtransaction ROLLBACKs + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(4001,4500) s(i); + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001,5500) s(i); + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(6001,6500) s(i); + SAVEPOINT s3; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(7001,7500) s(i); + RELEASE s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(8001,8500) s(i); + ROLLBACK TO s1; + COMMIT; + }); + + $node_publisher->wait_for_catchup($appname); + + $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c) FROM test_tab"); + is($result, qq(2500|0), + 'check rollback to savepoint was reflected on subscriber'); + + # large (streamed) transaction with subscriber receiving rollback + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(8501,9000) s(i); + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(9001,9500) s(i); + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(9501,10000) s(i); + ROLLBACK; + }); + + $node_publisher->wait_for_catchup($appname); + + $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c) FROM test_tab"); + is($result, qq(2500|0), 'check rollback was reflected on subscriber'); + + # Cleanup the test data + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE (a > 2)"); + $node_publisher->wait_for_catchup($appname); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -36,6 +143,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -53,81 +164,25 @@ my $result = "SELECT count(*), count(c) FROM test_tab"); is($result, qq(2|0), 'check initial data was copied to subscriber'); -# large (streamed) transaction with DDL, DML and ROLLBACKs -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i); -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501,1000) s(i); -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001,1500) s(i); -SAVEPOINT s3; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501,2000) s(i); -ROLLBACK TO s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001,2500) s(i); -ROLLBACK TO s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2501,3000) s(i); -SAVEPOINT s4; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3001,3500) s(i); -SAVEPOINT s5; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3501,4000) s(i); -COMMIT; -}); - -$node_publisher->wait_for_catchup($appname); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c) FROM test_tab"); -is($result, qq(2000|0), - 'check rollback to savepoint was reflected on subscriber and extra columns contain local defaults' +################################### +# Test using streaming mode 'apply' +################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" ); -# large (streamed) transaction with subscriber receiving out of order -# subtransaction ROLLBACKs -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(4001,4500) s(i); -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001,5500) s(i); -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(6001,6500) s(i); -SAVEPOINT s3; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(7001,7500) s(i); -RELEASE s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(8001,8500) s(i); -ROLLBACK TO s1; -COMMIT; -}); - -$node_publisher->wait_for_catchup($appname); +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = apply)"); -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c) FROM test_tab"); -is($result, qq(2500|0), - 'check rollback to savepoint was reflected on subscriber'); - -# large (streamed) transaction with subscriber receiving rollback -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(8501,9000) s(i); -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(9001,9500) s(i); -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(9501,10000) s(i); -ROLLBACK; -}); +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c) FROM test_tab"); -is($result, qq(2500|0), 'check rollback was reflected on subscriber'); +test_streaming($node_publisher, $node_subscriber, $appname, 1); $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/019_stream_subxact_ddl_abort.pl b/src/test/subscription/t/019_stream_subxact_ddl_abort.pl index dbd0fca4d1..d0cd869430 100644 --- a/src/test/subscription/t/019_stream_subxact_ddl_abort.pl +++ b/src/test/subscription/t/019_stream_subxact_ddl_abort.pl @@ -9,6 +9,69 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" option +# so the same code can be run both for the streaming=on and streaming=apply +# cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_apply) = @_; + + # Check that a background worker starts if "streaming" option is + # specified as "apply". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_apply) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + # large (streamed) transaction with DDL, DML and ROLLBACKs + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i); + ALTER TABLE test_tab ADD COLUMN c INT; + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(501,1000) s(i); + ALTER TABLE test_tab ADD COLUMN d INT; + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001,1500) s(i); + ALTER TABLE test_tab ADD COLUMN e INT; + SAVEPOINT s3; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(1501,2000) s(i); + ALTER TABLE test_tab DROP COLUMN c; + ROLLBACK TO s1; + INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(501,1000) s(i); + COMMIT; + }); + + if ($is_apply) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c) FROM test_tab"); + is($result, qq(1000|500), + 'check rollback to savepoint was reflected on subscriber and extra columns contain local defaults' + ); + + # Cleanup the test data + $node_publisher->safe_psql( + 'postgres', q{ + DELETE FROM test_tab WHERE (a > 2); + ALTER TABLE test_tab DROP COLUMN c; + }); + $node_publisher->wait_for_catchup($appname); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -37,6 +100,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -54,35 +121,26 @@ my $result = "SELECT count(*), count(c) FROM test_tab"); is($result, qq(2|0), 'check initial data was copied to subscriber'); -# large (streamed) transaction with DDL, DML and ROLLBACKs -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i); -ALTER TABLE test_tab ADD COLUMN c INT; -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(501,1000) s(i); -ALTER TABLE test_tab ADD COLUMN d INT; -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001,1500) s(i); -ALTER TABLE test_tab ADD COLUMN e INT; -SAVEPOINT s3; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(1501,2000) s(i); -ALTER TABLE test_tab DROP COLUMN c; -ROLLBACK TO s1; -INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(501,1000) s(i); -COMMIT; -}); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c) FROM test_tab"); -is($result, qq(1000|500), - 'check rollback to savepoint was reflected on subscriber and extra columns contain local defaults' +################################### +# Test using streaming mode 'apply' +################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" ); +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = apply)"); + +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; + +test_streaming($node_publisher, $node_subscriber, $appname, 1); + $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/022_twophase_cascade.pl b/src/test/subscription/t/022_twophase_cascade.pl index 7a797f37ba..b52af74e35 100644 --- a/src/test/subscription/t/022_twophase_cascade.pl +++ b/src/test/subscription/t/022_twophase_cascade.pl @@ -11,6 +11,208 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" option +# so the same code can be run both for the streaming=on and streaming=apply +# cases. +sub test_streaming +{ + my ($node_A, $node_B, $node_C, $appname_B, $appname_C, $streaming_mode) = + @_; + + my $oldpid_B = $node_A->safe_psql( + 'postgres', " + SELECT pid FROM pg_stat_replication + WHERE application_name = '$appname_B' AND state = 'streaming';"); + my $oldpid_C = $node_B->safe_psql( + 'postgres', " + SELECT pid FROM pg_stat_replication + WHERE application_name = '$appname_C' AND state = 'streaming';"); + + # Setup logical replication streaming mode + + $node_B->safe_psql( + 'postgres', " + ALTER SUBSCRIPTION tap_sub_B + SET (streaming = $streaming_mode);"); + $node_C->safe_psql( + 'postgres', " + ALTER SUBSCRIPTION tap_sub_C + SET (streaming = $streaming_mode)"); + + # Wait for subscribers to finish initialization + + $node_A->poll_query_until( + 'postgres', " + SELECT pid != $oldpid_B FROM pg_stat_replication + WHERE application_name = '$appname_B' AND state = 'streaming';" + ) or die "Timed out while waiting for apply to restart"; + $node_B->poll_query_until( + 'postgres', " + SELECT pid != $oldpid_C FROM pg_stat_replication + WHERE application_name = '$appname_C' AND state = 'streaming';" + ) or die "Timed out while waiting for apply to restart"; + + ############################### + # Test 2PC PREPARE / COMMIT PREPARED. + # 1. Data is streamed as a 2PC transaction. + # 2. Then do commit prepared. + # + # Expect all data is replicated on subscriber(s) after the commit. + ############################### + + # Check that a background worker starts if "streaming" option is + # specified as "apply". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($streaming_mode eq 'apply') + { + $node_B->append_conf('postgresql.conf', "log_min_messages = debug1"); + $node_B->reload; + + $node_C->append_conf('postgresql.conf', "log_min_messages = debug1"); + $node_C->reload; + } + + # Insert, update and delete enough rows to exceed the 64kB limit. + # Then 2PC PREPARE + $node_A->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + if ($streaming_mode eq 'apply') + { + $node_B->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_B->append_conf('postgresql.conf', "log_min_messages = warning"); + $node_B->reload; + + $node_C->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_C->append_conf('postgresql.conf', "log_min_messages = warning"); + $node_C->reload; + } + + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); + + # check the transaction state is prepared on subscriber(s) + my $result = + $node_B->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber B'); + $result = + $node_C->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber C'); + + # 2PC COMMIT + $node_A->safe_psql('postgres', "COMMIT PREPARED 'test_prepared_tab';"); + + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); + + # check that transaction was committed on subscriber(s) + $result = $node_B->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), + 'Rows inserted by 2PC have committed on subscriber B, and extra columns have local defaults' + ); + $result = $node_C->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), + 'Rows inserted by 2PC have committed on subscriber C, and extra columns have local defaults' + ); + + # check the transaction state is ended on subscriber(s) + $result = + $node_B->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is committed on subscriber B'); + $result = + $node_C->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is committed on subscriber C'); + + ############################### + # Test 2PC PREPARE with a nested ROLLBACK TO SAVEPOINT. + # 0. Cleanup from previous test leaving only 2 rows. + # 1. Insert one more row. + # 2. Record a SAVEPOINT. + # 3. Data is streamed using 2PC. + # 4. Do rollback to SAVEPOINT prior to the streamed inserts. + # 5. Then COMMIT PREPARED. + # + # Expect data after the SAVEPOINT is aborted leaving only 3 rows (= 2 original + 1 from step 1). + ############################### + + # First, delete the data except for 2 rows (delete will be replicated) + $node_A->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); + + # 2PC PREPARE with a nested ROLLBACK TO SAVEPOINT + $node_A->safe_psql( + 'postgres', " + BEGIN; + INSERT INTO test_tab VALUES (9999, 'foobar'); + SAVEPOINT sp_inner; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + ROLLBACK TO SAVEPOINT sp_inner; + PREPARE TRANSACTION 'outer'; + "); + + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); + + # check the transaction state prepared on subscriber(s) + $result = + $node_B->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber B'); + $result = + $node_C->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber C'); + + # 2PC COMMIT + $node_A->safe_psql('postgres', "COMMIT PREPARED 'outer';"); + + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); + + # check the transaction state is ended on subscriber + $result = + $node_B->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is ended on subscriber B'); + $result = + $node_C->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is ended on subscriber C'); + + # check inserts are visible at subscriber(s). + # All the streamed data (prior to the SAVEPOINT) should be rolled back. + # (9999, 'foobar') should be committed. + $result = $node_B->safe_psql('postgres', + "SELECT count(*) FROM test_tab where b = 'foobar';"); + is($result, qq(1), 'Rows committed are present on subscriber B'); + $result = + $node_B->safe_psql('postgres', "SELECT count(*) FROM test_tab;"); + is($result, qq(3), 'Rows committed are present on subscriber B'); + $result = $node_C->safe_psql('postgres', + "SELECT count(*) FROM test_tab where b = 'foobar';"); + is($result, qq(1), 'Rows committed are present on subscriber C'); + $result = + $node_C->safe_psql('postgres', "SELECT count(*) FROM test_tab;"); + is($result, qq(3), 'Rows committed are present on subscriber C'); + + # Cleanup the test data + $node_A->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); +} + ############################### # Setup a cascade of pub/sub nodes. # node_A -> node_B -> node_C @@ -260,160 +462,15 @@ is($result, qq(21), 'Rows committed are present on subscriber C'); # 2PC + STREAMING TESTS # --------------------- -my $oldpid_B = $node_A->safe_psql( - 'postgres', " - SELECT pid FROM pg_stat_replication - WHERE application_name = '$appname_B' AND state = 'streaming';"); -my $oldpid_C = $node_B->safe_psql( - 'postgres', " - SELECT pid FROM pg_stat_replication - WHERE application_name = '$appname_C' AND state = 'streaming';"); - -# Setup logical replication (streaming = on) - -$node_B->safe_psql( - 'postgres', " - ALTER SUBSCRIPTION tap_sub_B - SET (streaming = on);"); -$node_C->safe_psql( - 'postgres', " - ALTER SUBSCRIPTION tap_sub_C - SET (streaming = on)"); - -# Wait for subscribers to finish initialization - -$node_A->poll_query_until( - 'postgres', " - SELECT pid != $oldpid_B FROM pg_stat_replication - WHERE application_name = '$appname_B' AND state = 'streaming';" -) or die "Timed out while waiting for apply to restart"; -$node_B->poll_query_until( - 'postgres', " - SELECT pid != $oldpid_C FROM pg_stat_replication - WHERE application_name = '$appname_C' AND state = 'streaming';" -) or die "Timed out while waiting for apply to restart"; - -############################### -# Test 2PC PREPARE / COMMIT PREPARED. -# 1. Data is streamed as a 2PC transaction. -# 2. Then do commit prepared. -# -# Expect all data is replicated on subscriber(s) after the commit. -############################### - -# Insert, update and delete enough rows to exceed the 64kB limit. -# Then 2PC PREPARE -$node_A->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_A->wait_for_catchup($appname_B); -$node_B->wait_for_catchup($appname_C); +################################ +# Test using streaming mode 'on' +################################ +test_streaming($node_A, $node_B, $node_C, $appname_B, $appname_C, 'on'); -# check the transaction state is prepared on subscriber(s) -$result = - $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber B'); -$result = - $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber C'); - -# 2PC COMMIT -$node_A->safe_psql('postgres', "COMMIT PREPARED 'test_prepared_tab';"); - -$node_A->wait_for_catchup($appname_B); -$node_B->wait_for_catchup($appname_C); - -# check that transaction was committed on subscriber(s) -$result = $node_B->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), - 'Rows inserted by 2PC have committed on subscriber B, and extra columns have local defaults' -); -$result = $node_C->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), - 'Rows inserted by 2PC have committed on subscriber C, and extra columns have local defaults' -); - -# check the transaction state is ended on subscriber(s) -$result = - $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is committed on subscriber B'); -$result = - $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is committed on subscriber C'); - -############################### -# Test 2PC PREPARE with a nested ROLLBACK TO SAVEPOINT. -# 0. Cleanup from previous test leaving only 2 rows. -# 1. Insert one more row. -# 2. Record a SAVEPOINT. -# 3. Data is streamed using 2PC. -# 4. Do rollback to SAVEPOINT prior to the streamed inserts. -# 5. Then COMMIT PREPARED. -# -# Expect data after the SAVEPOINT is aborted leaving only 3 rows (= 2 original + 1 from step 1). -############################### - -# First, delete the data except for 2 rows (delete will be replicated) -$node_A->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); - -# 2PC PREPARE with a nested ROLLBACK TO SAVEPOINT -$node_A->safe_psql( - 'postgres', " - BEGIN; - INSERT INTO test_tab VALUES (9999, 'foobar'); - SAVEPOINT sp_inner; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - ROLLBACK TO SAVEPOINT sp_inner; - PREPARE TRANSACTION 'outer'; - "); - -$node_A->wait_for_catchup($appname_B); -$node_B->wait_for_catchup($appname_C); - -# check the transaction state prepared on subscriber(s) -$result = - $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber B'); -$result = - $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber C'); - -# 2PC COMMIT -$node_A->safe_psql('postgres', "COMMIT PREPARED 'outer';"); - -$node_A->wait_for_catchup($appname_B); -$node_B->wait_for_catchup($appname_C); - -# check the transaction state is ended on subscriber -$result = - $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is ended on subscriber B'); -$result = - $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is ended on subscriber C'); - -# check inserts are visible at subscriber(s). -# All the streamed data (prior to the SAVEPOINT) should be rolled back. -# (9999, 'foobar') should be committed. -$result = $node_B->safe_psql('postgres', - "SELECT count(*) FROM test_tab where b = 'foobar';"); -is($result, qq(1), 'Rows committed are present on subscriber B'); -$result = $node_B->safe_psql('postgres', "SELECT count(*) FROM test_tab;"); -is($result, qq(3), 'Rows committed are present on subscriber B'); -$result = $node_C->safe_psql('postgres', - "SELECT count(*) FROM test_tab where b = 'foobar';"); -is($result, qq(1), 'Rows committed are present on subscriber C'); -$result = $node_C->safe_psql('postgres', "SELECT count(*) FROM test_tab;"); -is($result, qq(3), 'Rows committed are present on subscriber C'); +################################### +# Test using streaming mode 'apply' +################################### +test_streaming($node_A, $node_B, $node_C, $appname_B, $appname_C, 'apply'); ############################### # check all the cleanup diff --git a/src/test/subscription/t/023_twophase_stream.pl b/src/test/subscription/t/023_twophase_stream.pl index d8475d25a4..f55c5c95e7 100644 --- a/src/test/subscription/t/023_twophase_stream.pl +++ b/src/test/subscription/t/023_twophase_stream.pl @@ -8,6 +8,266 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" option +# so the same code can be run both for the streaming=on and streaming=apply +# cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_apply) = @_; + + ############################### + # Test 2PC PREPARE / COMMIT PREPARED. + # 1. Data is streamed as a 2PC transaction. + # 2. Then do commit prepared. + # + # Expect all data is replicated on subscriber side after the commit. + ############################### + + # Check that a background worker starts if "streaming" option is + # specified as "apply". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_apply) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + # check that 2PC gets replicated to subscriber + # Insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + if ($is_apply) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is in prepared state on subscriber + my $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber'); + + # 2PC transaction gets committed + $node_publisher->safe_psql('postgres', + "COMMIT PREPARED 'test_prepared_tab';"); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is committed on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), + 'Rows inserted by 2PC have committed on subscriber, and extra columns contain local defaults' + ); + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is committed on subscriber'); + + ############################### + # Test 2PC PREPARE / ROLLBACK PREPARED. + # 1. Table is deleted back to 2 rows which are replicated on subscriber. + # 2. Data is streamed using 2PC. + # 3. Do rollback prepared. + # + # Expect data rolls back leaving only the original 2 rows. + ############################### + + # First, delete the data except for 2 rows (will be replicated) + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE a > 2;"); + + # Then insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is in prepared state on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber'); + + # 2PC transaction gets aborted + $node_publisher->safe_psql('postgres', + "ROLLBACK PREPARED 'test_prepared_tab';"); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is aborted on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(2|2|2), + 'Rows inserted by 2PC are rolled back, leaving only the original 2 rows' + ); + + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is aborted on subscriber'); + + ############################### + # Check that 2PC COMMIT PREPARED is decoded properly on crash restart. + # 1. insert, update and delete enough rows to exceed the 64kB limit. + # 2. Then server crashes before the 2PC transaction is committed. + # 3. After servers are restarted the pending transaction is committed. + # + # Expect all data is replicated on subscriber side after the commit. + # Note: both publisher and subscriber do crash/restart. + ############################### + + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + $node_subscriber->stop('immediate'); + $node_publisher->stop('immediate'); + + $node_publisher->start; + $node_subscriber->start; + + # commit post the restart + $node_publisher->safe_psql('postgres', + "COMMIT PREPARED 'test_prepared_tab';"); + $node_publisher->wait_for_catchup($appname); + + # check inserts are visible + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), + 'Rows inserted by 2PC have committed on subscriber, and extra columns contain local defaults' + ); + + ############################### + # Do INSERT after the PREPARE but before ROLLBACK PREPARED. + # 1. Table is deleted back to 2 rows which are replicated on subscriber. + # 2. Data is streamed using 2PC. + # 3. A single row INSERT is done which is after the PREPARE. + # 4. Then do a ROLLBACK PREPARED. + # + # Expect the 2PC data rolls back leaving only 3 rows on the subscriber + # (the original 2 + inserted 1). + ############################### + + # First, delete the data except for 2 rows (will be replicated) + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE a > 2;"); + + # Then insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is in prepared state on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber'); + + # Insert a different record (now we are outside of the 2PC transaction) + # Note: the 2PC transaction still holds row locks so make sure this insert is for a separate primary key + $node_publisher->safe_psql('postgres', + "INSERT INTO test_tab VALUES (99999, 'foobar')"); + + # 2PC transaction gets aborted + $node_publisher->safe_psql('postgres', + "ROLLBACK PREPARED 'test_prepared_tab';"); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is aborted on subscriber, + # but the extra INSERT outside of the 2PC still was replicated + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3|3|3), + 'check the outside insert was copied to subscriber'); + + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is aborted on subscriber'); + + ############################### + # Do INSERT after the PREPARE but before COMMIT PREPARED. + # 1. Table is deleted back to 2 rows which are replicated on subscriber. + # 2. Data is streamed using 2PC. + # 3. A single row INSERT is done which is after the PREPARE. + # 4. Then do a COMMIT PREPARED. + # + # Expect 2PC data + the extra row are on the subscriber + # (the 3334 + inserted 1 = 3335). + ############################### + + # First, delete the data except for 2 rows (will be replicated) + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE a > 2;"); + + # Then insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is in prepared state on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber'); + + # Insert a different record (now we are outside of the 2PC transaction) + # Note: the 2PC transaction still holds row locks so make sure this insert is for a separare primary key + $node_publisher->safe_psql('postgres', + "INSERT INTO test_tab VALUES (99999, 'foobar')"); + + # 2PC transaction gets committed + $node_publisher->safe_psql('postgres', + "COMMIT PREPARED 'test_prepared_tab';"); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is committed on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3335|3335|3335), + 'Rows inserted by 2PC (as well as outside insert) have committed on subscriber, and extra columns contain local defaults' + ); + + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is committed on subscriber'); + + # Cleanup the test data + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE a > 2;"); + $node_publisher->wait_for_catchup($appname); +} + ############################### # Setup ############################### @@ -48,6 +308,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql( 'postgres', " CREATE SUBSCRIPTION tap_sub @@ -70,236 +334,30 @@ my $twophase_query = $node_subscriber->poll_query_until('postgres', $twophase_query) or die "Timed out while waiting for subscriber to enable twophase"; -############################### # Check initial data was copied to subscriber -############################### my $result = $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab"); is($result, qq(2|2|2), 'check initial data was copied to subscriber'); -############################### -# Test 2PC PREPARE / COMMIT PREPARED. -# 1. Data is streamed as a 2PC transaction. -# 2. Then do commit prepared. -# -# Expect all data is replicated on subscriber side after the commit. -############################### - -# check that 2PC gets replicated to subscriber -# Insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is in prepared state on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber'); - -# 2PC transaction gets committed -$node_publisher->safe_psql('postgres', - "COMMIT PREPARED 'test_prepared_tab';"); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is committed on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), - 'Rows inserted by 2PC have committed on subscriber, and extra columns contain local defaults' -); -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is committed on subscriber'); - -############################### -# Test 2PC PREPARE / ROLLBACK PREPARED. -# 1. Table is deleted back to 2 rows which are replicated on subscriber. -# 2. Data is streamed using 2PC. -# 3. Do rollback prepared. -# -# Expect data rolls back leaving only the original 2 rows. -############################### - -# First, delete the data except for 2 rows (will be replicated) -$node_publisher->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); - -# Then insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is in prepared state on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber'); - -# 2PC transaction gets aborted -$node_publisher->safe_psql('postgres', - "ROLLBACK PREPARED 'test_prepared_tab';"); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is aborted on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(2|2|2), - 'Rows inserted by 2PC are rolled back, leaving only the original 2 rows'); - -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is aborted on subscriber'); - -############################### -# Check that 2PC COMMIT PREPARED is decoded properly on crash restart. -# 1. insert, update and delete enough rows to exceed the 64kB limit. -# 2. Then server crashes before the 2PC transaction is committed. -# 3. After servers are restarted the pending transaction is committed. -# -# Expect all data is replicated on subscriber side after the commit. -# Note: both publisher and subscriber do crash/restart. -############################### - -$node_publisher->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_subscriber->stop('immediate'); -$node_publisher->stop('immediate'); - -$node_publisher->start; -$node_subscriber->start; - -# commit post the restart -$node_publisher->safe_psql('postgres', - "COMMIT PREPARED 'test_prepared_tab';"); -$node_publisher->wait_for_catchup($appname); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -# check inserts are visible -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), - 'Rows inserted by 2PC have committed on subscriber, and extra columns contain local defaults' +################################### +# Test using streaming mode 'apply' +################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" ); -############################### -# Do INSERT after the PREPARE but before ROLLBACK PREPARED. -# 1. Table is deleted back to 2 rows which are replicated on subscriber. -# 2. Data is streamed using 2PC. -# 3. A single row INSERT is done which is after the PREPARE. -# 4. Then do a ROLLBACK PREPARED. -# -# Expect the 2PC data rolls back leaving only 3 rows on the subscriber -# (the original 2 + inserted 1). -############################### - -# First, delete the data except for 2 rows (will be replicated) -$node_publisher->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); - -# Then insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is in prepared state on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber'); - -# Insert a different record (now we are outside of the 2PC transaction) -# Note: the 2PC transaction still holds row locks so make sure this insert is for a separate primary key -$node_publisher->safe_psql('postgres', - "INSERT INTO test_tab VALUES (99999, 'foobar')"); - -# 2PC transaction gets aborted -$node_publisher->safe_psql('postgres', - "ROLLBACK PREPARED 'test_prepared_tab';"); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is aborted on subscriber, -# but the extra INSERT outside of the 2PC still was replicated -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3|3|3), 'check the outside insert was copied to subscriber'); - -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is aborted on subscriber'); - -############################### -# Do INSERT after the PREPARE but before COMMIT PREPARED. -# 1. Table is deleted back to 2 rows which are replicated on subscriber. -# 2. Data is streamed using 2PC. -# 3. A single row INSERT is done which is after the PREPARE. -# 4. Then do a COMMIT PREPARED. -# -# Expect 2PC data + the extra row are on the subscriber -# (the 3334 + inserted 1 = 3335). -############################### - -# First, delete the data except for 2 rows (will be replicated) -$node_publisher->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); - -# Then insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is in prepared state on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber'); - -# Insert a different record (now we are outside of the 2PC transaction) -# Note: the 2PC transaction still holds row locks so make sure this insert is for a separare primary key -$node_publisher->safe_psql('postgres', - "INSERT INTO test_tab VALUES (99999, 'foobar')"); - -# 2PC transaction gets committed -$node_publisher->safe_psql('postgres', - "COMMIT PREPARED 'test_prepared_tab';"); - -$node_publisher->wait_for_catchup($appname); +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = apply)"); -# check that transaction is committed on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3335|3335|3335), - 'Rows inserted by 2PC (as well as outside insert) have committed on subscriber, and extra columns contain local defaults' -); +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is committed on subscriber'); +test_streaming($node_publisher, $node_subscriber, $appname, 1); ############################### # check all the cleanup -- 2.23.0.windows.1 [application/octet-stream] v9-0003-Add-some-checks-before-using-apply-background-wor.patch (27.1K, ../../OS3PR01MB6275208A2F8ED832710F65E09EA49@OS3PR01MB6275.jpnprd01.prod.outlook.com/4-v9-0003-Add-some-checks-before-using-apply-background-wor.patch) download | inline diff: From 0deb8ef70730b478b88e25e8400bd690416951ec Mon Sep 17 00:00:00 2001 From: wangw <[email protected]> Date: Thu, 2 Jun 2022 17:03:02 +0800 Subject: [PATCH v9 3/4] Add some checks before using apply background worker to apply changes. If any of the following checks are violated, an error will be reported. 1. The unique columns between publisher and subscriber are difference. 2. There is any non-immutable function present in expression in subscriber's relation. Check from the following 3 items: a. The function in triggers; b. Column default value expressions and domain constraints; c. Constraint expressions. --- doc/src/sgml/ref/create_subscription.sgml | 4 + .../replication/logical/applybgwroker.c | 46 ++++ src/backend/replication/logical/proto.c | 62 +++++- src/backend/replication/logical/relation.c | 186 +++++++++++++++- src/backend/replication/logical/tablesync.c | 1 + src/backend/replication/logical/worker.c | 6 + src/backend/utils/cache/typcache.c | 17 ++ src/include/replication/logicalproto.h | 1 + src/include/replication/logicalrelation.h | 15 ++ src/include/replication/worker_internal.h | 1 + src/include/utils/typcache.h | 2 + .../subscription/t/022_twophase_cascade.pl | 7 + .../subscription/t/032_streaming_apply.pl | 206 ++++++++++++++++++ 13 files changed, 549 insertions(+), 5 deletions(-) create mode 100644 src/test/subscription/t/032_streaming_apply.pl diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml index d4caae0222..8de1a23ce4 100644 --- a/doc/src/sgml/ref/create_subscription.sgml +++ b/doc/src/sgml/ref/create_subscription.sgml @@ -240,6 +240,10 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl the transaction is committed. Note that if an error happens when applying changes in a background worker, it might not report the finish LSN of the remote transaction in the server log. + To run in this mode, there are following two requirements. The first + is that the unique column should be the same between publisher and + subscriber; the second is that there should not be any non-immutable + function in subscriber-side replicated table. </para> </listitem> </varlistentry> diff --git a/src/backend/replication/logical/applybgwroker.c b/src/backend/replication/logical/applybgwroker.c index 8af64974f2..0f9081d62f 100644 --- a/src/backend/replication/logical/applybgwroker.c +++ b/src/backend/replication/logical/applybgwroker.c @@ -754,3 +754,49 @@ apply_bgworker_subxact_info_add(TransactionId current_xid) MemoryContextSwitchTo(oldctx); } } + +/* + * Check if changes on this logical replication relation can be applied by + * apply background worker. + * + * Although we maintains the commit order by allowing only one process to + * commit at a time, our access order to the relation has changed. + * This could cause unexpected problems if the unique column on the replicated + * table is inconsistent with the publisher-side or contains non-immutable + * functions when applying transactions in the apply background worker. + */ +void +apply_bgworker_relation_check(LogicalRepRelMapEntry *rel) +{ + /* Check only we are in apply bgworker. */ + if (!am_apply_bgworker()) + return; + + /* + * FIXME: Checks on partitioned tables are currently not supported. Adding + * this check requires additional fixes for other issues. I will add this + * later, after the related issues are fixed. + */ + if (rel->localrel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + return; + + /* + * If any unique index exist, check that they are same as remoterel. + */ + if (!rel->sameunique) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot replicate relation with different unique index"), + errhint("Please change the streaming option to 'on' instead of 'apply'."))); + + /* + * Check if there is any non-immutable function present in expression in + * this relation. + */ + if (rel->volatility == FUNCTION_NONIMMUTABLE) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot replicate relation. There is at least one non-immutable function"), + errhint("Please change the streaming option to 'on' instead of 'apply'."))); + +} diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c index a888038eb4..6af3302270 100644 --- a/src/backend/replication/logical/proto.c +++ b/src/backend/replication/logical/proto.c @@ -23,7 +23,8 @@ /* * Protocol message flags. */ -#define LOGICALREP_IS_REPLICA_IDENTITY 1 +#define LOGICALREP_IS_REPLICA_IDENTITY 0x0001 +#define LOGICALREP_IS_UNIQUE 0x0002 #define MESSAGE_TRANSACTIONAL (1<<0) #define TRUNCATE_CASCADE (1<<0) @@ -933,11 +934,55 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns) TupleDesc desc; int i; uint16 nliveatts = 0; - Bitmapset *idattrs = NULL; + Bitmapset *idattrs = NULL, + *attunique = NULL; bool replidentfull; desc = RelationGetDescr(rel); + if (rel->rd_rel->relhasindex) + { + List *indexoidlist = RelationGetIndexList(rel); + ListCell *indexoidscan; + + foreach(indexoidscan, indexoidlist) + { + Oid indexoid = lfirst_oid(indexoidscan); + Relation indexRel; + + /* Look up the description for index */ + indexRel = RelationIdGetRelation(indexoid); + + if (!RelationIsValid(indexRel)) + elog(ERROR, "could not open relation with OID %u", indexoid); + + if (indexRel->rd_index->indisunique) + { + int i; + + /* Add referenced attributes to idindexattrs */ + for (i = 0; i < indexRel->rd_index->indnatts; i++) + { + int attrnum = indexRel->rd_index->indkey.values[i]; + + /* + * We don't include non-key columns into idindexattrs + * bitmaps. See RelationGetIndexAttrBitmap. + */ + if (attrnum != 0) + { + if (i < indexRel->rd_index->indnkeyatts && + !bms_is_member(attrnum - FirstLowInvalidHeapAttributeNumber, attunique)) + attunique = bms_add_member(attunique, + attrnum - FirstLowInvalidHeapAttributeNumber); + } + } + } + RelationClose(indexRel); + } + list_free(indexoidlist); + } + /* send number of live attributes */ for (i = 0; i < desc->natts; i++) { @@ -976,6 +1021,10 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns) idattrs)) flags |= LOGICALREP_IS_REPLICA_IDENTITY; + if (bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber, + attunique)) + flags |= LOGICALREP_IS_UNIQUE; + pq_sendbyte(out, flags); /* attribute name */ @@ -1001,7 +1050,8 @@ logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel) int natts; char **attnames; Oid *atttyps; - Bitmapset *attkeys = NULL; + Bitmapset *attkeys = NULL, + *attunique = NULL; natts = pq_getmsgint(in, 2); attnames = palloc(natts * sizeof(char *)); @@ -1012,11 +1062,14 @@ logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel) { uint8 flags; - /* Check for replica identity column */ + /* Check for replica identity and unique column */ flags = pq_getmsgbyte(in); if (flags & LOGICALREP_IS_REPLICA_IDENTITY) attkeys = bms_add_member(attkeys, i); + if (flags & LOGICALREP_IS_UNIQUE) + attunique = bms_add_member(attunique, i); + /* attribute name */ attnames[i] = pstrdup(pq_getmsgstring(in)); @@ -1030,6 +1083,7 @@ logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel) rel->attnames = attnames; rel->atttyps = atttyps; rel->attkeys = attkeys; + rel->attunique = attunique; rel->natts = natts; } diff --git a/src/backend/replication/logical/relation.c b/src/backend/replication/logical/relation.c index 80fb561a9a..3cea510453 100644 --- a/src/backend/replication/logical/relation.c +++ b/src/backend/replication/logical/relation.c @@ -19,12 +19,19 @@ #include "access/table.h" #include "catalog/namespace.h" +#include "catalog/pg_proc.h" #include "catalog/pg_subscription_rel.h" +#include "commands/trigger.h" #include "executor/executor.h" #include "nodes/makefuncs.h" +#include "optimizer/optimizer.h" #include "replication/logicalrelation.h" #include "replication/worker_internal.h" +#include "rewrite/rewriteHandler.h" #include "utils/inval.h" +#include "utils/lsyscache.h" +#include "utils/syscache.h" +#include "utils/typcache.h" static MemoryContext LogicalRepRelMapContext = NULL; @@ -91,6 +98,23 @@ logicalrep_relmap_invalidate_cb(Datum arg, Oid reloid) } } +/* + * Reset the flag volatility of all existing entry in the relation map cache. + */ +static void +logicalrep_relmap_reset_volatility_cb(Datum arg, int cacheid, uint32 hashvalue) +{ + HASH_SEQ_STATUS hash_seq; + LogicalRepRelMapEntry *entry; + + if (LogicalRepRelMap == NULL) + return; + + hash_seq_init(&hash_seq, LogicalRepRelMap); + while ((entry = hash_seq_search(&hash_seq)) != NULL) + entry->volatility = FUNCTION_UNKNOWN; +} + /* * Initialize the relation map cache. */ @@ -116,6 +140,9 @@ logicalrep_relmap_init(void) /* Watch for invalidation events. */ CacheRegisterRelcacheCallback(logicalrep_relmap_invalidate_cb, (Datum) 0); + CacheRegisterSyscacheCallback(PROCOID, + logicalrep_relmap_reset_volatility_cb, + (Datum) 0); } /* @@ -142,6 +169,7 @@ logicalrep_relmap_free_entry(LogicalRepRelMapEntry *entry) pfree(remoterel->atttyps); } bms_free(remoterel->attkeys); + bms_free(remoterel->attunique); if (entry->attrmap) pfree(entry->attrmap); @@ -190,6 +218,7 @@ logicalrep_relmap_update(LogicalRepRelation *remoterel) } entry->remoterel.replident = remoterel->replident; entry->remoterel.attkeys = bms_copy(remoterel->attkeys); + entry->remoterel.attunique = bms_copy(remoterel->attunique); MemoryContextSwitchTo(oldctx); } @@ -307,7 +336,8 @@ logicalrep_rel_open(LogicalRepRelId remoteid, LOCKMODE lockmode) if (!entry->localrelvalid) { Oid relid; - Bitmapset *idkey; + Bitmapset *idkey, + *ukey; TupleDesc desc; MemoryContext oldctx; int i; @@ -415,6 +445,159 @@ logicalrep_rel_open(LogicalRepRelId remoteid, LOCKMODE lockmode) } } + /* + * Check whether the unique index of publisher and subscriber are + * consistent. + */ + entry->sameunique = true; + ukey = RelationGetIndexAttrBitmap(entry->localrel, + INDEX_ATTR_BITMAP_KEY); + + if (ukey) + { + i = -1; + while ((i = bms_next_member(ukey, i)) >= 0) + { + int attnum = i + FirstLowInvalidHeapAttributeNumber; + + attnum = AttrNumberGetAttrOffset(attnum); + + if (entry->attrmap->attnums[attnum] < 0 || + !bms_is_member(entry->attrmap->attnums[attnum], remoterel->attunique)) + { + entry->sameunique = false; + break; + } + } + } + + /* + * Check whether there is any non-immutable function in the local + * table. + */ + if (entry->volatility == FUNCTION_UNKNOWN) + { + /* + * Check from the following points: + * + * a. The function in triggers; + * b. Column default value expressions and domain constraints; + * c. Constraint expressions. + */ + bool nonimmutable = false; + + /* Check the trigger functions. */ + if (!nonimmutable && entry->localrel->trigdesc != NULL) + for (i = 0; i < entry->localrel->trigdesc->numtriggers; i++) + { + Trigger *trig = entry->localrel->trigdesc->triggers + i; + + if (trig->tgenabled != TRIGGER_FIRES_ALWAYS && + trig->tgenabled != TRIGGER_FIRES_ON_REPLICA) + continue; + + if (func_volatile(trig->tgfoid) != PROVOLATILE_IMMUTABLE) + { + nonimmutable = true; + break; + } + } + + /* Check the columns. */ + if (!nonimmutable) + { + TupleDesc tupdesc = RelationGetDescr(entry->localrel); + int attnum; + + for (attnum = 0; attnum < tupdesc->natts; attnum++) + { + Form_pg_attribute att = TupleDescAttr(tupdesc, attnum); + + /* break if we set nonimmutable in last check for domain. */ + if (nonimmutable) + break; + + /* We don't need info for dropped or generated attributes */ + if (att->attisdropped || att->attgenerated) + continue; + + /* + * We don't need to check columns that only exist on the + * subscriber + */ + if (entry->attrmap->attnums[attnum] < 0) + continue; + + if (att->atthasdef) + { + Node *defaultexpr; + + defaultexpr = build_column_default(entry->localrel, attnum + 1); + if (contain_mutable_functions(defaultexpr)) + { + nonimmutable = true; + break; + } + } + + /* + * If the column is of a DOMAIN type, determine whether + * that domain has any CHECK expressions that are not + * immutable. + */ + if (get_typtype(att->atttypid) == TYPTYPE_DOMAIN) + { + List *domain_constraints; + ListCell *lc; + + domain_constraints = GetDomainConstraints(att->atttypid); + + foreach(lc, domain_constraints) + { + DomainConstraintState *con = (DomainConstraintState *) lfirst(lc); + + if (con->check_expr && contain_mutable_functions((Node *) con->check_expr)) + { + nonimmutable = true; + break; + } + } + } + } + } + + /* Check the constraints. */ + if (!nonimmutable) + { + TupleDesc tupdesc = RelationGetDescr(entry->localrel); + + if (tupdesc->constr) + { + ConstrCheck *check = tupdesc->constr->check; + + /* + * Determine if there are any CHECK constraints which + * contains non-immutable function. + */ + for (i = 0; i < tupdesc->constr->num_check; i++) + { + Expr *check_expr = stringToNode(check[i].ccbin); + + if (contain_mutable_functions((Node *) check_expr)) + { + nonimmutable = true; + break; + } + } + } + } + + if (nonimmutable) + entry->volatility = FUNCTION_NONIMMUTABLE; + else + entry->volatility = FUNCTION_IMMUTABLE; + } + entry->localrelvalid = true; } @@ -570,6 +753,7 @@ logicalrep_partition_open(LogicalRepRelMapEntry *root, } entry->remoterel.replident = remoterel->replident; entry->remoterel.attkeys = bms_copy(remoterel->attkeys); + entry->remoterel.attunique = bms_copy(remoterel->attunique); entry->localrel = partrel; entry->localreloid = partOid; diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c index 8ffba7e2e5..3cdbf8b457 100644 --- a/src/backend/replication/logical/tablesync.c +++ b/src/backend/replication/logical/tablesync.c @@ -884,6 +884,7 @@ fetch_remote_table_info(char *nspname, char *relname, lrel->attnames = palloc0(MaxTupleAttributeNumber * sizeof(char *)); lrel->atttyps = palloc0(MaxTupleAttributeNumber * sizeof(Oid)); lrel->attkeys = NULL; + lrel->attunique = NULL; /* * Store the columns as a list of names. Ignore those that are not diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c index 32bf516ebf..29f260d1e5 100644 --- a/src/backend/replication/logical/worker.c +++ b/src/backend/replication/logical/worker.c @@ -1915,6 +1915,8 @@ apply_handle_insert(StringInfo s) /* Set relation for error callback */ apply_error_callback_arg.rel = rel; + apply_bgworker_relation_check(rel); + /* Initialize the executor state. */ edata = create_edata_for_relation(rel); estate = edata->estate; @@ -2051,6 +2053,8 @@ apply_handle_update(StringInfo s) /* Check if we can do the update. */ check_relation_updatable(rel); + apply_bgworker_relation_check(rel); + /* Initialize the executor state. */ edata = create_edata_for_relation(rel); estate = edata->estate; @@ -2219,6 +2223,8 @@ apply_handle_delete(StringInfo s) /* Check if we can do the delete. */ check_relation_updatable(rel); + apply_bgworker_relation_check(rel); + /* Initialize the executor state. */ edata = create_edata_for_relation(rel); estate = edata->estate; diff --git a/src/backend/utils/cache/typcache.c b/src/backend/utils/cache/typcache.c index 808f9ebd0d..b248899d82 100644 --- a/src/backend/utils/cache/typcache.c +++ b/src/backend/utils/cache/typcache.c @@ -2540,6 +2540,23 @@ compare_values_of_enum(TypeCacheEntry *tcache, Oid arg1, Oid arg2) return 0; } +/* + * GetDomainConstraints --- get DomainConstraintState list of specified domain type + */ +List * +GetDomainConstraints(Oid type_id) +{ + TypeCacheEntry *typentry; + List *constraints = NIL; + + typentry = lookup_type_cache(type_id, TYPECACHE_DOMAIN_CONSTR_INFO); + + if(typentry->domainData != NULL) + constraints = typentry->domainData->constraints; + + return constraints; +} + /* * Load (or re-load) the enumData member of the typcache entry. */ diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h index 7bba77c9e7..a503eb62c2 100644 --- a/src/include/replication/logicalproto.h +++ b/src/include/replication/logicalproto.h @@ -108,6 +108,7 @@ typedef struct LogicalRepRelation char replident; /* replica identity */ char relkind; /* remote relation kind */ Bitmapset *attkeys; /* Bitmap of key columns */ + Bitmapset *attunique; /* Bitmap of unique columns */ } LogicalRepRelation; /* Type mapping info */ diff --git a/src/include/replication/logicalrelation.h b/src/include/replication/logicalrelation.h index 7bf8cd22bd..9f568e9c00 100644 --- a/src/include/replication/logicalrelation.h +++ b/src/include/replication/logicalrelation.h @@ -15,6 +15,17 @@ #include "access/attmap.h" #include "replication/logicalproto.h" +/* + * States to determine volatility of the function in expressions in one + * relation. + */ +typedef enum RelFuncVolatility +{ + FUNCTION_UNKNOWN = 0, /* initializing */ + FUNCTION_IMMUTABLE, /* all functions are immutable function */ + FUNCTION_NONIMMUTABLE /* at least one non-immutable function */ +} RelFuncVolatility; + typedef struct LogicalRepRelMapEntry { LogicalRepRelation remoterel; /* key is remoterel.remoteid */ @@ -31,6 +42,10 @@ typedef struct LogicalRepRelMapEntry Relation localrel; /* relcache entry (NULL when closed) */ AttrMap *attrmap; /* map of local attributes to remote ones */ bool updatable; /* Can apply updates/deletes? */ + bool sameunique; /* Is the unique column of local and remote + consistent? */ + RelFuncVolatility volatility; /* all functions in localrel are + immutable function? */ /* Sync state. */ char state; diff --git a/src/include/replication/worker_internal.h b/src/include/replication/worker_internal.h index 5b9fea2c8c..deb80cdb19 100644 --- a/src/include/replication/worker_internal.h +++ b/src/include/replication/worker_internal.h @@ -195,6 +195,7 @@ extern void apply_bgworker_free(ApplyBgworkerState *wstate); extern void apply_bgworker_check_status(void); extern void apply_bgworker_set_status(ApplyBgworkerStatus status); extern void apply_bgworker_subxact_info_add(TransactionId current_xid); +extern void apply_bgworker_relation_check(LogicalRepRelMapEntry *rel); static inline bool am_tablesync_worker(void) diff --git a/src/include/utils/typcache.h b/src/include/utils/typcache.h index 431ad7f1b3..ed7c2e7f48 100644 --- a/src/include/utils/typcache.h +++ b/src/include/utils/typcache.h @@ -199,6 +199,8 @@ extern uint64 assign_record_type_identifier(Oid type_id, int32 typmod); extern int compare_values_of_enum(TypeCacheEntry *tcache, Oid arg1, Oid arg2); +extern List *GetDomainConstraints(Oid type_id); + extern size_t SharedRecordTypmodRegistryEstimate(void); extern void SharedRecordTypmodRegistryInit(SharedRecordTypmodRegistry *, diff --git a/src/test/subscription/t/022_twophase_cascade.pl b/src/test/subscription/t/022_twophase_cascade.pl index b52af74e35..81791c8d3a 100644 --- a/src/test/subscription/t/022_twophase_cascade.pl +++ b/src/test/subscription/t/022_twophase_cascade.pl @@ -39,6 +39,13 @@ sub test_streaming ALTER SUBSCRIPTION tap_sub_C SET (streaming = $streaming_mode)"); + if ($streaming_mode eq 'apply') + { + $node_C->safe_psql( + 'postgres', " + ALTER TABLE test_tab ALTER c DROP DEFAULT"); + } + # Wait for subscribers to finish initialization $node_A->poll_query_until( diff --git a/src/test/subscription/t/032_streaming_apply.pl b/src/test/subscription/t/032_streaming_apply.pl new file mode 100644 index 0000000000..764ea2ed59 --- /dev/null +++ b/src/test/subscription/t/032_streaming_apply.pl @@ -0,0 +1,206 @@ +# Copyright (c) 2022, PostgreSQL Global Development Group + +# Test the restrictions of streaming mode "apply" in logical replication + +use strict; +use warnings; +use PostgreSQL::Test::Cluster; +use PostgreSQL::Test::Utils; +use Test::More; + +my $offset = 0; + +# Create publisher node +my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); +$node_publisher->init(allows_streaming => 'logical'); +$node_publisher->append_conf('postgresql.conf', + 'logical_decoding_work_mem = 64kB'); +$node_publisher->start; + +# Create subscriber node +my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber'); +$node_subscriber->init; +$node_subscriber->start; + +# Setup structure on publisher +$node_publisher->safe_psql('postgres', + "CREATE TABLE test_tab (a int primary key, b varchar)"); + +# Setup structure on subscriber +$node_subscriber->safe_psql('postgres', + "CREATE TABLE test_tab (a int primary key, b varchar)"); + +# Setup logical replication +my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres'; +$node_publisher->safe_psql('postgres', + "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); + +my $appname = 'tap_sub'; +$node_subscriber->safe_psql( + 'postgres', " + CREATE SUBSCRIPTION tap_sub + CONNECTION '$publisher_connstr application_name=$appname' + PUBLICATION tap_pub + WITH (streaming = apply, copy_data = false)"); + +$node_publisher->wait_for_catchup($appname); + +# It is not allowed that the unique index on the publisher and the subscriber +# is different. Check the error reported by background worker in this case. +$node_subscriber->safe_psql('postgres', + "CREATE UNIQUE INDEX test_tab_b_idx ON test_tab (b)"); + +# Check that a background worker starts if "streaming" option is +# specified as "apply". We have to look for the DEBUG1 log messages +# about that, so temporarily bump up the log verbosity. +$node_subscriber->append_conf('postgresql.conf', "log_min_messages = debug1"); +$node_subscriber->reload; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); +$node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); +$node_subscriber->reload; + +$node_subscriber->wait_for_log( + qr/ERROR: cannot replicate relation with different unique index/, + $offset); + +# Drop the unique index on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', "DROP INDEX test_tab_b_idx"); + +$node_publisher->wait_for_catchup($appname); + +my $result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); +is($result, qq(5000), 'data replicated to subscriber after dropping index'); + +# Triggers which execute non-immutable function are not allowed on the +# subscriber side. Check the error reported by background worker in this case. +$node_subscriber->safe_psql( + 'postgres', qq{ +CREATE FUNCTION trigger_func() RETURNS TRIGGER AS \$\$ + BEGIN + RETURN NULL; + END +\$\$ language plpgsql; +CREATE TRIGGER insert_trig +BEFORE INSERT ON test_tab +FOR EACH ROW EXECUTE PROCEDURE trigger_func(); +ALTER TABLE test_tab ENABLE REPLICA TRIGGER insert_trig; +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', "DELETE FROM test_tab"); + +$node_subscriber->wait_for_log( + qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, + $offset); + +# Drop the trigger on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "DROP TRIGGER insert_trig ON test_tab"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); +is($result, qq(0), 'data replicated to subscriber after dropping trigger'); + +# It is not allowed that column default value expression contains a +# non-immutable function on the subscriber side. Check the error reported by +# background worker in this case. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab ALTER COLUMN b SET DEFAULT random()"); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log( + qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, + $offset); + +# Drop default value on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab ALTER COLUMN b DROP DEFAULT"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); +is($result, qq(5000), + 'data replicated to subscriber after dropping default value expression'); + +# It is not allowed that domain constraint expression contains a non-immutable +# function on the subscriber side. Check the error reported by background +# worker in this case. +$node_subscriber->safe_psql( + 'postgres', qq{ +CREATE DOMAIN test_domain AS int CHECK(VALUE > random()); +ALTER TABLE test_tab ALTER COLUMN a TYPE test_domain; +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', "DELETE FROM test_tab"); + +$node_subscriber->wait_for_log( + qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, + $offset); + +# Drop domain constraint expression on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab ALTER COLUMN a TYPE int"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); +is($result, qq(0), + 'data replicated to subscriber after dropping domain constraint expression' +); + +# It is not allowed that constraint expression contains a non-immutable function +# on the subscriber side. Check the error reported by background worker in this +# case. +$node_subscriber->safe_psql( + 'postgres', qq{ +ALTER TABLE test_tab ADD CONSTRAINT test_tab_con check (a > random()); +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log( + qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, + $offset); + +# Drop default value on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab DROP CONSTRAINT test_tab_con"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); +is($result, qq(5000), + 'data replicated to subscriber after dropping constraint expression'); + +$node_subscriber->stop; +$node_publisher->stop; + +done_testing(); -- 2.23.0.windows.1 [application/octet-stream] v9-0004-Add-a-GUC-max_apply_bgworkers_per_subscription-to.patch (6.9K, ../../OS3PR01MB6275208A2F8ED832710F65E09EA49@OS3PR01MB6275.jpnprd01.prod.outlook.com/5-v9-0004-Add-a-GUC-max_apply_bgworkers_per_subscription-to.patch) download | inline diff: From 2383a430c4e6f811bfe2ad0e40f5c4ba092c6bed Mon Sep 17 00:00:00 2001 From: wangw <[email protected]> Date: Wed, 8 Jun 2022 11:52:54 +0800 Subject: [PATCH v9 4/4] Add a GUC "max_apply_bgworkers_per_subscription" to control parallelism. This GUC controls how many apply background workers can be launched per subscription. --- doc/src/sgml/config.sgml | 25 +++++++++++++++++++ .../replication/logical/applybgwroker.c | 9 +++++++ src/backend/replication/logical/launcher.c | 25 +++++++++++++++++++ src/backend/utils/misc/guc.c | 12 +++++++++ src/backend/utils/misc/postgresql.conf.sample | 1 + src/include/replication/logicallauncher.h | 1 + src/include/replication/worker_internal.h | 1 + 7 files changed, 74 insertions(+) diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml index 5b7ce6531d..ce7f6827f2 100644 --- a/doc/src/sgml/config.sgml +++ b/doc/src/sgml/config.sgml @@ -4970,6 +4970,31 @@ ANY <replaceable class="parameter">num_sync</replaceable> ( <replaceable class=" </listitem> </varlistentry> + <varlistentry id="guc-max-apply-bgworkers-per-subscription" xreflabel="max_apply_bgworkers_per_subscription"> + <term><varname>max_apply_bgworkers_per_subscription</varname> (<type>integer</type>) + <indexterm> + <primary><varname>max_apply_bgworkers_per_subscription</varname> configuration parameter</primary> + </indexterm> + </term> + <listitem> + <para> + Maximum number of apply background workers per subscription. This + parameter controls the amount of parallelism of the streaming of + in-progress transactions if we set subscription option + <literal>streaming</literal> to <literal>apply</literal>. + </para> + <para> + The apply background workers are taken from the pool defined by + <varname>max_logical_replication_workers</varname>. + </para> + <para> + The default value is 3. This parameter can only be set in the + <filename>postgresql.conf</filename> file or on the server command + line. + </para> + </listitem> + </varlistentry> + </variablelist> </sect2> diff --git a/src/backend/replication/logical/applybgwroker.c b/src/backend/replication/logical/applybgwroker.c index 0f9081d62f..9d08261906 100644 --- a/src/backend/replication/logical/applybgwroker.c +++ b/src/backend/replication/logical/applybgwroker.c @@ -21,6 +21,7 @@ #include "mb/pg_wchar.h" #include "pgstat.h" #include "postmaster/interrupt.h" +#include "replication/logicallauncher.h" #include "replication/logicalworker.h" #include "replication/origin.h" #include "replication/walreceiver.h" @@ -571,9 +572,17 @@ apply_bgworker_setup(void) MemoryContext oldcontext; bool launched; ApplyBgworkerState *wstate; + int napplyworkers; elog(DEBUG1, "setting up apply worker #%u", list_length(ApplyWorkersList) + 1); + /* Check If there are free worker slot(s) */ + LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); + napplyworkers = logicalrep_apply_background_worker_count(MyLogicalRepWorker->subid); + LWLockRelease(LogicalRepWorkerLock); + if (napplyworkers >= max_apply_bgworkers_per_subscription) + return NULL; + oldcontext = MemoryContextSwitchTo(ApplyContext); wstate = (ApplyBgworkerState *) palloc0(sizeof(ApplyBgworkerState)); diff --git a/src/backend/replication/logical/launcher.c b/src/backend/replication/logical/launcher.c index 201f51232e..dd68651314 100644 --- a/src/backend/replication/logical/launcher.c +++ b/src/backend/replication/logical/launcher.c @@ -54,6 +54,7 @@ int max_logical_replication_workers = 4; int max_sync_workers_per_subscription = 2; +int max_apply_bgworkers_per_subscription = 3; LogicalRepWorker *MyLogicalRepWorker = NULL; @@ -736,6 +737,30 @@ logicalrep_sync_worker_count(Oid subid) return res; } +/* + * Count the number of registered (not necessarily running) apply background + * worker for a subscription. + */ +int +logicalrep_apply_background_worker_count(Oid subid) +{ + int i; + int res = 0; + + Assert(LWLockHeldByMe(LogicalRepWorkerLock)); + + /* Search for attached worker for a given subscription id. */ + for (i = 0; i < max_logical_replication_workers; i++) + { + LogicalRepWorker *w = &LogicalRepCtx->workers[i]; + + if (w->subid == subid && w->subworker) + res++; + } + + return res; +} + /* * ApplyLauncherShmemSize * Compute space needed for replication launcher shared memory diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c index 55d41ae7d6..539ec07ff5 100644 --- a/src/backend/utils/misc/guc.c +++ b/src/backend/utils/misc/guc.c @@ -3220,6 +3220,18 @@ static struct config_int ConfigureNamesInt[] = NULL, NULL, NULL }, + { + {"max_apply_bgworkers_per_subscription", + PGC_SIGHUP, + REPLICATION_SUBSCRIBERS, + gettext_noop("Maximum number of apply backgrand workers per subscription."), + NULL, + }, + &max_apply_bgworkers_per_subscription, + 3, 0, MAX_BACKENDS, + NULL, NULL, NULL + }, + { {"log_rotation_age", PGC_SIGHUP, LOGGING_WHERE, gettext_noop("Sets the amount of time to wait before forcing " diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample index 48ad80cf2e..b71340549d 100644 --- a/src/backend/utils/misc/postgresql.conf.sample +++ b/src/backend/utils/misc/postgresql.conf.sample @@ -360,6 +360,7 @@ #max_logical_replication_workers = 4 # taken from max_worker_processes # (change requires restart) #max_sync_workers_per_subscription = 2 # taken from max_logical_replication_workers +#max_apply_bgworkers_per_subscription = 3 # taken from max_logical_replication_workers #------------------------------------------------------------------------------ diff --git a/src/include/replication/logicallauncher.h b/src/include/replication/logicallauncher.h index f1e2821e25..ac8ef94381 100644 --- a/src/include/replication/logicallauncher.h +++ b/src/include/replication/logicallauncher.h @@ -14,6 +14,7 @@ extern PGDLLIMPORT int max_logical_replication_workers; extern PGDLLIMPORT int max_sync_workers_per_subscription; +extern PGDLLIMPORT int max_apply_bgworkers_per_subscription; extern void ApplyLauncherRegister(void); extern void ApplyLauncherMain(Datum main_arg); diff --git a/src/include/replication/worker_internal.h b/src/include/replication/worker_internal.h index deb80cdb19..dd49d8e2ec 100644 --- a/src/include/replication/worker_internal.h +++ b/src/include/replication/worker_internal.h @@ -161,6 +161,7 @@ extern void logicalrep_worker_wakeup(Oid subid, Oid relid); extern void logicalrep_worker_wakeup_ptr(LogicalRepWorker *worker); extern int logicalrep_sync_worker_count(Oid subid); +extern int logicalrep_apply_background_worker_count(Oid subid); extern void ReplicationOriginNameForTablesync(Oid suboid, Oid relid, char *originname, int szorgname); -- 2.23.0.windows.1 ^ permalink raw reply [nested|flat] 66+ messages in thread
* RE: Perform streaming logical transactions by background workers and parallel apply @ 2022-06-14 03:37 [email protected] <[email protected]> parent: [email protected] <[email protected]> 0 siblings, 2 replies; 66+ messages in thread From: [email protected] @ 2022-06-14 03:37 UTC (permalink / raw) To: Amit Kapila <[email protected]>; Masahiko Sawada <[email protected]>; +Cc: Peter Smith <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; PostgreSQL Hackers <[email protected]> On Wed, Jun 8, 2022 3:13 PM I wrote: > Attach the new patches.(only changed 0001 and 0003) I tried to improve the patches by following points: 1. Initialize variable include_abort_lsn to false. It reports a warning in cfbot. (see patch v10-0001) BTW, I merged the patch that added the new GUC (see v9-0004) into patch 0001. 2. Because of the improvement #2 in [1], the foreign key could not be detected when checking trigger function. So added additional checks for the foreign key. (see patch 0004) 3. Adding a check for the partition table when trying to apply changes in the apply background worker. (see patch 0004) In additional, the partition cache map on subscriber have several bugs (see thread [2]). Because patch 0004 is developed based on the patches in [2], so I merged the patches(v4-0001~v4-0003) in [2] into a temporary patch 0003 here. After the patches in [2] is committed, I will delete patch 0003 and rebase patch 0004. 4. Improve constraint checking in a separate patch as suggested by Amit-san in [3] #6.(see patch 0005) I added a new field "bool subretry" in catalog pg_subscription. I use this field to indicate whether the transaction that we are going to process has failed before. If apply worker/bgworker was exit with an error, this field will be set to true; If we successfully apply a transaction, this field will be set to false. If we retry to apply a streaming transaction, whether the user sets the streaming option to "on" or "apply", we will apply the transaction in the apply worker. Attach the new patches. Only changed patches 0001, 0004 and added new separate patch 0005. [1] - https://www.postgresql.org/message-id/OS3PR01MB6275208A2F8ED832710F65E09EA49%40OS3PR01MB6275.jpnprd0... [2] - https://www.postgresql.org/message-id/flat/OSZPR01MB6310F46CD425A967E4AEF736FDA49%40OSZPR01MB6310.jp... [3] - https://www.postgresql.org/message-id/CAA4eK1Jt08SYbRt_-rbSWNg%3DX9-m8%2BRdP5PosfnQgyF-z8bkxQ%40mail... Regards, Wang wei Attachments: [application/octet-stream] v10-0001-Perform-streaming-logical-transactions-by-backgr.patch (98.2K, ../../OS3PR01MB62756BD9482EB6BB1CA4CD4D9EAA9@OS3PR01MB6275.jpnprd01.prod.outlook.com/2-v10-0001-Perform-streaming-logical-transactions-by-backgr.patch) download | inline diff: From 9f43f193f4c264693b1bbea5fa2188888ff0a1be Mon Sep 17 00:00:00 2001 From: "houzj.fnst" <[email protected]> Date: Wed, 20 Apr 2022 16:45:07 +0800 Subject: [PATCH v10 1/5] Perform streaming logical transactions by background workers Currently, for large transactions, the publisher sends the data in multiple streams (changes divided into chunks depending upon logical_decoding_work_mem), and then on the subscriber-side, the apply worker writes the changes into temporary files and once it receives the commit, it reads from the file and applies the entire transaction. To improve the performance of such transactions, we can instead allow them to be applied via background workers. In this approach, we assign a new apply background worker (if available) as soon as the xact's first stream is received and the main apply worker will send changes to this new worker via shared memory. The apply background worker will directly apply the change instead of writing it to temporary files. We keep this worker assigned till the transaction commit is received and also wait for the worker to finish at commit. This preserves commit ordering and avoids writing to and reading from file in most cases. We still need to spill if there is no worker available. We also need to allow stream_stop to complete by the apply background worker to avoid deadlocks because T-1's current stream of changes can update rows in conflicting order with T-2's next stream of changes. This patch also extends the SUBSCRIPTION 'streaming' option so that the user can control whether to apply the streaming transaction in an apply background worker or spill the change to disk. User can set the streaming option to 'on/off', 'apply'. For now, 'apply' means the streaming will be applied via a apply background worker if available. 'on' means the streaming transaction will be spilled to disk. --- doc/src/sgml/catalogs.sgml | 10 +- doc/src/sgml/config.sgml | 25 + doc/src/sgml/logical-replication.sgml | 7 + doc/src/sgml/protocol.sgml | 19 + doc/src/sgml/ref/create_subscription.sgml | 24 +- src/backend/access/transam/xact.c | 13 + src/backend/commands/subscriptioncmds.c | 66 +- src/backend/postmaster/bgworker.c | 3 + src/backend/replication/logical/Makefile | 3 +- .../replication/logical/applybgwroker.c | 765 ++++++++++++++++++ src/backend/replication/logical/decode.c | 10 +- src/backend/replication/logical/launcher.c | 112 ++- src/backend/replication/logical/origin.c | 26 +- src/backend/replication/logical/proto.c | 35 +- .../replication/logical/reorderbuffer.c | 10 +- src/backend/replication/logical/tablesync.c | 10 +- src/backend/replication/logical/worker.c | 645 +++++++++++---- src/backend/replication/pgoutput/pgoutput.c | 2 +- src/backend/utils/activity/wait_event.c | 3 + src/backend/utils/misc/guc.c | 12 + src/backend/utils/misc/postgresql.conf.sample | 1 + src/bin/pg_dump/pg_dump.c | 6 +- src/include/catalog/pg_subscription.h | 17 +- src/include/replication/logicallauncher.h | 1 + src/include/replication/logicalproto.h | 19 +- src/include/replication/logicalworker.h | 1 + src/include/replication/origin.h | 2 +- src/include/replication/reorderbuffer.h | 7 +- src/include/replication/worker_internal.h | 104 ++- src/include/utils/wait_event.h | 1 + src/test/regress/expected/subscription.out | 2 +- src/tools/pgindent/typedefs.list | 5 + 32 files changed, 1751 insertions(+), 215 deletions(-) create mode 100644 src/backend/replication/logical/applybgwroker.c diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index c00c93dd7b..5cb39b18bd 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -7873,11 +7873,15 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l <row> <entry role="catalog_table_entry"><para role="column_definition"> - <structfield>substream</structfield> <type>bool</type> + <structfield>substream</structfield> <type>char</type> </para> <para> - If true, the subscription will allow streaming of in-progress - transactions + Controls how to handle the streaming of in-progress transactions: + <literal>f</literal> = disallow streaming of in-progress transactions, + <literal>t</literal> = spill the changes of in-progress transactions to + disk and apply at once after the transaction is committed on the + publisher, + <literal>a</literal> = apply changes directly using a background worker </para></entry> </row> diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml index 5b7ce6531d..ce7f6827f2 100644 --- a/doc/src/sgml/config.sgml +++ b/doc/src/sgml/config.sgml @@ -4970,6 +4970,31 @@ ANY <replaceable class="parameter">num_sync</replaceable> ( <replaceable class=" </listitem> </varlistentry> + <varlistentry id="guc-max-apply-bgworkers-per-subscription" xreflabel="max_apply_bgworkers_per_subscription"> + <term><varname>max_apply_bgworkers_per_subscription</varname> (<type>integer</type>) + <indexterm> + <primary><varname>max_apply_bgworkers_per_subscription</varname> configuration parameter</primary> + </indexterm> + </term> + <listitem> + <para> + Maximum number of apply background workers per subscription. This + parameter controls the amount of parallelism of the streaming of + in-progress transactions if we set subscription option + <literal>streaming</literal> to <literal>apply</literal>. + </para> + <para> + The apply background workers are taken from the pool defined by + <varname>max_logical_replication_workers</varname>. + </para> + <para> + The default value is 3. This parameter can only be set in the + <filename>postgresql.conf</filename> file or on the server command + line. + </para> + </listitem> + </varlistentry> + </variablelist> </sect2> diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml index 145ea71d61..cca6c97d79 100644 --- a/doc/src/sgml/logical-replication.sgml +++ b/doc/src/sgml/logical-replication.sgml @@ -948,6 +948,13 @@ CONTEXT: processing remote data for replication origin "pg_16395" during "INSER might not violate any constraint. This can easily make the subscriber inconsistent. </para> + + <para> + Setting streaming mode to <literal>apply</literal> could export invalid LSN + as finish LSN of failed transaction. Changing the streaming mode and making + the same conflict writes the finish LSN of the failed transaction in the + server log if required. + </para> </sect1> <sect1 id="logical-replication-restrictions"> diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml index bb30340892..34be1b2698 100644 --- a/doc/src/sgml/protocol.sgml +++ b/doc/src/sgml/protocol.sgml @@ -6809,6 +6809,25 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;" </listitem> </varlistentry> + <varlistentry> + <term>Int64 (XLogRecPtr)</term> + <listitem> + <para> + The LSN of the abort. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Int64 (TimestampTz)</term> + <listitem> + <para> + Abort timestamp of the transaction. The value is in number + of microseconds since PostgreSQL epoch (2000-01-01). + </para> + </listitem> + </varlistentry> + <varlistentry> <term>Int32 (TransactionId)</term> <listitem> diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml index 35b39c28da..d4caae0222 100644 --- a/doc/src/sgml/ref/create_subscription.sgml +++ b/doc/src/sgml/ref/create_subscription.sgml @@ -217,13 +217,29 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl </varlistentry> <varlistentry> - <term><literal>streaming</literal> (<type>boolean</type>)</term> + <term><literal>streaming</literal> (<type>enum</type>)</term> <listitem> <para> Specifies whether to enable streaming of in-progress transactions - for this subscription. By default, all transactions - are fully decoded on the publisher and only then sent to the - subscriber as a whole. + for this subscription. The default value is <literal>off</literal>, + all transactions are fully decoded on the publisher and only then + sent to the subscriber as a whole. + </para> + + <para> + If set to <literal>on</literal>, the changes of transaction are + written to temporary files and then applied at once after the + transaction is committed on the publisher. + </para> + + <para> + If set to <literal>apply</literal> incoming + changes are directly applied via one of the background workers, if + available. If no background worker is free to handle streaming + transaction then the changes are written to a file and applied after + the transaction is committed. Note that if an error happens when + applying changes in a background worker, it might not report the + finish LSN of the remote transaction in the server log. </para> </listitem> </varlistentry> diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c index 47d80b0d25..678540ba25 100644 --- a/src/backend/access/transam/xact.c +++ b/src/backend/access/transam/xact.c @@ -1711,6 +1711,7 @@ RecordTransactionAbort(bool isSubXact) int nchildren; TransactionId *children; TimestampTz xact_time; + bool replorigin; /* * If we haven't been assigned an XID, nobody will care whether we aborted @@ -1741,6 +1742,13 @@ RecordTransactionAbort(bool isSubXact) elog(PANIC, "cannot abort transaction %u, it was already committed", xid); + /* + * Are we using the replication origins feature? Or, in other words, + * are we replaying remote actions? + */ + replorigin = (replorigin_session_origin != InvalidRepOriginId && + replorigin_session_origin != DoNotReplicateId); + /* Fetch the data we need for the abort record */ nrels = smgrGetPendingDeletes(false, &rels); nchildren = xactGetCommittedChildren(&children); @@ -1765,6 +1773,11 @@ RecordTransactionAbort(bool isSubXact) MyXactFlags, InvalidTransactionId, NULL); + if (replorigin) + /* Move LSNs forward for this replication origin */ + replorigin_session_advance(replorigin_session_origin_lsn, + XactLastRecEnd); + /* * Report the latest async abort LSN, so that the WAL writer knows to * flush this abort. There's nothing to be gained by delaying this, since diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c index 83e6eae855..4080bba987 100644 --- a/src/backend/commands/subscriptioncmds.c +++ b/src/backend/commands/subscriptioncmds.c @@ -83,7 +83,7 @@ typedef struct SubOpts bool copy_data; bool refresh; bool binary; - bool streaming; + char streaming; bool twophase; bool disableonerr; XLogRecPtr lsn; @@ -95,6 +95,62 @@ static List *merge_publications(List *oldpublist, List *newpublist, bool addpub, static void ReportSlotConnectionError(List *rstates, Oid subid, char *slotname, char *err); +/* + * Extract the streaming mode value from a DefElem. This is like + * defGetBoolean() but also accepts the special value of "apply". + */ +static char +defGetStreamingMode(DefElem *def) +{ + /* + * If no parameter given, assume "true" is meant. + */ + if (def->arg == NULL) + return SUBSTREAM_ON; + + /* + * Allow 0, 1, "true", "false", "on", "off" or "apply". + */ + switch (nodeTag(def->arg)) + { + case T_Integer: + switch (intVal(def->arg)) + { + case 0: + return SUBSTREAM_OFF; + case 1: + return SUBSTREAM_ON; + default: + /* otherwise, error out below */ + break; + } + break; + default: + { + char *sval = defGetString(def); + + /* + * The set of strings accepted here should match up with the + * grammar's opt_boolean_or_string production. + */ + if (pg_strcasecmp(sval, "false") == 0 || + pg_strcasecmp(sval, "off") == 0) + return SUBSTREAM_OFF; + if (pg_strcasecmp(sval, "true") == 0 || + pg_strcasecmp(sval, "on") == 0) + return SUBSTREAM_ON; + if (pg_strcasecmp(sval, "apply") == 0) + return SUBSTREAM_APPLY; + } + break; + } + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s requires a Boolean value or \"apply\"", + def->defname))); + return SUBSTREAM_OFF; /* keep compiler quiet */ +} + /* * Common option parsing function for CREATE and ALTER SUBSCRIPTION commands. * @@ -132,7 +188,7 @@ parse_subscription_options(ParseState *pstate, List *stmt_options, if (IsSet(supported_opts, SUBOPT_BINARY)) opts->binary = false; if (IsSet(supported_opts, SUBOPT_STREAMING)) - opts->streaming = false; + opts->streaming = SUBSTREAM_OFF; if (IsSet(supported_opts, SUBOPT_TWOPHASE_COMMIT)) opts->twophase = false; if (IsSet(supported_opts, SUBOPT_DISABLE_ON_ERR)) @@ -233,7 +289,7 @@ parse_subscription_options(ParseState *pstate, List *stmt_options, errorConflictingDefElem(defel, pstate); opts->specified_opts |= SUBOPT_STREAMING; - opts->streaming = defGetBoolean(defel); + opts->streaming = defGetStreamingMode(defel); } else if (strcmp(defel->defname, "two_phase") == 0) { @@ -601,7 +657,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt, values[Anum_pg_subscription_subowner - 1] = ObjectIdGetDatum(owner); values[Anum_pg_subscription_subenabled - 1] = BoolGetDatum(opts.enabled); values[Anum_pg_subscription_subbinary - 1] = BoolGetDatum(opts.binary); - values[Anum_pg_subscription_substream - 1] = BoolGetDatum(opts.streaming); + values[Anum_pg_subscription_substream - 1] = CharGetDatum(opts.streaming); values[Anum_pg_subscription_subtwophasestate - 1] = CharGetDatum(opts.twophase ? LOGICALREP_TWOPHASE_STATE_PENDING : @@ -1060,7 +1116,7 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt, if (IsSet(opts.specified_opts, SUBOPT_STREAMING)) { values[Anum_pg_subscription_substream - 1] = - BoolGetDatum(opts.streaming); + CharGetDatum(opts.streaming); replaces[Anum_pg_subscription_substream - 1] = true; } diff --git a/src/backend/postmaster/bgworker.c b/src/backend/postmaster/bgworker.c index 40601aefd9..40ccb8993c 100644 --- a/src/backend/postmaster/bgworker.c +++ b/src/backend/postmaster/bgworker.c @@ -128,6 +128,9 @@ static const struct }, { "ApplyWorkerMain", ApplyWorkerMain + }, + { + "ApplyBgworkerMain", ApplyBgworkerMain } }; diff --git a/src/backend/replication/logical/Makefile b/src/backend/replication/logical/Makefile index c4e2fdeb71..a24a41969e 100644 --- a/src/backend/replication/logical/Makefile +++ b/src/backend/replication/logical/Makefile @@ -26,6 +26,7 @@ OBJS = \ reorderbuffer.o \ snapbuild.o \ tablesync.o \ - worker.o + worker.o \ + applybgwroker.o include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/replication/logical/applybgwroker.c b/src/backend/replication/logical/applybgwroker.c new file mode 100644 index 0000000000..1f52b155d7 --- /dev/null +++ b/src/backend/replication/logical/applybgwroker.c @@ -0,0 +1,765 @@ +/*------------------------------------------------------------------------- + * applybgwroker.c + * Support routines for applying xact by apply background worker + * + * Copyright (c) 2016-2022, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/replication/logical/applybgwroker.c + * + * This file contains routines that are intended to support setting up, using, + * and tearing down a ApplyBgworkerState. + * Refer to the comments in file header of logical/worker.c to see more + * informations about apply background worker. + * + *------------------------------------------------------------------------- + */ + +#include "postgres.h" + +#include "libpq/pqformat.h" +#include "mb/pg_wchar.h" +#include "pgstat.h" +#include "postmaster/interrupt.h" +#include "replication/logicallauncher.h" +#include "replication/logicalworker.h" +#include "replication/origin.h" +#include "replication/walreceiver.h" +#include "replication/worker_internal.h" +#include "storage/ipc.h" +#include "storage/procarray.h" +#include "tcop/tcopprot.h" +#include "utils/inval.h" +#include "utils/memutils.h" +#include "utils/resowner.h" +#include "utils/syscache.h" + +#define PG_LOGICAL_APPLY_SHM_MAGIC 0x79fb2447 // TODO Consider change + +/* + * DSM keys for apply background worker. Unlike other parallel execution code, + * since we don't need to worry about DSM keys conflicting with plan_node_id we + * can use small integers. + */ +#define APPLY_BGWORKER_KEY_SHARED 1 +#define APPLY_BGWORKER_KEY_MQ 2 + +/* + * entry for a hash table we use to map from xid to our apply background worker + * state. + */ +typedef struct ApplyBgworkerEntry +{ + TransactionId xid; + ApplyBgworkerState *wstate; +} ApplyBgworkerEntry; + +/* Apply background workers hash table (initialized on first use) */ +static HTAB *ApplyWorkersHash = NULL; +static List *ApplyWorkersFreeList = NIL; +static List *ApplyWorkersList = NIL; + +/* + * Fields to record the share informations between main apply worker and apply + * background worker. + */ +volatile ApplyBgworkerShared *MyParallelState = NULL; + +List *subxactlist = NIL; + +/* apply background worker setup */ +static ApplyBgworkerState *apply_bgworker_setup(void); +static void apply_bgworker_setup_dsm(ApplyBgworkerState *wstate); + +/* + * Look up worker inside ApplyWorkersHash for requested xid. + * + * If start flag is true, try to start a new worker if not found in hash table. + */ +ApplyBgworkerState * +apply_bgworker_find_or_start(TransactionId xid, bool start) +{ + bool found; + ApplyBgworkerState *wstate; + ApplyBgworkerEntry *entry = NULL; + + if (!TransactionIdIsValid(xid)) + return NULL; + + /* + * We don't start new background worker if we are not in streaming apply + * mode. + */ + if (MySubscription->stream != SUBSTREAM_APPLY) + return NULL; + + /* + * We don't start new background worker if user has set skiplsn as it's + * possible that user want to skip the streaming transaction. For + * streaming transaction, we need to spill the transaction to disk so that + * we can get the last LSN of the transaction to judge whether to skip + * before starting to apply the change. + */ + if (start && !XLogRecPtrIsInvalid(MySubscription->skiplsn)) + return NULL; + + /* + * For streaming transactions that are being applied in apply background + * worker, we cannot decide whether to apply the change for a relation + * that is not in the READY state (see should_apply_changes_for_rel) as we + * won't know remote_final_lsn by that time. So, we don't start new apply + * background worker in this case. + */ + if (start && !AllTablesyncsReady()) + return NULL; + + /* First time through, initialize apply workers hashtable */ + if (ApplyWorkersHash == NULL) + { + HASHCTL ctl; + + MemSet(&ctl, 0, sizeof(ctl)); + ctl.keysize = sizeof(TransactionId); + ctl.entrysize = sizeof(ApplyBgworkerEntry); + ctl.hcxt = ApplyContext; + + ApplyWorkersHash = hash_create("logical apply workers hash", 8, &ctl, + HASH_ELEM | HASH_BLOBS | HASH_CONTEXT); + } + + /* + * Find entry for requested transaction. + */ + entry = hash_search(ApplyWorkersHash, &xid, start ? HASH_ENTER : HASH_FIND, + &found); + if (found) + { + entry->wstate->pstate->status = APPLY_BGWORKER_BUSY; + return entry->wstate; + } + else if (!start) + return NULL; + + /* + * Now, we try to get a apply background worker. If there is at least one + * worker in the idle list, then take one. Otherwise, we try to start a + * new apply background worker. + */ + if (list_length(ApplyWorkersFreeList) > 0) + { + wstate = (ApplyBgworkerState *) llast(ApplyWorkersFreeList); + ApplyWorkersFreeList = list_delete_last(ApplyWorkersFreeList); + } + else + { + wstate = apply_bgworker_setup(); + + if (wstate == NULL) + { + /* + * If the apply background worker cannot be launched, remove entry + * in hash table. + */ + hash_search(ApplyWorkersHash, &xid, HASH_REMOVE, &found); + return NULL; + } + } + + /* Fill up the hash entry */ + wstate->pstate->status = APPLY_BGWORKER_BUSY; + wstate->pstate->server_version = walrcv_server_version(LogRepWorkerWalRcvConn); + wstate->pstate->stream_xid = xid; + entry->wstate = wstate; + entry->xid = xid; + + return wstate; +} + +/* + * Add the worker to the free list and remove the entry from hash table. + */ +void +apply_bgworker_free(ApplyBgworkerState *wstate) +{ + MemoryContext oldctx; + TransactionId xid = wstate->pstate->stream_xid; + + Assert(wstate->pstate->status == APPLY_BGWORKER_FINISHED); + + oldctx = MemoryContextSwitchTo(ApplyContext); + + hash_search(ApplyWorkersHash, &xid, HASH_REMOVE, NULL); + + elog(DEBUG1, "adding finished apply worker #%u for xid %u to the idle list", + wstate->pstate->n, wstate->pstate->stream_xid); + + ApplyWorkersFreeList = lappend(ApplyWorkersFreeList, wstate); + + MemoryContextSwitchTo(oldctx); +} + +/* Apply Background Worker main loop */ +static void +LogicalApplyBgwLoop(shm_mq_handle *mqh, volatile ApplyBgworkerShared *pst) +{ + shm_mq_result shmq_res; + PGPROC *registrant; + ErrorContextCallback errcallback; + XLogRecPtr last_received = InvalidXLogRecPtr; + + registrant = BackendPidGetProc(MyBgworkerEntry->bgw_notify_pid); + SetLatch(®istrant->procLatch); + + /* + * Push apply error context callback. Fields will be filled during + * applying a change. + */ + errcallback.callback = apply_error_callback; + errcallback.previous = error_context_stack; + error_context_stack = &errcallback; + + for (;;) + { + void *data; + Size len; + int c; + StringInfoData s; + MemoryContext oldctx; + XLogRecPtr start_lsn; + XLogRecPtr end_lsn; + TimestampTz send_time; + + CHECK_FOR_INTERRUPTS(); + + /* Ensure we are reading the data into our memory context. */ + oldctx = MemoryContextSwitchTo(ApplyMessageContext); + + shmq_res = shm_mq_receive(mqh, &len, &data, false); + + if (shmq_res != SHM_MQ_SUCCESS) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("lost connection to the main apply worker"))); + + if (len == 0) + break; + + s.cursor = 0; + s.maxlen = -1; + s.data = (char *) data; + s.len = len; + + /* + * We use first byte of message for additional communication between + * main Logical replication worker and apply bgworkers, so if it + * differs from 'w', then process it first. + */ + c = pq_getmsgbyte(&s); + switch (c) + { + /* End message of streaming chunk */ + case LOGICAL_REP_MSG_STREAM_STOP: + elog(DEBUG1, "[Apply BGW #%u] ended processing streaming chunk," + "waiting on shm_mq_receive", pst->n); + + apply_bgworker_set_status(APPLY_BGWORKER_READY); + + SetLatch(®istrant->procLatch); + + in_streamed_transaction = false; + pgstat_report_activity(STATE_IDLE, NULL); + continue; + case 'w': + break; + default: + elog(ERROR, "[Apply BGW #%u] unexpected message \"%c\"", + pst->n, c); + break; + } + + start_lsn = pq_getmsgint64(&s); + end_lsn = pq_getmsgint64(&s); + send_time = pq_getmsgint64(&s); + + if (last_received < start_lsn) + last_received = start_lsn; + + if (last_received < end_lsn) + last_received = end_lsn; + + /* + * TO IMPROVE: Do we need to display the apply background worker's + * information in pg_stat_replication ? + */ + UpdateWorkerStats(last_received, send_time, false); + + apply_dispatch(&s); + + if (ConfigReloadPending) + { + ConfigReloadPending = false; + ProcessConfigFile(PGC_SIGHUP); + } + + MemoryContextSwitchTo(oldctx); + MemoryContextReset(ApplyMessageContext); + } + + MemoryContextSwitchTo(TopMemoryContext); + MemoryContextReset(ApplyContext); + + /* Pop the error context stack */ + error_context_stack = errcallback.previous; + + elog(DEBUG1, "[Apply BGW #%u] exiting", pst->n); + + /* Signal main process that we are done. */ + SetLatch(®istrant->procLatch); +} + +/* + * Set the exit status so that the main apply worker can realize we have + * shutdown. + */ +static void +ApplyBgwShutdown(int code, Datum arg) +{ + SpinLockAcquire(&MyParallelState->mutex); + MyParallelState->status = APPLY_BGWORKER_EXIT; + SpinLockRelease(&MyParallelState->mutex); + + dsm_detach((dsm_segment *) DatumGetPointer(arg)); +} + +/* + * Apply Background Worker entry point + */ +void +ApplyBgworkerMain(Datum main_arg) +{ + volatile ApplyBgworkerShared *pst; + + dsm_handle handle; + dsm_segment *seg; + shm_toc *toc; + shm_mq *mq; + shm_mq_handle *mqh; + MemoryContext oldcontext; + RepOriginId originid; + int worker_slot = DatumGetInt32(main_arg); + char originname[NAMEDATALEN]; + + /* Attach to slot */ + logicalrep_worker_attach(worker_slot); + + MemoryContextSwitchTo(TopMemoryContext); + + /* Load the subscription into persistent memory context. */ + ApplyContext = AllocSetContextCreate(TopMemoryContext, + "ApplyContext", + ALLOCSET_DEFAULT_SIZES); + + /* + * Init the ApplyMessageContext which we clean up after each replication + * protocol message. + */ + ApplyMessageContext = AllocSetContextCreate(ApplyContext, + "ApplyMessageContext", + ALLOCSET_DEFAULT_SIZES); + + /* Setup signal handling */ + pqsignal(SIGHUP, SignalHandlerForConfigReload); + pqsignal(SIGTERM, die); + BackgroundWorkerUnblockSignals(); + + /* + * Connect to the dynamic shared memory segment. + * + * The backend that registered this worker passed us the ID of a shared + * memory segment to which we must attach for further instructions. In + * order to attach to dynamic shared memory, we need a resource owner. + * Once we've mapped the segment in our address space, attach to the table + * of contents so we can locate the various data structures we'll need to + * find within the segment. + */ + CurrentResourceOwner = ResourceOwnerCreate(NULL, "Logical apply worker"); + memcpy(&handle, MyBgworkerEntry->bgw_extra, sizeof(dsm_handle)); + seg = dsm_attach(handle); + if (seg == NULL) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("unable to map dynamic shared memory segment"))); + toc = shm_toc_attach(PG_LOGICAL_APPLY_SHM_MAGIC, dsm_segment_address(seg)); + if (toc == NULL) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("bad magic number in dynamic shared memory segment"))); + + before_shmem_exit(ApplyBgwShutdown, PointerGetDatum(seg)); + + /* Look up the parallel state. */ + pst = shm_toc_lookup(toc, APPLY_BGWORKER_KEY_SHARED, false); + MyParallelState = pst; + + /* + * Attach to the message queue. + */ + mq = shm_toc_lookup(toc, APPLY_BGWORKER_KEY_MQ, false); + shm_mq_set_receiver(mq, MyProc); + mqh = shm_mq_attach(mq, seg, NULL); + + /* Run as replica session replication role. */ + SetConfigOption("session_replication_role", "replica", + PGC_SUSET, PGC_S_OVERRIDE); + + /* Connect to our database. */ + BackgroundWorkerInitializeConnectionByOid(MyLogicalRepWorker->dbid, + MyLogicalRepWorker->userid, + 0); + + /* + * Set always-secure search path, so malicious users can't redirect user + * code (e.g. pg_index.indexprs). + */ + SetConfigOption("search_path", "", PGC_SUSET, PGC_S_OVERRIDE); + + /* + * Set the client encoding to the database encoding, since that is what + * the leader will expect. + */ + SetClientEncoding(GetDatabaseEncoding()); + + stream_xid = pst->stream_xid; + + StartTransactionCommand(); + oldcontext = MemoryContextSwitchTo(ApplyContext); + + MySubscription = GetSubscription(MyLogicalRepWorker->subid, true); + if (!MySubscription) + { + ereport(LOG, + (errmsg("logical replication apply worker for subscription %u will not " + "start because the subscription was removed during startup", + MyLogicalRepWorker->subid))); + proc_exit(0); + } + + MySubscriptionValid = true; + MemoryContextSwitchTo(oldcontext); + + /* Setup synchronous commit according to the user's wishes */ + SetConfigOption("synchronous_commit", MySubscription->synccommit, + PGC_BACKEND, PGC_S_OVERRIDE); + + /* Keep us informed about subscription changes. */ + CacheRegisterSyscacheCallback(SUBSCRIPTIONOID, + subscription_change_cb, + (Datum) 0); + + CommitTransactionCommand(); + + /* Setup replication origin tracking. */ + StartTransactionCommand(); + snprintf(originname, sizeof(originname), "pg_%u", MySubscription->oid); + originid = replorigin_by_name(originname, true); + if (!OidIsValid(originid)) + originid = replorigin_create(originname); + + /* + * The apply background worker doesn't need to monopolize this replication + * origin which was already acquired by its leader process. + */ + replorigin_session_setup(originid, false); + replorigin_session_origin = originid; + CommitTransactionCommand(); + + /* + * Allocate the origin name in long-lived context for error context + * message. + */ + apply_error_callback_arg.origin_name = MemoryContextStrdup(ApplyContext, + originname); + + /* + * Indicate that we're fully initialized and ready to begin the main part + * of the apply operation. + */ + apply_bgworker_set_status(APPLY_BGWORKER_ATTACHED); + + elog(DEBUG1, "[Apply BGW #%u] started", pst->n); + + LogicalApplyBgwLoop(mqh, pst); + + /* + * We're done. Explicitly detach the shared memory segment so that we + * don't get a resource leak warning at commit time. This will fire any + * on_dsm_detach callbacks we've registered, as well. Once that's done, + * we can go ahead and exit. + */ + dsm_detach(seg); + proc_exit(0); +} + +/* + * Set up a dynamic shared memory segment. + * + * We set up a control region that contains a ApplyBgworkerShared, + * plus one region per message queue. There are as many message queues as + * the number of workers. + */ +static void +apply_bgworker_setup_dsm(ApplyBgworkerState *wstate) +{ + shm_toc_estimator e; + Size segsize; + dsm_segment *seg; + shm_toc *toc; + ApplyBgworkerShared *pst; + shm_mq *mq; + int64 queue_size = 160000000; /* 16 MB for now */ + + /* + * Estimate how much shared memory we need. + * + * Because the TOC machinery may choose to insert padding of oddly-sized + * requests, we must estimate each chunk separately. + * + * We need one key to register the location of the header, and we need + * another key to track the location of the message queue. + */ + shm_toc_initialize_estimator(&e); + shm_toc_estimate_chunk(&e, sizeof(ApplyBgworkerShared)); + shm_toc_estimate_chunk(&e, (Size) queue_size); + + shm_toc_estimate_keys(&e, 1 + 1); + segsize = shm_toc_estimate(&e); + + /* Create the shared memory segment and establish a table of contents. */ + seg = dsm_create(shm_toc_estimate(&e), 0); + toc = shm_toc_create(PG_LOGICAL_APPLY_SHM_MAGIC, dsm_segment_address(seg), + segsize); + + /* Set up the header region. */ + pst = shm_toc_allocate(toc, sizeof(ApplyBgworkerShared)); + SpinLockInit(&pst->mutex); + pst->status = APPLY_BGWORKER_BUSY; + pst->server_version = walrcv_server_version(LogRepWorkerWalRcvConn); + pst->stream_xid = stream_xid; + pst->n = list_length(ApplyWorkersList) + 1; + + shm_toc_insert(toc, APPLY_BGWORKER_KEY_SHARED, pst); + + /* Set up one message queue per worker, plus one. */ + mq = shm_mq_create(shm_toc_allocate(toc, (Size) queue_size), + (Size) queue_size); + shm_toc_insert(toc, APPLY_BGWORKER_KEY_MQ, mq); + shm_mq_set_sender(mq, MyProc); + + /* Attach the queues. */ + wstate->mq_handle = shm_mq_attach(mq, seg, NULL); + + /* Return results to caller. */ + wstate->dsm_seg = seg; + wstate->pstate = pst; +} + +/* + * Start apply background worker process and allocate shared memory for it. + */ +static ApplyBgworkerState * +apply_bgworker_setup(void) +{ + MemoryContext oldcontext; + bool launched; + ApplyBgworkerState *wstate; + int napplyworkers; + + elog(DEBUG1, "setting up apply worker #%u", list_length(ApplyWorkersList) + 1); + + /* Check If there are free worker slot(s) */ + LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); + napplyworkers = logicalrep_apply_background_worker_count(MyLogicalRepWorker->subid); + LWLockRelease(LogicalRepWorkerLock); + if (napplyworkers >= max_apply_bgworkers_per_subscription) + return NULL; + + oldcontext = MemoryContextSwitchTo(ApplyContext); + + wstate = (ApplyBgworkerState *) palloc0(sizeof(ApplyBgworkerState)); + + /* Setup shared memory */ + apply_bgworker_setup_dsm(wstate); + + launched = logicalrep_worker_launch(MyLogicalRepWorker->dbid, + MySubscription->oid, + MySubscription->name, + MyLogicalRepWorker->userid, + InvalidOid, + dsm_segment_handle(wstate->dsm_seg)); + + if (launched) + { + /* Wait for worker to attach. */ + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_ATTACHED); + + ApplyWorkersList = lappend(ApplyWorkersList, wstate); + } + else + { + pfree(wstate); + wstate = NULL; + } + + MemoryContextSwitchTo(oldcontext); + + return wstate; +} + +/* + * Send the data to the specified apply background worker via shared-memory queue. + */ +void +apply_bgworker_send_data(ApplyBgworkerState *wstate, Size nbytes, const void *data) +{ + shm_mq_result result; + + result = shm_mq_send(wstate->mq_handle, nbytes, data, false, true); + + if (result != SHM_MQ_SUCCESS) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("could not send tuples to shared-memory queue"))); +} + +/* + * Wait until the status of apply background worker reaches the + * 'wait_for_status' + */ +void +apply_bgworker_wait_for(ApplyBgworkerState *wstate, + ApplyBgworkerStatus wait_for_status) +{ + for (;;) + { + char status; + + SpinLockAcquire(&wstate->pstate->mutex); + status = wstate->pstate->status; + SpinLockRelease(&wstate->pstate->mutex); + + /* Done if already in correct status. */ + if (status == wait_for_status) + break; + + /* If any workers (or the postmaster) have died, we have failed. */ + if (status == APPLY_BGWORKER_EXIT) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("background worker %u failed to apply transaction %u", + wstate->pstate->n, wstate->pstate->stream_xid))); + + /* Wait to be signalled. */ + WaitLatch(MyLatch, WL_LATCH_SET | WL_EXIT_ON_PM_DEATH, 0, + WAIT_EVENT_LOGICAL_APPLY_WORKER_STATE_CHANGE); + + /* Reset the latch so we don't spin. */ + ResetLatch(MyLatch); + + /* An interrupt may have occurred while we were waiting. */ + CHECK_FOR_INTERRUPTS(); + } +} + +/* + * Check the status of workers and report an error if any apply background + * worker has exited unexpectedly. + * + * Exit if any relation is not in the READY state and if any worker is handling + * the streaming transaction at the same time. Because for streaming + * transactions that is being applied in apply background worker, we cannot + * decide whether to apply the change for a relation that is not in the READY + * state (see should_apply_changes_for_rel) as we won't know remote_final_lsn + * by that time. + */ +void +apply_bgworker_check_status(void) +{ + ListCell *lc; + + if (am_apply_bgworker() || MySubscription->stream != SUBSTREAM_APPLY) + return; + + foreach(lc, ApplyWorkersList) + { + ApplyBgworkerState *wstate = (ApplyBgworkerState *) lfirst(lc); + + /* + * We don't lock here as in the worst case we will just detect the + * failure of worker a bit later. + */ + if (wstate->pstate->status == APPLY_BGWORKER_EXIT) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("background worker %u exited unexpectedly", + wstate->pstate->n))); + } + + if (list_length(ApplyWorkersFreeList) != list_length(ApplyWorkersList) && + !AllTablesyncsReady()) + { + ereport(LOG, + (errmsg("logical replication apply workers for subscription \"%s\" will restart", + MySubscription->name), + errdetail("Cannot handle streamed replication transaction by apply " + "bgworkers until all tables are synchronized"))); + + proc_exit(0); + } +} + +/* Set the apply background worker status */ +void +apply_bgworker_set_status(ApplyBgworkerStatus status) +{ + if (!am_apply_bgworker()) + return; + + elog(DEBUG1, "[Apply BGW #%u] set status to %d", MyParallelState->n, status); + + SpinLockAcquire(&MyParallelState->mutex); + MyParallelState->status = status; + SpinLockRelease(&MyParallelState->mutex); +} + +/* + * Define a savepoint for a subxact in apply background worker if needed. + * + * Inside apply background worker we can figure out that new subtransaction was + * started if new change arrived with different xid. In that case we can define + * named savepoint, so that we were able to commit/rollback it separately + * later. + * Special case is if the first change comes from subtransaction, then + * we check that current_xid differs from stream_xid. + */ +void +apply_bgworker_subxact_info_add(TransactionId current_xid) +{ + if (current_xid != stream_xid && + !list_member_int(subxactlist, (int) current_xid)) + { + MemoryContext oldctx; + char spname[MAXPGPATH]; + + snprintf(spname, MAXPGPATH, "savepoint_for_xid_%u", current_xid); + + elog(DEBUG1, "[Apply BGW #%u] defining savepoint %s", + MyParallelState->n, spname); + + DefineSavepoint(spname); + CommitTransactionCommand(); + + oldctx = MemoryContextSwitchTo(ApplyContext); + subxactlist = lappend_int(subxactlist, (int) current_xid); + MemoryContextSwitchTo(oldctx); + } +} diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c index aa2427ba73..00fdde0830 100644 --- a/src/backend/replication/logical/decode.c +++ b/src/backend/replication/logical/decode.c @@ -651,9 +651,10 @@ DecodeCommit(LogicalDecodingContext *ctx, XLogRecordBuffer *buf, { for (i = 0; i < parsed->nsubxacts; i++) { - ReorderBufferForget(ctx->reorder, parsed->subxacts[i], buf->origptr); + ReorderBufferForget(ctx->reorder, parsed->subxacts[i], buf->origptr, + commit_time); } - ReorderBufferForget(ctx->reorder, xid, buf->origptr); + ReorderBufferForget(ctx->reorder, xid, buf->origptr, commit_time); return; } @@ -821,10 +822,11 @@ DecodeAbort(LogicalDecodingContext *ctx, XLogRecordBuffer *buf, for (i = 0; i < parsed->nsubxacts; i++) { ReorderBufferAbort(ctx->reorder, parsed->subxacts[i], - buf->record->EndRecPtr); + buf->record->EndRecPtr, abort_time); } - ReorderBufferAbort(ctx->reorder, xid, buf->record->EndRecPtr); + ReorderBufferAbort(ctx->reorder, xid, buf->record->EndRecPtr, + abort_time); } /* update the decoding stats */ diff --git a/src/backend/replication/logical/launcher.c b/src/backend/replication/logical/launcher.c index 2bdab53e19..dd68651314 100644 --- a/src/backend/replication/logical/launcher.c +++ b/src/backend/replication/logical/launcher.c @@ -54,6 +54,7 @@ int max_logical_replication_workers = 4; int max_sync_workers_per_subscription = 2; +int max_apply_bgworkers_per_subscription = 3; LogicalRepWorker *MyLogicalRepWorker = NULL; @@ -73,6 +74,7 @@ static void logicalrep_launcher_onexit(int code, Datum arg); static void logicalrep_worker_onexit(int code, Datum arg); static void logicalrep_worker_detach(void); static void logicalrep_worker_cleanup(LogicalRepWorker *worker); +static void logicalrep_worker_stop_internal(LogicalRepWorker *worker); static bool on_commit_launcher_wakeup = false; @@ -223,6 +225,10 @@ logicalrep_worker_find(Oid subid, Oid relid, bool only_running) { LogicalRepWorker *w = &LogicalRepCtx->workers[i]; + /* We only need main apply worker or table sync worker here */ + if (w->subworker) + continue; + if (w->in_use && w->subid == subid && w->relid == relid && (!only_running || w->proc)) { @@ -259,11 +265,11 @@ logicalrep_workers_find(Oid subid, bool only_running) } /* - * Start new apply background worker, if possible. + * Start new background worker, if possible. */ -void +bool logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, Oid userid, - Oid relid) + Oid relid, dsm_handle subworker_dsm) { BackgroundWorker bgw; BackgroundWorkerHandle *bgw_handle; @@ -273,6 +279,10 @@ logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, Oid userid, LogicalRepWorker *worker = NULL; int nsyncworkers; TimestampTz now; + bool is_subworker = (subworker_dsm != DSM_HANDLE_INVALID); + + /* We don't support table sync in subworker */ + Assert(!(is_subworker && OidIsValid(relid))); ereport(DEBUG1, (errmsg_internal("starting logical replication worker for subscription \"%s\"", @@ -350,7 +360,7 @@ retry: if (OidIsValid(relid) && nsyncworkers >= max_sync_workers_per_subscription) { LWLockRelease(LogicalRepWorkerLock); - return; + return false; } /* @@ -364,7 +374,7 @@ retry: (errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED), errmsg("out of logical replication worker slots"), errhint("You might need to increase max_logical_replication_workers."))); - return; + return false; } /* Prepare the worker slot. */ @@ -379,6 +389,7 @@ retry: worker->relstate = SUBREL_STATE_UNKNOWN; worker->relstate_lsn = InvalidXLogRecPtr; worker->stream_fileset = NULL; + worker->subworker = is_subworker; worker->last_lsn = InvalidXLogRecPtr; TIMESTAMP_NOBEGIN(worker->last_send_time); TIMESTAMP_NOBEGIN(worker->last_recv_time); @@ -396,19 +407,30 @@ retry: BGWORKER_BACKEND_DATABASE_CONNECTION; bgw.bgw_start_time = BgWorkerStart_RecoveryFinished; snprintf(bgw.bgw_library_name, BGW_MAXLEN, "postgres"); - snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyWorkerMain"); + + if (!is_subworker) + snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyWorkerMain"); + else + snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyBgworkerMain"); + if (OidIsValid(relid)) snprintf(bgw.bgw_name, BGW_MAXLEN, "logical replication worker for subscription %u sync %u", subid, relid); - else + else if (!is_subworker) snprintf(bgw.bgw_name, BGW_MAXLEN, "logical replication worker for subscription %u", subid); + else + snprintf(bgw.bgw_name, BGW_MAXLEN, + "logical replication background apply worker for subscription %u ", subid); snprintf(bgw.bgw_type, BGW_MAXLEN, "logical replication worker"); bgw.bgw_restart_time = BGW_NEVER_RESTART; bgw.bgw_notify_pid = MyProcPid; bgw.bgw_main_arg = Int32GetDatum(slot); + if (is_subworker) + memcpy(bgw.bgw_extra, &subworker_dsm, sizeof(dsm_handle)); + if (!RegisterDynamicBackgroundWorker(&bgw, &bgw_handle)) { /* Failed to start worker, so clean up the worker slot. */ @@ -421,11 +443,13 @@ retry: (errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED), errmsg("out of background worker slots"), errhint("You might need to increase max_worker_processes."))); - return; + return false; } /* Now wait until it attaches. */ WaitForReplicationWorkerAttach(worker, generation, bgw_handle); + + return true; } /* @@ -436,7 +460,6 @@ void logicalrep_worker_stop(Oid subid, Oid relid) { LogicalRepWorker *worker; - uint16 generation; LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); @@ -449,6 +472,22 @@ logicalrep_worker_stop(Oid subid, Oid relid) return; } + logicalrep_worker_stop_internal(worker); + + LWLockRelease(LogicalRepWorkerLock); +} + +/* + * Workhorse for logicalrep_worker_stop() and logicalrep_worker_detach(). Stop + * the worker and wait for it to die. + */ +static void +logicalrep_worker_stop_internal(LogicalRepWorker *worker) +{ + uint16 generation; + + Assert(LWLockHeldByMe(LogicalRepWorkerLock)); + /* * Remember which generation was our worker so we can check if what we see * is still the same one. @@ -485,10 +524,7 @@ logicalrep_worker_stop(Oid subid, Oid relid) * different, meaning that a different worker has taken the slot. */ if (!worker->in_use || worker->generation != generation) - { - LWLockRelease(LogicalRepWorkerLock); return; - } /* Worker has assigned proc, so it has started. */ if (worker->proc) @@ -522,8 +558,6 @@ logicalrep_worker_stop(Oid subid, Oid relid) LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); } - - LWLockRelease(LogicalRepWorkerLock); } /* @@ -599,6 +633,29 @@ logicalrep_worker_attach(int slot) static void logicalrep_worker_detach(void) { + /* + * If we are the main apply worker, stop all the apply background workers + * we started before. + */ + if (!MyLogicalRepWorker->subworker) + { + List *workers; + ListCell *lc; + + LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); + + workers = logicalrep_workers_find(MyLogicalRepWorker->subid, true); + foreach(lc, workers) + { + LogicalRepWorker *w = (LogicalRepWorker *) lfirst(lc); + + if (w->subworker) + logicalrep_worker_stop_internal(w); + } + + LWLockRelease(LogicalRepWorkerLock); + } + /* Block concurrent access. */ LWLockAcquire(LogicalRepWorkerLock, LW_EXCLUSIVE); @@ -621,6 +678,7 @@ logicalrep_worker_cleanup(LogicalRepWorker *worker) worker->userid = InvalidOid; worker->subid = InvalidOid; worker->relid = InvalidOid; + worker->subworker = false; } /* @@ -679,6 +737,30 @@ logicalrep_sync_worker_count(Oid subid) return res; } +/* + * Count the number of registered (not necessarily running) apply background + * worker for a subscription. + */ +int +logicalrep_apply_background_worker_count(Oid subid) +{ + int i; + int res = 0; + + Assert(LWLockHeldByMe(LogicalRepWorkerLock)); + + /* Search for attached worker for a given subscription id. */ + for (i = 0; i < max_logical_replication_workers; i++) + { + LogicalRepWorker *w = &LogicalRepCtx->workers[i]; + + if (w->subid == subid && w->subworker) + res++; + } + + return res; +} + /* * ApplyLauncherShmemSize * Compute space needed for replication launcher shared memory @@ -868,7 +950,7 @@ ApplyLauncherMain(Datum main_arg) wait_time = wal_retrieve_retry_interval; logicalrep_worker_launch(sub->dbid, sub->oid, sub->name, - sub->owner, InvalidOid); + sub->owner, InvalidOid, DSM_HANDLE_INVALID); } } diff --git a/src/backend/replication/logical/origin.c b/src/backend/replication/logical/origin.c index 21937ab2d3..9273011b0a 100644 --- a/src/backend/replication/logical/origin.c +++ b/src/backend/replication/logical/origin.c @@ -1063,12 +1063,21 @@ ReplicationOriginExitCleanup(int code, Datum arg) * array doesn't have to be searched when calling * replorigin_session_advance(). * - * Obviously only one such cached origin can exist per process and the current + * Normally only one such cached origin can exist per process and the current * cached value can only be set again after the previous value is torn down * with replorigin_session_reset(). + * + * However, If must_acquire is false, we allow process to get the slot which is + * already acquired by other process. It's safe because 1) The only caller + * (apply background workers) will maintain the commit order by allowing only + * one process to commit at a time, so no two workers will be operating on the + * same origin at the same time (see comments in logical/worker.c). 2) Even + * though we try to advance the session's origin concurrently, it's safe to do + * so as we change/advance the session_origin LSNs under replicate_state + * LWLock. */ void -replorigin_session_setup(RepOriginId node) +replorigin_session_setup(RepOriginId node, bool must_acquire) { static bool registered_cleanup; int i; @@ -1110,7 +1119,7 @@ replorigin_session_setup(RepOriginId node) if (curstate->roident != node) continue; - else if (curstate->acquired_by != 0) + else if (curstate->acquired_by != 0 && must_acquire) { ereport(ERROR, (errcode(ERRCODE_OBJECT_IN_USE), @@ -1141,7 +1150,14 @@ replorigin_session_setup(RepOriginId node) Assert(session_replication_state->roident != InvalidRepOriginId); - session_replication_state->acquired_by = MyProcPid; + if (must_acquire) + session_replication_state->acquired_by = MyProcPid; + else if (session_replication_state->acquired_by == 0) + ereport(ERROR, + (errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED), + errmsg("could not find correct replication state slot for replication origin with OID %u for apply background worker", + node), + errhint("There is no replication state slot set by its main apply worker."))); LWLockRelease(ReplicationOriginLock); @@ -1321,7 +1337,7 @@ pg_replication_origin_session_setup(PG_FUNCTION_ARGS) name = text_to_cstring((text *) DatumGetPointer(PG_GETARG_DATUM(0))); origin = replorigin_by_name(name, false); - replorigin_session_setup(origin); + replorigin_session_setup(origin, true); replorigin_session_origin = origin; diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c index ff8513e2d2..a888038eb4 100644 --- a/src/backend/replication/logical/proto.c +++ b/src/backend/replication/logical/proto.c @@ -1166,28 +1166,47 @@ logicalrep_read_stream_commit(StringInfo in, LogicalRepCommitData *commit_data) */ void logicalrep_write_stream_abort(StringInfo out, TransactionId xid, - TransactionId subxid) + ReorderBufferTXN *txn, XLogRecPtr abort_lsn) { pq_sendbyte(out, LOGICAL_REP_MSG_STREAM_ABORT); - Assert(TransactionIdIsValid(xid) && TransactionIdIsValid(subxid)); + Assert(TransactionIdIsValid(xid) && TransactionIdIsValid(txn->xid)); /* transaction ID */ pq_sendint32(out, xid); - pq_sendint32(out, subxid); + pq_sendint32(out, txn->xid); + pq_sendint64(out, abort_lsn); + pq_sendint64(out, txn->xact_time.abort_time); } /* * Read STREAM ABORT from the output stream. */ void -logicalrep_read_stream_abort(StringInfo in, TransactionId *xid, - TransactionId *subxid) +logicalrep_read_stream_abort(StringInfo in, + LogicalRepStreamAbortData *abort_data, + bool include_abort_lsn) { - Assert(xid && subxid); + Assert(abort_data); + + abort_data->xid = pq_getmsgint(in, 4); + abort_data->subxid = pq_getmsgint(in, 4); - *xid = pq_getmsgint(in, 4); - *subxid = pq_getmsgint(in, 4); + /* + * If the version of the publisher is lower than the version of the + * subscriber, it may not support sending these two fields, so only take + * these fields when include_abort_lsn is true. + */ + if (include_abort_lsn) + { + abort_data->abort_lsn = pq_getmsgint64(in); + abort_data->abort_time = pq_getmsgint64(in); + } + else + { + abort_data->abort_lsn = InvalidXLogRecPtr; + abort_data->abort_time = 0; + } } /* diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c index 8da5f9089c..3d2bbdb38d 100644 --- a/src/backend/replication/logical/reorderbuffer.c +++ b/src/backend/replication/logical/reorderbuffer.c @@ -2826,7 +2826,8 @@ ReorderBufferFinishPrepared(ReorderBuffer *rb, TransactionId xid, * disk. */ void -ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn) +ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn, + TimestampTz abort_time) { ReorderBufferTXN *txn; @@ -2837,6 +2838,8 @@ ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn) if (txn == NULL) return; + txn->xact_time.abort_time = abort_time; + /* For streamed transactions notify the remote node about the abort. */ if (rbtxn_is_streamed(txn)) { @@ -2911,7 +2914,8 @@ ReorderBufferAbortOld(ReorderBuffer *rb, TransactionId oldestRunningXid) * to this xid might re-create the transaction incompletely. */ void -ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn) +ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn, + TimestampTz abort_time) { ReorderBufferTXN *txn; @@ -2922,6 +2926,8 @@ ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn) if (txn == NULL) return; + txn->xact_time.abort_time = abort_time; + /* For streamed transactions notify the remote node about the abort. */ if (rbtxn_is_streamed(txn)) rb->stream_abort(rb, txn, lsn); diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c index 670c6fcada..8ffba7e2e5 100644 --- a/src/backend/replication/logical/tablesync.c +++ b/src/backend/replication/logical/tablesync.c @@ -568,7 +568,8 @@ process_syncing_tables_for_apply(XLogRecPtr current_lsn) MySubscription->oid, MySubscription->name, MyLogicalRepWorker->userid, - rstate->relid); + rstate->relid, + DSM_HANDLE_INVALID); hentry->last_start_time = now; } } @@ -589,6 +590,9 @@ process_syncing_tables_for_apply(XLogRecPtr current_lsn) void process_syncing_tables(XLogRecPtr current_lsn) { + if (MyLogicalRepWorker->subworker) + return; + if (am_tablesync_worker()) process_syncing_tables_for_sync(current_lsn); else @@ -1273,7 +1277,7 @@ LogicalRepSyncTableStart(XLogRecPtr *origin_startpos) * time this tablesync was launched. */ originid = replorigin_by_name(originname, false); - replorigin_session_setup(originid); + replorigin_session_setup(originid, true); replorigin_session_origin = originid; *origin_startpos = replorigin_session_get_progress(false); @@ -1384,7 +1388,7 @@ LogicalRepSyncTableStart(XLogRecPtr *origin_startpos) true /* go backward */ , true /* WAL log */ ); UnlockRelationOid(ReplicationOriginRelationId, RowExclusiveLock); - replorigin_session_setup(originid); + replorigin_session_setup(originid, true); replorigin_session_origin = originid; } else diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c index fc210a9e7b..3f8dea0fc1 100644 --- a/src/backend/replication/logical/worker.c +++ b/src/backend/replication/logical/worker.c @@ -22,8 +22,25 @@ * STREAMED TRANSACTIONS * --------------------- * Streamed transactions (large transactions exceeding a memory limit on the - * upstream) are not applied immediately, but instead, the data is written - * to temporary files and then applied at once when the final commit arrives. + * upstream) are applied via one of two approaches. + * + * 1) Separate background workers + * + * Assign a new apply background worker (if available) as soon as the xact's + * first stream is received and the main apply worker will send changes to this + * new worker via shared memory. We keep this worker assigned till the + * transaction commit is received and also wait for the worker to finish at + * commit. This preserves commit ordering and avoids writing to and reading + * from file in most cases. We still need to spill if there is no worker + * available. We also need to allow stream_stop to complete by the background + * worker to avoid deadlocks because T-1's current stream of changes can update + * rows in conflicting order with T-2's next stream of changes. + * + * 2) Write to temporary files and apply when the final commit arrives + * + * If no worker is available to handle streamed transaction, the data is + * written to temporary files and then applied at once when the final commit + * arrives. * * Unlike the regular (non-streamed) case, handling streamed transactions has * to handle aborts of both the toplevel transaction and subtransactions. This @@ -219,20 +236,7 @@ typedef struct ApplyExecutionData PartitionTupleRouting *proute; /* partition routing info */ } ApplyExecutionData; -/* Struct for saving and restoring apply errcontext information */ -typedef struct ApplyErrorCallbackArg -{ - LogicalRepMsgType command; /* 0 if invalid */ - LogicalRepRelMapEntry *rel; - - /* Remote node information */ - int remote_attnum; /* -1 if invalid */ - TransactionId remote_xid; - XLogRecPtr finish_lsn; - char *origin_name; -} ApplyErrorCallbackArg; - -static ApplyErrorCallbackArg apply_error_callback_arg = +ApplyErrorCallbackArg apply_error_callback_arg = { .command = 0, .rel = NULL, @@ -242,7 +246,7 @@ static ApplyErrorCallbackArg apply_error_callback_arg = .origin_name = NULL, }; -static MemoryContext ApplyMessageContext = NULL; +MemoryContext ApplyMessageContext = NULL; MemoryContext ApplyContext = NULL; /* per stream context for streaming transactions */ @@ -251,27 +255,38 @@ static MemoryContext LogicalStreamingContext = NULL; WalReceiverConn *LogRepWorkerWalRcvConn = NULL; Subscription *MySubscription = NULL; -static bool MySubscriptionValid = false; +bool MySubscriptionValid = false; bool in_remote_transaction = false; static XLogRecPtr remote_final_lsn = InvalidXLogRecPtr; /* fields valid only when processing streamed transaction */ -static bool in_streamed_transaction = false; +bool in_streamed_transaction = false; + +TransactionId stream_xid = InvalidTransactionId; +static ApplyBgworkerState *stream_apply_worker = NULL; + +/* check if we are applying the transaction in apply background worker */ +#define apply_bgworker_active() (in_streamed_transaction && stream_apply_worker != NULL) -static TransactionId stream_xid = InvalidTransactionId; +/* + * The number of changes during one streaming block (only for apply background + * workers) + */ +static uint32 nchanges = 0; /* * We enable skipping all data modification changes (INSERT, UPDATE, etc.) for * the subscription if the remote transaction's finish LSN matches the subskiplsn. * Once we start skipping changes, we don't stop it until we skip all changes of * the transaction even if pg_subscription is updated and MySubscription->skiplsn - * gets changed or reset during that. Also, in streaming transaction cases, we - * don't skip receiving and spooling the changes since we decide whether or not + * gets changed or reset during that. Also, in streaming transaction cases (streaming = on), + * we don't skip receiving and spooling the changes since we decide whether or not * to skip applying the changes when starting to apply changes. The subskiplsn is * cleared after successfully skipping the transaction or applying non-empty * transaction. The latter prevents the mistakenly specified subskiplsn from - * being left. + * being left. Note that we cannot skip the streaming transaction in apply mode, + * because we cannot get the finish LSN before applying the changes. */ static XLogRecPtr skip_xact_finish_lsn = InvalidXLogRecPtr; #define is_skipping_changes() (unlikely(!XLogRecPtrIsInvalid(skip_xact_finish_lsn))) @@ -324,9 +339,6 @@ static void maybe_reread_subscription(void); static void DisableSubscriptionAndExit(void); -/* prototype needed because of stream_commit */ -static void apply_dispatch(StringInfo s); - static void apply_handle_commit_internal(LogicalRepCommitData *commit_data); static void apply_handle_insert_internal(ApplyExecutionData *edata, ResultRelInfo *relinfo, @@ -359,7 +371,6 @@ static void stop_skipping_changes(void); static void clear_subscription_skip_lsn(XLogRecPtr finish_lsn); /* Functions for apply error callback */ -static void apply_error_callback(void *arg); static inline void set_apply_error_context_xact(TransactionId xid, XLogRecPtr lsn); static inline void reset_apply_error_context_info(void); @@ -426,41 +437,76 @@ end_replication_step(void) } /* - * Handle streamed transactions. + * Handle streamed transactions for both main apply worker and apply background + * worker. * - * If in streaming mode (receiving a block of streamed transaction), we - * simply redirect it to a file for the proper toplevel transaction. + * In streaming case (receiving a block of streamed transaction), for + * SUBSTREAM_ON mode, we simply redirect it to a file for the proper toplevel + * transaction, and for SUBSTREAM_APPLY mode, we send the changes to background + * apply worker (LOGICAL_REP_MSG_RELATION or LOGICAL_REP_MSG_TYPE changes will + * also be applied in main apply worker). * - * Returns true for streamed transactions, false otherwise (regular mode). + * For non-streamed transactions, returns false; + * For streamed transactions, returns true if in main apply worker (except we + * apply streamed transaction in "apply" mode and address + * LOGICAL_REP_MSG_RELATION or LOGICAL_REP_MSG_TYPE changes), false otherwise. */ static bool handle_streamed_transaction(LogicalRepMsgType action, StringInfo s) { - TransactionId xid; + TransactionId current_xid = InvalidTransactionId; /* not in streaming mode */ - if (!in_streamed_transaction) + if (!(in_streamed_transaction || am_apply_bgworker())) return false; - Assert(stream_fd != NULL); Assert(TransactionIdIsValid(stream_xid)); /* * We should have received XID of the subxact as the first part of the * message, so extract it. */ - xid = pq_getmsgint(s, 4); + current_xid = pq_getmsgint(s, 4); - if (!TransactionIdIsValid(xid)) + if (!TransactionIdIsValid(current_xid)) ereport(ERROR, (errcode(ERRCODE_PROTOCOL_VIOLATION), errmsg_internal("invalid transaction ID in streamed replication transaction"))); - /* Add the new subxact to the array (unless already there). */ - subxact_info_add(xid); + if (am_apply_bgworker()) + { + /* define a savepoint for a subxact if needed. */ + apply_bgworker_subxact_info_add(current_xid); - /* write the change to the current file */ - stream_write_change(action, s); + return false; + } + else if (apply_bgworker_active()) + { + /* + * If we decided to apply the changes of this transaction in an apply + * background worker, pass the data to the worker. + */ + apply_bgworker_send_data(stream_apply_worker, s->len, s->data); + nchanges += 1; + + /* + * XXX The publisher side doesn't always send relation/type update + * messages after the streaming transaction, so also update the + * relation/type in main apply worker here. See function + * cleanup_rel_sync_cache. + */ + if (action == LOGICAL_REP_MSG_RELATION || + action == LOGICAL_REP_MSG_TYPE) + return false; + } + else + { + /* Add the new subxact to the array (unless already there). */ + subxact_info_add(current_xid); + + /* write the change to the current file */ + stream_write_change(action, s); + } return true; } @@ -844,6 +890,9 @@ apply_handle_commit(StringInfo s) apply_handle_commit_internal(&commit_data); + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); + /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(commit_data.end_lsn); @@ -898,7 +947,9 @@ apply_handle_prepare_internal(LogicalRepPreparedTxnData *prepare_data) * BeginTransactionBlock is necessary to balance the EndTransactionBlock * called within the PrepareTransactionBlock below. */ - BeginTransactionBlock(); + if (!IsTransactionBlock()) + BeginTransactionBlock(); + CommitTransactionCommand(); /* Completes the preceding Begin command. */ /* @@ -950,6 +1001,9 @@ apply_handle_prepare(StringInfo s) in_remote_transaction = false; + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); + /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(prepare_data.end_lsn); @@ -1064,10 +1118,6 @@ apply_handle_rollback_prepared(StringInfo s) /* * Handle STREAM PREPARE. - * - * Logic is in two parts: - * 1. Replay all the spooled operations - * 2. Mark the transaction as prepared */ static void apply_handle_stream_prepare(StringInfo s) @@ -1088,24 +1138,70 @@ apply_handle_stream_prepare(StringInfo s) logicalrep_read_stream_prepare(s, &prepare_data); set_apply_error_context_xact(prepare_data.xid, prepare_data.prepare_lsn); - elog(DEBUG1, "received prepare for streamed transaction %u", prepare_data.xid); + if (am_apply_bgworker()) + { + /* Mark the transaction as prepared. */ + apply_handle_prepare_internal(&prepare_data); - /* Replay all the spooled operations. */ - apply_spooled_messages(prepare_data.xid, prepare_data.prepare_lsn); + CommitTransactionCommand(); - /* Mark the transaction as prepared. */ - apply_handle_prepare_internal(&prepare_data); + pgstat_report_stat(false); - CommitTransactionCommand(); + list_free(subxactlist); + subxactlist = NIL; - pgstat_report_stat(false); + apply_bgworker_set_status(APPLY_BGWORKER_FINISHED); + } + else + { + ApplyBgworkerState *wstate = apply_bgworker_find_or_start(prepare_data.xid, false); - store_flush_position(prepare_data.end_lsn); + elog(DEBUG1, "received prepare for streamed transaction %u", + prepare_data.xid); + + /* + * This is the main apply worker. Check if we are processing this + * transaction in a apply background worker. + */ + if (wstate) + { + apply_bgworker_send_data(wstate, s->len, s->data); + + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_FINISHED); + apply_bgworker_free(wstate); + + pgstat_report_stat(false); + store_flush_position(prepare_data.end_lsn); + } + else + { + /* + * This is the main apply worker and the transaction has been + * serialized to file, replay all the spooled operations. + */ + apply_spooled_messages(prepare_data.xid, prepare_data.prepare_lsn); + + /* Mark the transaction as prepared. */ + apply_handle_prepare_internal(&prepare_data); + + CommitTransactionCommand(); + + pgstat_report_stat(false); + + store_flush_position(prepare_data.end_lsn); + + in_remote_transaction = false; + + /* unlink the files with serialized changes and subxact info. */ + stream_cleanup_files(MyLogicalRepWorker->subid, prepare_data.xid); + } + } in_remote_transaction = false; + stream_apply_worker = NULL; - /* unlink the files with serialized changes and subxact info. */ - stream_cleanup_files(MyLogicalRepWorker->subid, prepare_data.xid); + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(prepare_data.end_lsn); @@ -1155,15 +1251,6 @@ apply_handle_stream_start(StringInfo s) (errcode(ERRCODE_PROTOCOL_VIOLATION), errmsg_internal("duplicate STREAM START message"))); - /* - * Start a transaction on stream start, this transaction will be committed - * on the stream stop unless it is a tablesync worker in which case it - * will be committed after processing all the messages. We need the - * transaction for handling the buffile, used for serializing the - * streaming data and subxact info. - */ - begin_replication_step(); - /* notify handle methods we're processing a remote transaction */ in_streamed_transaction = true; @@ -1177,36 +1264,92 @@ apply_handle_stream_start(StringInfo s) set_apply_error_context_xact(stream_xid, InvalidXLogRecPtr); + if (am_apply_bgworker()) + { + /* + * Make sure the handle apply_dispatch methods are aware we're in a + * remote transaction. + */ + in_remote_transaction = true; + + /* If we are in a apply background worker, begin the transaction */ + AcceptInvalidationMessages(); + maybe_reread_subscription(); + + StartTransactionCommand(); + BeginTransactionBlock(); + CommitTransactionCommand(); + + return; + } + /* - * Initialize the worker's stream_fileset if we haven't yet. This will be - * used for the entire duration of the worker so create it in a permanent - * context. We create this on the very first streaming message from any - * transaction and then use it for this and other streaming transactions. - * Now, we could create a fileset at the start of the worker as well but - * then we won't be sure that it will ever be used. + * This is the main apply worker. Check if there is any free apply + * background worker we can use to process this transaction. */ - if (MyLogicalRepWorker->stream_fileset == NULL) + stream_apply_worker = apply_bgworker_find_or_start(stream_xid, first_segment); + + if (stream_apply_worker) { - MemoryContext oldctx; + /* + * If we have found a free worker or if we are already applying this + * transaction in an apply background worker, then we pass the data to + * that worker. + */ + if (first_segment) + apply_bgworker_send_data(stream_apply_worker, s->len, s->data); - oldctx = MemoryContextSwitchTo(ApplyContext); + nchanges = 0; + elog(DEBUG1, "starting streaming of xid %u", stream_xid); + } - MyLogicalRepWorker->stream_fileset = palloc(sizeof(FileSet)); - FileSetInit(MyLogicalRepWorker->stream_fileset); + /* + * If no worker is available for the first stream start, we start to + * serialize all the changes of the transaction. + */ + else + { + /* + * Start a transaction on stream start, this transaction will be + * committed on the stream stop unless it is a tablesync worker in + * which case it will be committed after processing all the messages. + * We need the transaction for handling the buffile, used for + * serializing the streaming data and subxact info. + */ + begin_replication_step(); - MemoryContextSwitchTo(oldctx); - } + /* + * Initialize the worker's stream_fileset if we haven't yet. This will + * be used for the entire duration of the worker so create it in a + * permanent context. We create this on the very first streaming + * message from any transaction and then use it for this and other + * streaming transactions. Now, we could create a fileset at the start + * of the worker as well but then we won't be sure that it will ever + * be used. + */ + if (MyLogicalRepWorker->stream_fileset == NULL) + { + MemoryContext oldctx; - /* open the spool file for this transaction */ - stream_open_file(MyLogicalRepWorker->subid, stream_xid, first_segment); + oldctx = MemoryContextSwitchTo(ApplyContext); - /* if this is not the first segment, open existing subxact file */ - if (!first_segment) - subxact_info_read(MyLogicalRepWorker->subid, stream_xid); + MyLogicalRepWorker->stream_fileset = palloc(sizeof(FileSet)); + FileSetInit(MyLogicalRepWorker->stream_fileset); - pgstat_report_activity(STATE_RUNNING, NULL); + MemoryContextSwitchTo(oldctx); + } - end_replication_step(); + /* open the spool file for this transaction */ + stream_open_file(MyLogicalRepWorker->subid, stream_xid, first_segment); + + /* if this is not the first segment, open existing subxact file */ + if (!first_segment) + subxact_info_read(MyLogicalRepWorker->subid, stream_xid); + + end_replication_step(); + } + + pgstat_report_activity(STATE_RUNNING, NULL); } /* @@ -1220,44 +1363,47 @@ apply_handle_stream_stop(StringInfo s) (errcode(ERRCODE_PROTOCOL_VIOLATION), errmsg_internal("STREAM STOP message without STREAM START"))); - /* - * Close the file with serialized changes, and serialize information about - * subxacts for the toplevel transaction. - */ - subxact_info_write(MyLogicalRepWorker->subid, stream_xid); - stream_close_file(); + if (apply_bgworker_active()) + { + char action = LOGICAL_REP_MSG_STREAM_STOP; - /* We must be in a valid transaction state */ - Assert(IsTransactionState()); + apply_bgworker_send_data(stream_apply_worker, 1, &action); + apply_bgworker_wait_for(stream_apply_worker, APPLY_BGWORKER_READY); - /* Commit the per-stream transaction */ - CommitTransactionCommand(); + elog(DEBUG1, "stopped streaming of xid %u, %u changes streamed", stream_xid, nchanges); + } + else + { + /* + * Close the file with serialized changes, and serialize information + * about subxacts for the toplevel transaction. + */ + subxact_info_write(MyLogicalRepWorker->subid, stream_xid); + stream_close_file(); - in_streamed_transaction = false; + /* We must be in a valid transaction state */ + Assert(IsTransactionState()); + + /* Commit the per-stream transaction */ + CommitTransactionCommand(); + + /* Reset per-stream context */ + MemoryContextReset(LogicalStreamingContext); + } - /* Reset per-stream context */ - MemoryContextReset(LogicalStreamingContext); + in_streamed_transaction = false; + stream_apply_worker = NULL; pgstat_report_activity(STATE_IDLE, NULL); reset_apply_error_context_info(); } /* - * Handle STREAM abort message. + * Handle STREAM ABORT message when the transaction was spilled to disk. */ static void -apply_handle_stream_abort(StringInfo s) +serialize_stream_abort(TransactionId xid, TransactionId subxid) { - TransactionId xid; - TransactionId subxid; - - if (in_streamed_transaction) - ereport(ERROR, - (errcode(ERRCODE_PROTOCOL_VIOLATION), - errmsg_internal("STREAM ABORT message without STREAM STOP"))); - - logicalrep_read_stream_abort(s, &xid, &subxid); - /* * If the two XIDs are the same, it's in fact abort of toplevel xact, so * just delete the files with serialized info. @@ -1339,8 +1485,137 @@ apply_handle_stream_abort(StringInfo s) end_replication_step(); CommitTransactionCommand(); } +} - reset_apply_error_context_info(); +/* + * Handle STREAM ABORT message. + */ +static void +apply_handle_stream_abort(StringInfo s) +{ + TransactionId xid; + TransactionId subxid; + LogicalRepStreamAbortData abort_data; + bool include_abort_lsn = false; + + if (in_streamed_transaction) + ereport(ERROR, + (errcode(ERRCODE_PROTOCOL_VIOLATION), + errmsg_internal("STREAM ABORT message without STREAM STOP"))); + + /* Check whether the publisher sends abort_lsn and abort_time. */ + if (am_apply_bgworker()) + include_abort_lsn = MyParallelState->server_version >= 150000; + + logicalrep_read_stream_abort(s, &abort_data, include_abort_lsn); + + xid = abort_data.xid; + subxid = abort_data.subxid; + + if (am_apply_bgworker()) + { + elog(DEBUG1, "[Apply BGW #%u] aborting current transaction xid=%u, subxid=%u", + MyParallelState->n, GetCurrentTransactionIdIfAny(), + GetCurrentSubTransactionId()); + + /* + * Update origin state so we can restart streaming from correct + * position in case of crash. + */ + if (include_abort_lsn) + { + replorigin_session_origin_lsn = abort_data.abort_lsn; + replorigin_session_origin_timestamp = abort_data.abort_time; + } + + /* + * If the two XIDs are the same, it's in fact abort of toplevel xact, + * so just free the subxactlist. + */ + if (subxid == xid) + { + set_apply_error_context_xact(subxid, InvalidXLogRecPtr); + + AbortCurrentTransaction(); + + EndTransactionBlock(false); + CommitTransactionCommand(); + + in_remote_transaction = false; + + list_free(subxactlist); + subxactlist = NIL; + + apply_bgworker_set_status(APPLY_BGWORKER_FINISHED); + } + else + { + /* + * OK, so it's a subxact. Rollback to the savepoint. + * + * We also need to read the subxactlist, determine the offset + * tracked for the subxact, and truncate the list. + */ + int i; + bool found = false; + char spname[MAXPGPATH]; + + set_apply_error_context_xact(subxid, InvalidXLogRecPtr); + snprintf(spname, MAXPGPATH, "savepoint_for_xid_%u", subxid); + + elog(DEBUG1, "[Apply BGW #%u] rolling back to savepoint %s", + MyParallelState->n, spname); + + for (i = list_length(subxactlist) - 1; i >= 0; i--) + { + xid = (TransactionId) list_nth_int(subxactlist, i); + if (xid == subxid) + { + found = true; + break; + } + } + + if (found) + { + RollbackToSavepoint(spname); + CommitTransactionCommand(); + subxactlist = list_truncate(subxactlist, i + 1); + } + + apply_bgworker_set_status(APPLY_BGWORKER_READY); + } + + reset_apply_error_context_info(); + } + else + { + ApplyBgworkerState *wstate = apply_bgworker_find_or_start(xid, false); + + /* + * This is the main apply worker. Check if we are processing this + * transaction in a apply background worker. + */ + if (wstate) + { + apply_bgworker_send_data(wstate, s->len, s->data); + + if (subxid == xid) + { + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_FINISHED); + apply_bgworker_free(wstate); + } + else + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_READY); + } + + /* + * We are in main apply worker and the transaction has been serialized + * to file. + */ + else + serialize_stream_abort(xid, subxid); + } } /* @@ -1462,40 +1737,6 @@ apply_spooled_messages(TransactionId xid, XLogRecPtr lsn) return; } -/* - * Handle STREAM COMMIT message. - */ -static void -apply_handle_stream_commit(StringInfo s) -{ - TransactionId xid; - LogicalRepCommitData commit_data; - - if (in_streamed_transaction) - ereport(ERROR, - (errcode(ERRCODE_PROTOCOL_VIOLATION), - errmsg_internal("STREAM COMMIT message without STREAM STOP"))); - - xid = logicalrep_read_stream_commit(s, &commit_data); - set_apply_error_context_xact(xid, commit_data.commit_lsn); - - elog(DEBUG1, "received commit for streamed transaction %u", xid); - - apply_spooled_messages(xid, commit_data.commit_lsn); - - apply_handle_commit_internal(&commit_data); - - /* unlink the files with serialized changes and subxact info */ - stream_cleanup_files(MyLogicalRepWorker->subid, xid); - - /* Process any tables that are being synchronized in parallel. */ - process_syncing_tables(commit_data.end_lsn); - - pgstat_report_activity(STATE_IDLE, NULL); - - reset_apply_error_context_info(); -} - /* * Helper function for apply_handle_commit and apply_handle_stream_commit. */ @@ -2445,11 +2686,105 @@ apply_handle_truncate(StringInfo s) end_replication_step(); } +/* + * Handle STREAM COMMIT message. + */ +static void +apply_handle_stream_commit(StringInfo s) +{ + LogicalRepCommitData commit_data; + TransactionId xid; + + if (in_streamed_transaction) + ereport(ERROR, + (errcode(ERRCODE_PROTOCOL_VIOLATION), + errmsg_internal("STREAM COMMIT message without STREAM STOP"))); + + xid = logicalrep_read_stream_commit(s, &commit_data); + set_apply_error_context_xact(xid, commit_data.commit_lsn); + + if (am_apply_bgworker()) + { + /* + * Update origin state so we can restart streaming from correct + * position in case of crash. + */ + replorigin_session_origin_lsn = commit_data.end_lsn; + replorigin_session_origin_timestamp = commit_data.committime; + + CommitTransactionCommand(); + EndTransactionBlock(false); + CommitTransactionCommand(); + + in_remote_transaction = false; + + pgstat_report_stat(false); + + list_free(subxactlist); + subxactlist = NIL; + + apply_bgworker_set_status(APPLY_BGWORKER_FINISHED); + } + else + { + /* + * This is the main apply worker. Check if we are processing this + * transaction in an apply background worker. + */ + ApplyBgworkerState *wstate = apply_bgworker_find_or_start(xid, false); + + elog(DEBUG1, "received commit for streamed transaction %u", xid); + + if (wstate) + { + /* Send commit message */ + apply_bgworker_send_data(wstate, s->len, s->data); + + /* Wait for apply background worker to finish */ + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_FINISHED); + + pgstat_report_stat(false); + store_flush_position(commit_data.end_lsn); + stop_skipping_changes(); + + apply_bgworker_free(wstate); + + /* + * The transaction is either non-empty or skipped, so we clear the + * subskiplsn. + */ + clear_subscription_skip_lsn(commit_data.commit_lsn); + } + else + { + /* + * This is the main apply worker and the transaction has been + * serialized to file, replay all the spooled operations. + */ + apply_spooled_messages(xid, commit_data.commit_lsn); + + apply_handle_commit_internal(&commit_data); + + /* unlink the files with serialized changes and subxact info */ + stream_cleanup_files(MyLogicalRepWorker->subid, xid); + } + } + + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); + + /* Process any tables that are being synchronized in parallel. */ + process_syncing_tables(commit_data.end_lsn); + + pgstat_report_activity(STATE_IDLE, NULL); + + reset_apply_error_context_info(); +} /* * Logical replication protocol message dispatcher. */ -static void +void apply_dispatch(StringInfo s) { LogicalRepMsgType action = pq_getmsgbyte(s); @@ -2618,6 +2953,10 @@ store_flush_position(XLogRecPtr remote_lsn) { FlushPosition *flushpos; + /* We only need to collect the LSN in main apply worker */ + if (am_apply_bgworker()) + return; + /* Need to do this in permanent context */ MemoryContextSwitchTo(ApplyContext); @@ -2632,7 +2971,7 @@ store_flush_position(XLogRecPtr remote_lsn) /* Update statistics of the worker. */ -static void +void UpdateWorkerStats(XLogRecPtr last_lsn, TimestampTz send_time, bool reply) { MyLogicalRepWorker->last_lsn = last_lsn; @@ -2794,6 +3133,9 @@ LogicalRepApplyLoop(XLogRecPtr last_received) AcceptInvalidationMessages(); maybe_reread_subscription(); + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); + /* Process any table synchronization changes. */ process_syncing_tables(last_received); } @@ -3095,7 +3437,7 @@ maybe_reread_subscription(void) /* * Callback from subscription syscache invalidation. */ -static void +void subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue) { MySubscriptionValid = false; @@ -3691,7 +4033,7 @@ ApplyWorkerMain(Datum main_arg) originid = replorigin_by_name(originname, true); if (!OidIsValid(originid)) originid = replorigin_create(originname); - replorigin_session_setup(originid); + replorigin_session_setup(originid, true); replorigin_session_origin = originid; origin_startpos = replorigin_session_get_progress(false); CommitTransactionCommand(); @@ -3738,7 +4080,7 @@ ApplyWorkerMain(Datum main_arg) options.proto.logical.publication_names = MySubscription->publications; options.proto.logical.binary = MySubscription->binary; - options.proto.logical.streaming = MySubscription->stream; + options.proto.logical.streaming = (MySubscription->stream != SUBSTREAM_OFF); options.proto.logical.twophase = false; if (!am_tablesync_worker()) @@ -3896,7 +4238,8 @@ clear_subscription_skip_lsn(XLogRecPtr finish_lsn) XLogRecPtr myskiplsn = MySubscription->skiplsn; bool started_tx = false; - if (likely(XLogRecPtrIsInvalid(myskiplsn))) + if (likely(XLogRecPtrIsInvalid(myskiplsn)) || + am_apply_bgworker()) return; if (!IsTransactionState()) @@ -3968,7 +4311,7 @@ clear_subscription_skip_lsn(XLogRecPtr finish_lsn) } /* Error callback to give more context info about the change being applied */ -static void +void apply_error_callback(void *arg) { ApplyErrorCallbackArg *errarg = &apply_error_callback_arg; diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c index 8deae57143..6ebfa24baf 100644 --- a/src/backend/replication/pgoutput/pgoutput.c +++ b/src/backend/replication/pgoutput/pgoutput.c @@ -1833,7 +1833,7 @@ pgoutput_stream_abort(struct LogicalDecodingContext *ctx, Assert(rbtxn_is_streamed(toptxn)); OutputPluginPrepareWrite(ctx, true); - logicalrep_write_stream_abort(ctx->out, toptxn->xid, txn->xid); + logicalrep_write_stream_abort(ctx->out, toptxn->xid, txn, abort_lsn); OutputPluginWrite(ctx, true); cleanup_rel_sync_cache(toptxn->xid, false); diff --git a/src/backend/utils/activity/wait_event.c b/src/backend/utils/activity/wait_event.c index 87c15b9c6f..ba781e6f08 100644 --- a/src/backend/utils/activity/wait_event.c +++ b/src/backend/utils/activity/wait_event.c @@ -388,6 +388,9 @@ pgstat_get_wait_ipc(WaitEventIPC w) case WAIT_EVENT_HASH_GROW_BUCKETS_REINSERT: event_name = "HashGrowBucketsReinsert"; break; + case WAIT_EVENT_LOGICAL_APPLY_WORKER_STATE_CHANGE: + event_name = "LogicalApplyWorkerStateChange"; + break; case WAIT_EVENT_LOGICAL_SYNC_DATA: event_name = "LogicalSyncData"; break; diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c index a7cc49898b..519efff1d2 100644 --- a/src/backend/utils/misc/guc.c +++ b/src/backend/utils/misc/guc.c @@ -3220,6 +3220,18 @@ static struct config_int ConfigureNamesInt[] = NULL, NULL, NULL }, + { + {"max_apply_bgworkers_per_subscription", + PGC_SIGHUP, + REPLICATION_SUBSCRIBERS, + gettext_noop("Maximum number of apply backgrand workers per subscription."), + NULL, + }, + &max_apply_bgworkers_per_subscription, + 3, 0, MAX_BACKENDS, + NULL, NULL, NULL + }, + { {"log_rotation_age", PGC_SIGHUP, LOGGING_WHERE, gettext_noop("Sets the amount of time to wait before forcing " diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample index 48ad80cf2e..b71340549d 100644 --- a/src/backend/utils/misc/postgresql.conf.sample +++ b/src/backend/utils/misc/postgresql.conf.sample @@ -360,6 +360,7 @@ #max_logical_replication_workers = 4 # taken from max_worker_processes # (change requires restart) #max_sync_workers_per_subscription = 2 # taken from max_logical_replication_workers +#max_apply_bgworkers_per_subscription = 3 # taken from max_logical_replication_workers #------------------------------------------------------------------------------ diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 7cc9c72e49..6f8b30abb0 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -4450,7 +4450,7 @@ getSubscriptions(Archive *fout) if (fout->remoteVersion >= 140000) appendPQExpBufferStr(query, " s.substream,\n"); else - appendPQExpBufferStr(query, " false AS substream,\n"); + appendPQExpBufferStr(query, " 'f' AS substream,\n"); if (fout->remoteVersion >= 150000) appendPQExpBufferStr(query, @@ -4580,8 +4580,10 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo) if (strcmp(subinfo->subbinary, "t") == 0) appendPQExpBufferStr(query, ", binary = true"); - if (strcmp(subinfo->substream, "f") != 0) + if (strcmp(subinfo->substream, "t") == 0) appendPQExpBufferStr(query, ", streaming = on"); + else if (strcmp(subinfo->substream, "a") == 0) + appendPQExpBufferStr(query, ", streaming = apply"); if (strcmp(subinfo->subtwophasestate, two_phase_disabled) != 0) appendPQExpBufferStr(query, ", two_phase = on"); diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h index d1260f590c..9b394a45fe 100644 --- a/src/include/catalog/pg_subscription.h +++ b/src/include/catalog/pg_subscription.h @@ -68,7 +68,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW bool subbinary; /* True if the subscription wants the * publisher to send data in binary */ - bool substream; /* Stream in-progress transactions. */ + char substream; /* Stream in-progress transactions. + * See SUBSTREAM_xxx constants. */ char subtwophasestate; /* Stream two-phase transactions */ @@ -109,7 +110,7 @@ typedef struct Subscription bool enabled; /* Indicates if the subscription is enabled */ bool binary; /* Indicates if the subscription wants data in * binary format */ - bool stream; /* Allow streaming in-progress transactions. */ + char stream; /* Allow streaming in-progress transactions. */ char twophasestate; /* Allow streaming two-phase transactions */ bool disableonerr; /* Indicates if the subscription should be * automatically disabled if a worker error @@ -120,6 +121,18 @@ typedef struct Subscription List *publications; /* List of publication names to subscribe to */ } Subscription; +/* Disallow streaming in-progress transactions */ +#define SUBSTREAM_OFF 'f' + +/* + * Streaming transactions are written to a temporary file and applied only + * after the transaction is committed on upstream. + */ +#define SUBSTREAM_ON 't' + +/* Streaming transactions are applied immediately via a background worker */ +#define SUBSTREAM_APPLY 'a' + extern Subscription *GetSubscription(Oid subid, bool missing_ok); extern void FreeSubscription(Subscription *sub); extern void DisableSubscription(Oid subid); diff --git a/src/include/replication/logicallauncher.h b/src/include/replication/logicallauncher.h index f1e2821e25..ac8ef94381 100644 --- a/src/include/replication/logicallauncher.h +++ b/src/include/replication/logicallauncher.h @@ -14,6 +14,7 @@ extern PGDLLIMPORT int max_logical_replication_workers; extern PGDLLIMPORT int max_sync_workers_per_subscription; +extern PGDLLIMPORT int max_apply_bgworkers_per_subscription; extern void ApplyLauncherRegister(void); extern void ApplyLauncherMain(Datum main_arg); diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h index a771ab8ff3..7bba77c9e7 100644 --- a/src/include/replication/logicalproto.h +++ b/src/include/replication/logicalproto.h @@ -175,6 +175,17 @@ typedef struct LogicalRepRollbackPreparedTxnData char gid[GIDSIZE]; } LogicalRepRollbackPreparedTxnData; +/* + * Transaction protocol information for stream abort. + */ +typedef struct LogicalRepStreamAbortData +{ + TransactionId xid; + TransactionId subxid; + XLogRecPtr abort_lsn; + TimestampTz abort_time; +} LogicalRepStreamAbortData; + extern void logicalrep_write_begin(StringInfo out, ReorderBufferTXN *txn); extern void logicalrep_read_begin(StringInfo in, LogicalRepBeginData *begin_data); @@ -246,9 +257,11 @@ extern void logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn extern TransactionId logicalrep_read_stream_commit(StringInfo out, LogicalRepCommitData *commit_data); extern void logicalrep_write_stream_abort(StringInfo out, TransactionId xid, - TransactionId subxid); -extern void logicalrep_read_stream_abort(StringInfo in, TransactionId *xid, - TransactionId *subxid); + ReorderBufferTXN *txn, + XLogRecPtr abort_lsn); +extern void logicalrep_read_stream_abort(StringInfo in, + LogicalRepStreamAbortData *abort_data, + bool include_abort_lsn); extern char *logicalrep_message_type(LogicalRepMsgType action); #endif /* LOGICAL_PROTO_H */ diff --git a/src/include/replication/logicalworker.h b/src/include/replication/logicalworker.h index cd1b6e8afc..6a1af7f13c 100644 --- a/src/include/replication/logicalworker.h +++ b/src/include/replication/logicalworker.h @@ -13,6 +13,7 @@ #define LOGICALWORKER_H extern void ApplyWorkerMain(Datum main_arg); +extern void ApplyBgworkerMain(Datum main_arg); extern bool IsLogicalWorker(void); diff --git a/src/include/replication/origin.h b/src/include/replication/origin.h index 2d1b5e5c28..c7389b40a7 100644 --- a/src/include/replication/origin.h +++ b/src/include/replication/origin.h @@ -53,7 +53,7 @@ extern XLogRecPtr replorigin_get_progress(RepOriginId node, bool flush); extern void replorigin_session_advance(XLogRecPtr remote_commit, XLogRecPtr local_commit); -extern void replorigin_session_setup(RepOriginId node); +extern void replorigin_session_setup(RepOriginId node, bool must_acquire); extern void replorigin_session_reset(void); extern XLogRecPtr replorigin_session_get_progress(bool flush); diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h index 4a01f877e5..aba1640f4d 100644 --- a/src/include/replication/reorderbuffer.h +++ b/src/include/replication/reorderbuffer.h @@ -301,6 +301,7 @@ typedef struct ReorderBufferTXN { TimestampTz commit_time; TimestampTz prepare_time; + TimestampTz abort_time; } xact_time; /* @@ -647,9 +648,11 @@ extern void ReorderBufferFinishPrepared(ReorderBuffer *rb, TransactionId xid, extern void ReorderBufferAssignChild(ReorderBuffer *, TransactionId, TransactionId, XLogRecPtr commit_lsn); extern void ReorderBufferCommitChild(ReorderBuffer *, TransactionId, TransactionId, XLogRecPtr commit_lsn, XLogRecPtr end_lsn); -extern void ReorderBufferAbort(ReorderBuffer *, TransactionId, XLogRecPtr lsn); +extern void ReorderBufferAbort(ReorderBuffer *, TransactionId, XLogRecPtr lsn, + TimestampTz abort_time); extern void ReorderBufferAbortOld(ReorderBuffer *, TransactionId xid); -extern void ReorderBufferForget(ReorderBuffer *, TransactionId, XLogRecPtr lsn); +extern void ReorderBufferForget(ReorderBuffer *, TransactionId, XLogRecPtr lsn, + TimestampTz abort_time); extern void ReorderBufferInvalidate(ReorderBuffer *, TransactionId, XLogRecPtr lsn); extern void ReorderBufferSetBaseSnapshot(ReorderBuffer *, TransactionId, XLogRecPtr lsn, struct SnapshotData *snap); diff --git a/src/include/replication/worker_internal.h b/src/include/replication/worker_internal.h index 901845abc2..08e63de013 100644 --- a/src/include/replication/worker_internal.h +++ b/src/include/replication/worker_internal.h @@ -17,8 +17,11 @@ #include "access/xlogdefs.h" #include "catalog/pg_subscription.h" #include "datatype/timestamp.h" +#include "replication/logicalrelation.h" #include "storage/fileset.h" #include "storage/lock.h" +#include "storage/shm_mq.h" +#include "storage/shm_toc.h" #include "storage/spin.h" @@ -60,6 +63,9 @@ typedef struct LogicalRepWorker */ FileSet *stream_fileset; + /* Indicates if this slot is used for an apply background worker. */ + bool subworker; + /* Stats. */ XLogRecPtr last_lsn; TimestampTz last_send_time; @@ -68,9 +74,70 @@ typedef struct LogicalRepWorker TimestampTz reply_time; } LogicalRepWorker; +/* Struct for saving and restoring apply errcontext information */ +typedef struct ApplyErrorCallbackArg +{ + LogicalRepMsgType command; /* 0 if invalid */ + LogicalRepRelMapEntry *rel; + + /* Remote node information */ + int remote_attnum; /* -1 if invalid */ + TransactionId remote_xid; + XLogRecPtr finish_lsn; + char *origin_name; +} ApplyErrorCallbackArg; + +/* + * Status for apply background worker. + */ +typedef enum ApplyBgworkerStatus +{ + APPLY_BGWORKER_ATTACHED = 0, + APPLY_BGWORKER_READY, + APPLY_BGWORKER_BUSY, + APPLY_BGWORKER_FINISHED, + APPLY_BGWORKER_EXIT +} ApplyBgworkerStatus; + +/* + * Shared information among apply workers. + */ +typedef struct ApplyBgworkerShared +{ + slock_t mutex; + + /* Status for apply background worker. */ + ApplyBgworkerStatus status; + + /* server version of publisher. */ + int server_version; + + TransactionId stream_xid; + uint32 n; /* id of apply background worker */ +} ApplyBgworkerShared; + +/* + * Struct for maintaining an apply background worker. + */ +typedef struct ApplyBgworkerState +{ + shm_mq_handle *mq_handle; + dsm_segment *dsm_seg; + ApplyBgworkerShared volatile *pstate; +} ApplyBgworkerState; + /* Main memory context for apply worker. Permanent during worker lifetime. */ extern PGDLLIMPORT MemoryContext ApplyContext; +extern PGDLLIMPORT MemoryContext ApplyMessageContext; + +extern PGDLLIMPORT ApplyErrorCallbackArg apply_error_callback_arg; + +extern PGDLLIMPORT bool MySubscriptionValid; + +extern PGDLLIMPORT volatile ApplyBgworkerShared *MyParallelState; +extern PGDLLIMPORT List *subxactlist; + /* libpqreceiver connection */ extern PGDLLIMPORT struct WalReceiverConn *LogRepWorkerWalRcvConn; @@ -79,18 +146,22 @@ extern PGDLLIMPORT Subscription *MySubscription; extern PGDLLIMPORT LogicalRepWorker *MyLogicalRepWorker; extern PGDLLIMPORT bool in_remote_transaction; +extern PGDLLIMPORT bool in_streamed_transaction; +extern PGDLLIMPORT TransactionId stream_xid; extern void logicalrep_worker_attach(int slot); extern LogicalRepWorker *logicalrep_worker_find(Oid subid, Oid relid, bool only_running); extern List *logicalrep_workers_find(Oid subid, bool only_running); -extern void logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, - Oid userid, Oid relid); +extern bool logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, + Oid userid, Oid relid, + dsm_handle subworker_dsm); extern void logicalrep_worker_stop(Oid subid, Oid relid); extern void logicalrep_worker_wakeup(Oid subid, Oid relid); extern void logicalrep_worker_wakeup_ptr(LogicalRepWorker *worker); extern int logicalrep_sync_worker_count(Oid subid); +extern int logicalrep_apply_background_worker_count(Oid subid); extern void ReplicationOriginNameForTablesync(Oid suboid, Oid relid, char *originname, int szorgname); @@ -103,10 +174,39 @@ extern void process_syncing_tables(XLogRecPtr current_lsn); extern void invalidate_syncing_table_states(Datum arg, int cacheid, uint32 hashvalue); +extern void UpdateWorkerStats(XLogRecPtr last_lsn, TimestampTz send_time, + bool reply); + +/* prototype needed because of stream_commit */ +extern void apply_dispatch(StringInfo s); + +/* Function for apply error callback */ +extern void apply_error_callback(void *arg); + +extern void subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue); + +/* apply background worker setup and interactions */ +extern ApplyBgworkerState *apply_bgworker_find_or_start(TransactionId xid, + bool start); +extern void apply_bgworker_wait_for(ApplyBgworkerState *wstate, + ApplyBgworkerStatus wait_for_status); +extern void apply_bgworker_send_data(ApplyBgworkerState *wstate, Size nbytes, + const void *data); +extern void apply_bgworker_free(ApplyBgworkerState *wstate); +extern void apply_bgworker_check_status(void); +extern void apply_bgworker_set_status(ApplyBgworkerStatus status); +extern void apply_bgworker_subxact_info_add(TransactionId current_xid); + static inline bool am_tablesync_worker(void) { return OidIsValid(MyLogicalRepWorker->relid); } +static inline bool +am_apply_bgworker(void) +{ + return MyLogicalRepWorker->subworker; +} + #endif /* WORKER_INTERNAL_H */ diff --git a/src/include/utils/wait_event.h b/src/include/utils/wait_event.h index b578e2ec75..c2d2a114d7 100644 --- a/src/include/utils/wait_event.h +++ b/src/include/utils/wait_event.h @@ -105,6 +105,7 @@ typedef enum WAIT_EVENT_HASH_GROW_BUCKETS_ALLOCATE, WAIT_EVENT_HASH_GROW_BUCKETS_ELECT, WAIT_EVENT_HASH_GROW_BUCKETS_REINSERT, + WAIT_EVENT_LOGICAL_APPLY_WORKER_STATE_CHANGE, WAIT_EVENT_LOGICAL_SYNC_DATA, WAIT_EVENT_LOGICAL_SYNC_STATE_CHANGE, WAIT_EVENT_MQ_INTERNAL, diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out index 7fcfad1591..f769835af0 100644 --- a/src/test/regress/expected/subscription.out +++ b/src/test/regress/expected/subscription.out @@ -197,7 +197,7 @@ ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); DROP SUBSCRIPTION regress_testsub; -- fail - streaming must be boolean CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = foo); -ERROR: streaming requires a Boolean value +ERROR: streaming requires a Boolean value or "apply" -- now it works CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = true); WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 4fb746930a..664d4b8b1a 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -116,6 +116,10 @@ Append AppendPath AppendRelInfo AppendState +ApplyBgworkerEntry +ApplyBgworkerShared +ApplyBgworkerState +ApplyBgworkerStatus ApplyErrorCallbackArg ApplyExecutionData ApplySubXactData @@ -1483,6 +1487,7 @@ LogicalRepRelId LogicalRepRelMapEntry LogicalRepRelation LogicalRepRollbackPreparedTxnData +LogicalRepStreamAbortData LogicalRepTupleData LogicalRepTyp LogicalRepWorker -- 2.23.0.windows.1 [application/octet-stream] v10-0002-Test-streaming-apply-option-in-tap-test.patch (68.9K, ../../OS3PR01MB62756BD9482EB6BB1CA4CD4D9EAA9@OS3PR01MB6275.jpnprd01.prod.outlook.com/3-v10-0002-Test-streaming-apply-option-in-tap-test.patch) download | inline diff: From feac9609b26da1deaef14898ac364dd91d18a99e Mon Sep 17 00:00:00 2001 From: "shiy.fnst" <[email protected]> Date: Fri, 13 May 2022 14:50:30 +0800 Subject: [PATCH v10 2/5] Test streaming apply option in tap test Change all TAP tests using the SUBSCRIPTION "streaming" option, so they now test both 'on' and 'apply' values. --- src/test/subscription/t/015_stream.pl | 199 ++++--- src/test/subscription/t/016_stream_subxact.pl | 119 +++-- src/test/subscription/t/017_stream_ddl.pl | 188 ++++--- .../t/018_stream_subxact_abort.pl | 195 ++++--- .../t/019_stream_subxact_ddl_abort.pl | 110 +++- .../subscription/t/022_twophase_cascade.pl | 363 +++++++------ .../subscription/t/023_twophase_stream.pl | 498 ++++++++++-------- 7 files changed, 1035 insertions(+), 637 deletions(-) diff --git a/src/test/subscription/t/015_stream.pl b/src/test/subscription/t/015_stream.pl index 6561b189de..b3d84b1c7c 100644 --- a/src/test/subscription/t/015_stream.pl +++ b/src/test/subscription/t/015_stream.pl @@ -8,6 +8,116 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" option +# so the same code can be run both for the streaming=on and streaming=apply +# cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_apply) = @_; + + # Interleave a pair of transactions, each exceeding the 64kB limit. + my $in = ''; + my $out = ''; + + my $timer = IPC::Run::timeout($PostgreSQL::Test::Utils::timeout_default); + + my $h = $node_publisher->background_psql('postgres', \$in, \$out, $timer, + on_error_stop => 0); + + # Check that a background worker starts if "streaming" option is + # specified as "apply". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_apply) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + $in .= q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + }; + $h->pump_nb; + + if ($is_apply) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001, 9999) s(i); + DELETE FROM test_tab WHERE a > 5000; + COMMIT; + }); + + $in .= q{ + COMMIT; + \q + }; + $h->finish; # errors make the next test fail, so ignore them here + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), + 'check extra columns contain local defaults'); + + # Test the streaming in binary mode + $node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET (binary = on)"); + + # Insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001, 10000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + COMMIT; + }); + + $node_publisher->wait_for_catchup($appname); + + $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(6667|6667|6667), + 'check extra columns contain local defaults'); + + # Change the local values of the extra columns on the subscriber, + # update publisher, and check that subscriber retains the expected + # values. This is to ensure that non-streaming transactions behave + # properly after a streaming transaction. + $node_subscriber->safe_psql('postgres', + "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'" + ); + $node_publisher->safe_psql('postgres', + "UPDATE test_tab SET b = md5(a::text)"); + + $node_publisher->wait_for_catchup($appname); + + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab" + ); + is($result, qq(6667|6667|6667), + 'check extra columns contain locally changed data'); + + # Cleanup the test data + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE (a > 2)"); + $node_publisher->wait_for_catchup($appname); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -37,6 +147,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -54,82 +168,25 @@ my $result = "SELECT count(*), count(c), count(d = 999) FROM test_tab"); is($result, qq(2|2|2), 'check initial data was copied to subscriber'); -# Interleave a pair of transactions, each exceeding the 64kB limit. -my $in = ''; -my $out = ''; - -my $timer = IPC::Run::timeout($PostgreSQL::Test::Utils::timeout_default); - -my $h = $node_publisher->background_psql('postgres', \$in, \$out, $timer, - on_error_stop => 0); - -$in .= q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -}; -$h->pump_nb; - -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001, 9999) s(i); -DELETE FROM test_tab WHERE a > 5000; -COMMIT; -}); - -$in .= q{ -COMMIT; -\q -}; -$h->finish; # errors make the next test fail, so ignore them here - -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), 'check extra columns contain local defaults'); - -# Test the streaming in binary mode -$node_subscriber->safe_psql('postgres', - "ALTER SUBSCRIPTION tap_sub SET (binary = on)"); - -# Insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001, 10000) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -COMMIT; -}); - -$node_publisher->wait_for_catchup($appname); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(6667|6667|6667), 'check extra columns contain local defaults'); +################################### +# Test using streaming mode 'apply' +################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" +); -# Change the local values of the extra columns on the subscriber, -# update publisher, and check that subscriber retains the expected -# values. This is to ensure that non-streaming transactions behave -# properly after a streaming transaction. $node_subscriber->safe_psql('postgres', - "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'" -); -$node_publisher->safe_psql('postgres', - "UPDATE test_tab SET b = md5(a::text)"); + "ALTER SUBSCRIPTION tap_sub SET(streaming = apply, binary = off)"); -$node_publisher->wait_for_catchup($appname); +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab" -); -is($result, qq(6667|6667|6667), - 'check extra columns contain locally changed data'); +test_streaming($node_publisher, $node_subscriber, $appname, 1); $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/016_stream_subxact.pl b/src/test/subscription/t/016_stream_subxact.pl index f27f1694f2..b5b6dc379f 100644 --- a/src/test/subscription/t/016_stream_subxact.pl +++ b/src/test/subscription/t/016_stream_subxact.pl @@ -8,6 +8,72 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" option +# so the same code can be run both for the streaming=on and streaming=apply +# cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_apply) = @_; + + # Check that a background worker starts if "streaming" option is + # specified as "apply". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_apply) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + # Insert, update and delete enough rows to exceed 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series( 3, 500) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501, 1000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001, 1500) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + SAVEPOINT s3; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501, 2000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + SAVEPOINT s4; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001, 2500) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + COMMIT; + }); + + if ($is_apply) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(1667|1667|1667), + 'check data was copied to subscriber in streaming mode and extra columns contain local defaults' + ); + + # Cleanup the test data + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE (a > 2)"); + $node_publisher->wait_for_catchup($appname); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -37,6 +103,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -54,41 +124,26 @@ my $result = "SELECT count(*), count(c), count(d = 999) FROM test_tab"); is($result, qq(2|2|2), 'check initial data was copied to subscriber'); -# Insert, update and delete enough rows to exceed 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series( 3, 500) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501, 1000) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001, 1500) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -SAVEPOINT s3; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501, 2000) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -SAVEPOINT s4; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001, 2500) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -COMMIT; -}); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(1667|1667|1667), - 'check data was copied to subscriber in streaming mode and extra columns contain local defaults' +################################### +# Test using streaming mode 'apply' +################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" ); +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = apply)"); + +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; + +test_streaming($node_publisher, $node_subscriber, $appname, 1); + $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/017_stream_ddl.pl b/src/test/subscription/t/017_stream_ddl.pl index 0bce63b716..7e5dc18cd2 100644 --- a/src/test/subscription/t/017_stream_ddl.pl +++ b/src/test/subscription/t/017_stream_ddl.pl @@ -8,6 +8,111 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" option +# so the same code can be run both for the streaming=on and streaming=apply +# cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_apply) = @_; + + # a small (non-streamed) transaction with DDL and DML + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab VALUES (3, md5(3::text)); + ALTER TABLE test_tab ADD COLUMN c INT; + SAVEPOINT s1; + INSERT INTO test_tab VALUES (4, md5(4::text), -4); + COMMIT; + }); + + # Check that a background worker starts if "streaming" option is + # specified as "apply". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_apply) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + # large (streamed) transaction with DDL and DML + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(5, 1000) s(i); + ALTER TABLE test_tab ADD COLUMN d INT; + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001, 2000) s(i); + COMMIT; + }); + + if ($is_apply) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + # a small (non-streamed) transaction with DDL and DML + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab VALUES (2001, md5(2001::text), -2001, 2*2001); + ALTER TABLE test_tab ADD COLUMN e INT; + SAVEPOINT s1; + INSERT INTO test_tab VALUES (2002, md5(2002::text), -2002, 2*2002, -3*2002); + COMMIT; + }); + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d), count(e) FROM test_tab"); + is($result, qq(2002|1999|1002|1), + 'check data was copied to subscriber in streaming mode and extra columns contain local defaults' + ); + + # A large (streamed) transaction with DDL and DML. One of the DDL is performed + # after DML to ensure that we invalidate the schema sent for test_tab so that + # the next transaction has to send the schema again. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(2003,5000) s(i); + ALTER TABLE test_tab ADD COLUMN f INT; + COMMIT; + }); + + # A small transaction that won't get streamed. This is just to ensure that we + # send the schema again to reflect the last column added in the previous test. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i, 4*i FROM generate_series(5001,5005) s(i); + COMMIT; + }); + + $node_publisher->wait_for_catchup($appname); + + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d), count(e), count(f) FROM test_tab" + ); + is($result, qq(5005|5002|4005|3004|5), + 'check data was copied to subscriber for both streaming and non-streaming transactions' + ); + + # Cleanup the test data + $node_publisher->safe_psql( + 'postgres', q{ + DELETE FROM test_tab WHERE (a > 2); + ALTER TABLE test_tab DROP COLUMN c, DROP COLUMN d, DROP COLUMN e, DROP COLUMN f; + }); + $node_publisher->wait_for_catchup($appname); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -37,6 +142,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -54,76 +163,25 @@ my $result = "SELECT count(*), count(c), count(d = 999) FROM test_tab"); is($result, qq(2|0|0), 'check initial data was copied to subscriber'); -# a small (non-streamed) transaction with DDL and DML -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab VALUES (3, md5(3::text)); -ALTER TABLE test_tab ADD COLUMN c INT; -SAVEPOINT s1; -INSERT INTO test_tab VALUES (4, md5(4::text), -4); -COMMIT; -}); - -# large (streamed) transaction with DDL and DML -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(5, 1000) s(i); -ALTER TABLE test_tab ADD COLUMN d INT; -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001, 2000) s(i); -COMMIT; -}); - -# a small (non-streamed) transaction with DDL and DML -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab VALUES (2001, md5(2001::text), -2001, 2*2001); -ALTER TABLE test_tab ADD COLUMN e INT; -SAVEPOINT s1; -INSERT INTO test_tab VALUES (2002, md5(2002::text), -2002, 2*2002, -3*2002); -COMMIT; -}); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d), count(e) FROM test_tab"); -is($result, qq(2002|1999|1002|1), - 'check data was copied to subscriber in streaming mode and extra columns contain local defaults' +################################### +# Test using streaming mode 'apply' +################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" ); -# A large (streamed) transaction with DDL and DML. One of the DDL is performed -# after DML to ensure that we invalidate the schema sent for test_tab so that -# the next transaction has to send the schema again. -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(2003,5000) s(i); -ALTER TABLE test_tab ADD COLUMN f INT; -COMMIT; -}); - -# A small transaction that won't get streamed. This is just to ensure that we -# send the schema again to reflect the last column added in the previous test. -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i, 4*i FROM generate_series(5001,5005) s(i); -COMMIT; -}); +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = apply)"); -$node_publisher->wait_for_catchup($appname); +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d), count(e), count(f) FROM test_tab"); -is($result, qq(5005|5002|4005|3004|5), - 'check data was copied to subscriber for both streaming and non-streaming transactions' -); +test_streaming($node_publisher, $node_subscriber, $appname, 1); $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/018_stream_subxact_abort.pl b/src/test/subscription/t/018_stream_subxact_abort.pl index 7155442e76..71f8c1dd05 100644 --- a/src/test/subscription/t/018_stream_subxact_abort.pl +++ b/src/test/subscription/t/018_stream_subxact_abort.pl @@ -8,6 +8,113 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" option +# so the same code can be run both for the streaming=on and streaming=apply +# cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_apply) = @_; + + # Check that a background worker starts if "streaming" option is + # specified as "apply". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_apply) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + # large (streamed) transaction with DDL, DML and ROLLBACKs + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i); + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501,1000) s(i); + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001,1500) s(i); + SAVEPOINT s3; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501,2000) s(i); + ROLLBACK TO s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001,2500) s(i); + ROLLBACK TO s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2501,3000) s(i); + SAVEPOINT s4; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3001,3500) s(i); + SAVEPOINT s5; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3501,4000) s(i); + COMMIT; + }); + + if ($is_apply) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c) FROM test_tab"); + is($result, qq(2000|0), + 'check rollback to savepoint was reflected on subscriber and extra columns contain local defaults' + ); + + # large (streamed) transaction with subscriber receiving out of order + # subtransaction ROLLBACKs + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(4001,4500) s(i); + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001,5500) s(i); + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(6001,6500) s(i); + SAVEPOINT s3; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(7001,7500) s(i); + RELEASE s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(8001,8500) s(i); + ROLLBACK TO s1; + COMMIT; + }); + + $node_publisher->wait_for_catchup($appname); + + $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c) FROM test_tab"); + is($result, qq(2500|0), + 'check rollback to savepoint was reflected on subscriber'); + + # large (streamed) transaction with subscriber receiving rollback + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(8501,9000) s(i); + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(9001,9500) s(i); + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(9501,10000) s(i); + ROLLBACK; + }); + + $node_publisher->wait_for_catchup($appname); + + $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c) FROM test_tab"); + is($result, qq(2500|0), 'check rollback was reflected on subscriber'); + + # Cleanup the test data + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE (a > 2)"); + $node_publisher->wait_for_catchup($appname); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -36,6 +143,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -53,81 +164,25 @@ my $result = "SELECT count(*), count(c) FROM test_tab"); is($result, qq(2|0), 'check initial data was copied to subscriber'); -# large (streamed) transaction with DDL, DML and ROLLBACKs -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i); -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501,1000) s(i); -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001,1500) s(i); -SAVEPOINT s3; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501,2000) s(i); -ROLLBACK TO s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001,2500) s(i); -ROLLBACK TO s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2501,3000) s(i); -SAVEPOINT s4; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3001,3500) s(i); -SAVEPOINT s5; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3501,4000) s(i); -COMMIT; -}); - -$node_publisher->wait_for_catchup($appname); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c) FROM test_tab"); -is($result, qq(2000|0), - 'check rollback to savepoint was reflected on subscriber and extra columns contain local defaults' +################################### +# Test using streaming mode 'apply' +################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" ); -# large (streamed) transaction with subscriber receiving out of order -# subtransaction ROLLBACKs -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(4001,4500) s(i); -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001,5500) s(i); -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(6001,6500) s(i); -SAVEPOINT s3; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(7001,7500) s(i); -RELEASE s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(8001,8500) s(i); -ROLLBACK TO s1; -COMMIT; -}); - -$node_publisher->wait_for_catchup($appname); +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = apply)"); -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c) FROM test_tab"); -is($result, qq(2500|0), - 'check rollback to savepoint was reflected on subscriber'); - -# large (streamed) transaction with subscriber receiving rollback -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(8501,9000) s(i); -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(9001,9500) s(i); -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(9501,10000) s(i); -ROLLBACK; -}); +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c) FROM test_tab"); -is($result, qq(2500|0), 'check rollback was reflected on subscriber'); +test_streaming($node_publisher, $node_subscriber, $appname, 1); $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/019_stream_subxact_ddl_abort.pl b/src/test/subscription/t/019_stream_subxact_ddl_abort.pl index dbd0fca4d1..d0cd869430 100644 --- a/src/test/subscription/t/019_stream_subxact_ddl_abort.pl +++ b/src/test/subscription/t/019_stream_subxact_ddl_abort.pl @@ -9,6 +9,69 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" option +# so the same code can be run both for the streaming=on and streaming=apply +# cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_apply) = @_; + + # Check that a background worker starts if "streaming" option is + # specified as "apply". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_apply) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + # large (streamed) transaction with DDL, DML and ROLLBACKs + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i); + ALTER TABLE test_tab ADD COLUMN c INT; + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(501,1000) s(i); + ALTER TABLE test_tab ADD COLUMN d INT; + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001,1500) s(i); + ALTER TABLE test_tab ADD COLUMN e INT; + SAVEPOINT s3; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(1501,2000) s(i); + ALTER TABLE test_tab DROP COLUMN c; + ROLLBACK TO s1; + INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(501,1000) s(i); + COMMIT; + }); + + if ($is_apply) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c) FROM test_tab"); + is($result, qq(1000|500), + 'check rollback to savepoint was reflected on subscriber and extra columns contain local defaults' + ); + + # Cleanup the test data + $node_publisher->safe_psql( + 'postgres', q{ + DELETE FROM test_tab WHERE (a > 2); + ALTER TABLE test_tab DROP COLUMN c; + }); + $node_publisher->wait_for_catchup($appname); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -37,6 +100,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -54,35 +121,26 @@ my $result = "SELECT count(*), count(c) FROM test_tab"); is($result, qq(2|0), 'check initial data was copied to subscriber'); -# large (streamed) transaction with DDL, DML and ROLLBACKs -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i); -ALTER TABLE test_tab ADD COLUMN c INT; -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(501,1000) s(i); -ALTER TABLE test_tab ADD COLUMN d INT; -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001,1500) s(i); -ALTER TABLE test_tab ADD COLUMN e INT; -SAVEPOINT s3; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(1501,2000) s(i); -ALTER TABLE test_tab DROP COLUMN c; -ROLLBACK TO s1; -INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(501,1000) s(i); -COMMIT; -}); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c) FROM test_tab"); -is($result, qq(1000|500), - 'check rollback to savepoint was reflected on subscriber and extra columns contain local defaults' +################################### +# Test using streaming mode 'apply' +################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" ); +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = apply)"); + +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; + +test_streaming($node_publisher, $node_subscriber, $appname, 1); + $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/022_twophase_cascade.pl b/src/test/subscription/t/022_twophase_cascade.pl index 7a797f37ba..b52af74e35 100644 --- a/src/test/subscription/t/022_twophase_cascade.pl +++ b/src/test/subscription/t/022_twophase_cascade.pl @@ -11,6 +11,208 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" option +# so the same code can be run both for the streaming=on and streaming=apply +# cases. +sub test_streaming +{ + my ($node_A, $node_B, $node_C, $appname_B, $appname_C, $streaming_mode) = + @_; + + my $oldpid_B = $node_A->safe_psql( + 'postgres', " + SELECT pid FROM pg_stat_replication + WHERE application_name = '$appname_B' AND state = 'streaming';"); + my $oldpid_C = $node_B->safe_psql( + 'postgres', " + SELECT pid FROM pg_stat_replication + WHERE application_name = '$appname_C' AND state = 'streaming';"); + + # Setup logical replication streaming mode + + $node_B->safe_psql( + 'postgres', " + ALTER SUBSCRIPTION tap_sub_B + SET (streaming = $streaming_mode);"); + $node_C->safe_psql( + 'postgres', " + ALTER SUBSCRIPTION tap_sub_C + SET (streaming = $streaming_mode)"); + + # Wait for subscribers to finish initialization + + $node_A->poll_query_until( + 'postgres', " + SELECT pid != $oldpid_B FROM pg_stat_replication + WHERE application_name = '$appname_B' AND state = 'streaming';" + ) or die "Timed out while waiting for apply to restart"; + $node_B->poll_query_until( + 'postgres', " + SELECT pid != $oldpid_C FROM pg_stat_replication + WHERE application_name = '$appname_C' AND state = 'streaming';" + ) or die "Timed out while waiting for apply to restart"; + + ############################### + # Test 2PC PREPARE / COMMIT PREPARED. + # 1. Data is streamed as a 2PC transaction. + # 2. Then do commit prepared. + # + # Expect all data is replicated on subscriber(s) after the commit. + ############################### + + # Check that a background worker starts if "streaming" option is + # specified as "apply". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($streaming_mode eq 'apply') + { + $node_B->append_conf('postgresql.conf', "log_min_messages = debug1"); + $node_B->reload; + + $node_C->append_conf('postgresql.conf', "log_min_messages = debug1"); + $node_C->reload; + } + + # Insert, update and delete enough rows to exceed the 64kB limit. + # Then 2PC PREPARE + $node_A->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + if ($streaming_mode eq 'apply') + { + $node_B->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_B->append_conf('postgresql.conf', "log_min_messages = warning"); + $node_B->reload; + + $node_C->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_C->append_conf('postgresql.conf', "log_min_messages = warning"); + $node_C->reload; + } + + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); + + # check the transaction state is prepared on subscriber(s) + my $result = + $node_B->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber B'); + $result = + $node_C->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber C'); + + # 2PC COMMIT + $node_A->safe_psql('postgres', "COMMIT PREPARED 'test_prepared_tab';"); + + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); + + # check that transaction was committed on subscriber(s) + $result = $node_B->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), + 'Rows inserted by 2PC have committed on subscriber B, and extra columns have local defaults' + ); + $result = $node_C->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), + 'Rows inserted by 2PC have committed on subscriber C, and extra columns have local defaults' + ); + + # check the transaction state is ended on subscriber(s) + $result = + $node_B->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is committed on subscriber B'); + $result = + $node_C->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is committed on subscriber C'); + + ############################### + # Test 2PC PREPARE with a nested ROLLBACK TO SAVEPOINT. + # 0. Cleanup from previous test leaving only 2 rows. + # 1. Insert one more row. + # 2. Record a SAVEPOINT. + # 3. Data is streamed using 2PC. + # 4. Do rollback to SAVEPOINT prior to the streamed inserts. + # 5. Then COMMIT PREPARED. + # + # Expect data after the SAVEPOINT is aborted leaving only 3 rows (= 2 original + 1 from step 1). + ############################### + + # First, delete the data except for 2 rows (delete will be replicated) + $node_A->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); + + # 2PC PREPARE with a nested ROLLBACK TO SAVEPOINT + $node_A->safe_psql( + 'postgres', " + BEGIN; + INSERT INTO test_tab VALUES (9999, 'foobar'); + SAVEPOINT sp_inner; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + ROLLBACK TO SAVEPOINT sp_inner; + PREPARE TRANSACTION 'outer'; + "); + + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); + + # check the transaction state prepared on subscriber(s) + $result = + $node_B->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber B'); + $result = + $node_C->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber C'); + + # 2PC COMMIT + $node_A->safe_psql('postgres', "COMMIT PREPARED 'outer';"); + + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); + + # check the transaction state is ended on subscriber + $result = + $node_B->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is ended on subscriber B'); + $result = + $node_C->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is ended on subscriber C'); + + # check inserts are visible at subscriber(s). + # All the streamed data (prior to the SAVEPOINT) should be rolled back. + # (9999, 'foobar') should be committed. + $result = $node_B->safe_psql('postgres', + "SELECT count(*) FROM test_tab where b = 'foobar';"); + is($result, qq(1), 'Rows committed are present on subscriber B'); + $result = + $node_B->safe_psql('postgres', "SELECT count(*) FROM test_tab;"); + is($result, qq(3), 'Rows committed are present on subscriber B'); + $result = $node_C->safe_psql('postgres', + "SELECT count(*) FROM test_tab where b = 'foobar';"); + is($result, qq(1), 'Rows committed are present on subscriber C'); + $result = + $node_C->safe_psql('postgres', "SELECT count(*) FROM test_tab;"); + is($result, qq(3), 'Rows committed are present on subscriber C'); + + # Cleanup the test data + $node_A->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); +} + ############################### # Setup a cascade of pub/sub nodes. # node_A -> node_B -> node_C @@ -260,160 +462,15 @@ is($result, qq(21), 'Rows committed are present on subscriber C'); # 2PC + STREAMING TESTS # --------------------- -my $oldpid_B = $node_A->safe_psql( - 'postgres', " - SELECT pid FROM pg_stat_replication - WHERE application_name = '$appname_B' AND state = 'streaming';"); -my $oldpid_C = $node_B->safe_psql( - 'postgres', " - SELECT pid FROM pg_stat_replication - WHERE application_name = '$appname_C' AND state = 'streaming';"); - -# Setup logical replication (streaming = on) - -$node_B->safe_psql( - 'postgres', " - ALTER SUBSCRIPTION tap_sub_B - SET (streaming = on);"); -$node_C->safe_psql( - 'postgres', " - ALTER SUBSCRIPTION tap_sub_C - SET (streaming = on)"); - -# Wait for subscribers to finish initialization - -$node_A->poll_query_until( - 'postgres', " - SELECT pid != $oldpid_B FROM pg_stat_replication - WHERE application_name = '$appname_B' AND state = 'streaming';" -) or die "Timed out while waiting for apply to restart"; -$node_B->poll_query_until( - 'postgres', " - SELECT pid != $oldpid_C FROM pg_stat_replication - WHERE application_name = '$appname_C' AND state = 'streaming';" -) or die "Timed out while waiting for apply to restart"; - -############################### -# Test 2PC PREPARE / COMMIT PREPARED. -# 1. Data is streamed as a 2PC transaction. -# 2. Then do commit prepared. -# -# Expect all data is replicated on subscriber(s) after the commit. -############################### - -# Insert, update and delete enough rows to exceed the 64kB limit. -# Then 2PC PREPARE -$node_A->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_A->wait_for_catchup($appname_B); -$node_B->wait_for_catchup($appname_C); +################################ +# Test using streaming mode 'on' +################################ +test_streaming($node_A, $node_B, $node_C, $appname_B, $appname_C, 'on'); -# check the transaction state is prepared on subscriber(s) -$result = - $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber B'); -$result = - $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber C'); - -# 2PC COMMIT -$node_A->safe_psql('postgres', "COMMIT PREPARED 'test_prepared_tab';"); - -$node_A->wait_for_catchup($appname_B); -$node_B->wait_for_catchup($appname_C); - -# check that transaction was committed on subscriber(s) -$result = $node_B->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), - 'Rows inserted by 2PC have committed on subscriber B, and extra columns have local defaults' -); -$result = $node_C->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), - 'Rows inserted by 2PC have committed on subscriber C, and extra columns have local defaults' -); - -# check the transaction state is ended on subscriber(s) -$result = - $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is committed on subscriber B'); -$result = - $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is committed on subscriber C'); - -############################### -# Test 2PC PREPARE with a nested ROLLBACK TO SAVEPOINT. -# 0. Cleanup from previous test leaving only 2 rows. -# 1. Insert one more row. -# 2. Record a SAVEPOINT. -# 3. Data is streamed using 2PC. -# 4. Do rollback to SAVEPOINT prior to the streamed inserts. -# 5. Then COMMIT PREPARED. -# -# Expect data after the SAVEPOINT is aborted leaving only 3 rows (= 2 original + 1 from step 1). -############################### - -# First, delete the data except for 2 rows (delete will be replicated) -$node_A->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); - -# 2PC PREPARE with a nested ROLLBACK TO SAVEPOINT -$node_A->safe_psql( - 'postgres', " - BEGIN; - INSERT INTO test_tab VALUES (9999, 'foobar'); - SAVEPOINT sp_inner; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - ROLLBACK TO SAVEPOINT sp_inner; - PREPARE TRANSACTION 'outer'; - "); - -$node_A->wait_for_catchup($appname_B); -$node_B->wait_for_catchup($appname_C); - -# check the transaction state prepared on subscriber(s) -$result = - $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber B'); -$result = - $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber C'); - -# 2PC COMMIT -$node_A->safe_psql('postgres', "COMMIT PREPARED 'outer';"); - -$node_A->wait_for_catchup($appname_B); -$node_B->wait_for_catchup($appname_C); - -# check the transaction state is ended on subscriber -$result = - $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is ended on subscriber B'); -$result = - $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is ended on subscriber C'); - -# check inserts are visible at subscriber(s). -# All the streamed data (prior to the SAVEPOINT) should be rolled back. -# (9999, 'foobar') should be committed. -$result = $node_B->safe_psql('postgres', - "SELECT count(*) FROM test_tab where b = 'foobar';"); -is($result, qq(1), 'Rows committed are present on subscriber B'); -$result = $node_B->safe_psql('postgres', "SELECT count(*) FROM test_tab;"); -is($result, qq(3), 'Rows committed are present on subscriber B'); -$result = $node_C->safe_psql('postgres', - "SELECT count(*) FROM test_tab where b = 'foobar';"); -is($result, qq(1), 'Rows committed are present on subscriber C'); -$result = $node_C->safe_psql('postgres', "SELECT count(*) FROM test_tab;"); -is($result, qq(3), 'Rows committed are present on subscriber C'); +################################### +# Test using streaming mode 'apply' +################################### +test_streaming($node_A, $node_B, $node_C, $appname_B, $appname_C, 'apply'); ############################### # check all the cleanup diff --git a/src/test/subscription/t/023_twophase_stream.pl b/src/test/subscription/t/023_twophase_stream.pl index d8475d25a4..f55c5c95e7 100644 --- a/src/test/subscription/t/023_twophase_stream.pl +++ b/src/test/subscription/t/023_twophase_stream.pl @@ -8,6 +8,266 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" option +# so the same code can be run both for the streaming=on and streaming=apply +# cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_apply) = @_; + + ############################### + # Test 2PC PREPARE / COMMIT PREPARED. + # 1. Data is streamed as a 2PC transaction. + # 2. Then do commit prepared. + # + # Expect all data is replicated on subscriber side after the commit. + ############################### + + # Check that a background worker starts if "streaming" option is + # specified as "apply". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_apply) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + # check that 2PC gets replicated to subscriber + # Insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + if ($is_apply) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is in prepared state on subscriber + my $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber'); + + # 2PC transaction gets committed + $node_publisher->safe_psql('postgres', + "COMMIT PREPARED 'test_prepared_tab';"); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is committed on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), + 'Rows inserted by 2PC have committed on subscriber, and extra columns contain local defaults' + ); + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is committed on subscriber'); + + ############################### + # Test 2PC PREPARE / ROLLBACK PREPARED. + # 1. Table is deleted back to 2 rows which are replicated on subscriber. + # 2. Data is streamed using 2PC. + # 3. Do rollback prepared. + # + # Expect data rolls back leaving only the original 2 rows. + ############################### + + # First, delete the data except for 2 rows (will be replicated) + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE a > 2;"); + + # Then insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is in prepared state on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber'); + + # 2PC transaction gets aborted + $node_publisher->safe_psql('postgres', + "ROLLBACK PREPARED 'test_prepared_tab';"); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is aborted on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(2|2|2), + 'Rows inserted by 2PC are rolled back, leaving only the original 2 rows' + ); + + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is aborted on subscriber'); + + ############################### + # Check that 2PC COMMIT PREPARED is decoded properly on crash restart. + # 1. insert, update and delete enough rows to exceed the 64kB limit. + # 2. Then server crashes before the 2PC transaction is committed. + # 3. After servers are restarted the pending transaction is committed. + # + # Expect all data is replicated on subscriber side after the commit. + # Note: both publisher and subscriber do crash/restart. + ############################### + + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + $node_subscriber->stop('immediate'); + $node_publisher->stop('immediate'); + + $node_publisher->start; + $node_subscriber->start; + + # commit post the restart + $node_publisher->safe_psql('postgres', + "COMMIT PREPARED 'test_prepared_tab';"); + $node_publisher->wait_for_catchup($appname); + + # check inserts are visible + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), + 'Rows inserted by 2PC have committed on subscriber, and extra columns contain local defaults' + ); + + ############################### + # Do INSERT after the PREPARE but before ROLLBACK PREPARED. + # 1. Table is deleted back to 2 rows which are replicated on subscriber. + # 2. Data is streamed using 2PC. + # 3. A single row INSERT is done which is after the PREPARE. + # 4. Then do a ROLLBACK PREPARED. + # + # Expect the 2PC data rolls back leaving only 3 rows on the subscriber + # (the original 2 + inserted 1). + ############################### + + # First, delete the data except for 2 rows (will be replicated) + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE a > 2;"); + + # Then insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is in prepared state on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber'); + + # Insert a different record (now we are outside of the 2PC transaction) + # Note: the 2PC transaction still holds row locks so make sure this insert is for a separate primary key + $node_publisher->safe_psql('postgres', + "INSERT INTO test_tab VALUES (99999, 'foobar')"); + + # 2PC transaction gets aborted + $node_publisher->safe_psql('postgres', + "ROLLBACK PREPARED 'test_prepared_tab';"); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is aborted on subscriber, + # but the extra INSERT outside of the 2PC still was replicated + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3|3|3), + 'check the outside insert was copied to subscriber'); + + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is aborted on subscriber'); + + ############################### + # Do INSERT after the PREPARE but before COMMIT PREPARED. + # 1. Table is deleted back to 2 rows which are replicated on subscriber. + # 2. Data is streamed using 2PC. + # 3. A single row INSERT is done which is after the PREPARE. + # 4. Then do a COMMIT PREPARED. + # + # Expect 2PC data + the extra row are on the subscriber + # (the 3334 + inserted 1 = 3335). + ############################### + + # First, delete the data except for 2 rows (will be replicated) + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE a > 2;"); + + # Then insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is in prepared state on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber'); + + # Insert a different record (now we are outside of the 2PC transaction) + # Note: the 2PC transaction still holds row locks so make sure this insert is for a separare primary key + $node_publisher->safe_psql('postgres', + "INSERT INTO test_tab VALUES (99999, 'foobar')"); + + # 2PC transaction gets committed + $node_publisher->safe_psql('postgres', + "COMMIT PREPARED 'test_prepared_tab';"); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is committed on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3335|3335|3335), + 'Rows inserted by 2PC (as well as outside insert) have committed on subscriber, and extra columns contain local defaults' + ); + + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is committed on subscriber'); + + # Cleanup the test data + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE a > 2;"); + $node_publisher->wait_for_catchup($appname); +} + ############################### # Setup ############################### @@ -48,6 +308,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql( 'postgres', " CREATE SUBSCRIPTION tap_sub @@ -70,236 +334,30 @@ my $twophase_query = $node_subscriber->poll_query_until('postgres', $twophase_query) or die "Timed out while waiting for subscriber to enable twophase"; -############################### # Check initial data was copied to subscriber -############################### my $result = $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab"); is($result, qq(2|2|2), 'check initial data was copied to subscriber'); -############################### -# Test 2PC PREPARE / COMMIT PREPARED. -# 1. Data is streamed as a 2PC transaction. -# 2. Then do commit prepared. -# -# Expect all data is replicated on subscriber side after the commit. -############################### - -# check that 2PC gets replicated to subscriber -# Insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is in prepared state on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber'); - -# 2PC transaction gets committed -$node_publisher->safe_psql('postgres', - "COMMIT PREPARED 'test_prepared_tab';"); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is committed on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), - 'Rows inserted by 2PC have committed on subscriber, and extra columns contain local defaults' -); -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is committed on subscriber'); - -############################### -# Test 2PC PREPARE / ROLLBACK PREPARED. -# 1. Table is deleted back to 2 rows which are replicated on subscriber. -# 2. Data is streamed using 2PC. -# 3. Do rollback prepared. -# -# Expect data rolls back leaving only the original 2 rows. -############################### - -# First, delete the data except for 2 rows (will be replicated) -$node_publisher->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); - -# Then insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is in prepared state on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber'); - -# 2PC transaction gets aborted -$node_publisher->safe_psql('postgres', - "ROLLBACK PREPARED 'test_prepared_tab';"); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is aborted on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(2|2|2), - 'Rows inserted by 2PC are rolled back, leaving only the original 2 rows'); - -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is aborted on subscriber'); - -############################### -# Check that 2PC COMMIT PREPARED is decoded properly on crash restart. -# 1. insert, update and delete enough rows to exceed the 64kB limit. -# 2. Then server crashes before the 2PC transaction is committed. -# 3. After servers are restarted the pending transaction is committed. -# -# Expect all data is replicated on subscriber side after the commit. -# Note: both publisher and subscriber do crash/restart. -############################### - -$node_publisher->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_subscriber->stop('immediate'); -$node_publisher->stop('immediate'); - -$node_publisher->start; -$node_subscriber->start; - -# commit post the restart -$node_publisher->safe_psql('postgres', - "COMMIT PREPARED 'test_prepared_tab';"); -$node_publisher->wait_for_catchup($appname); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -# check inserts are visible -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), - 'Rows inserted by 2PC have committed on subscriber, and extra columns contain local defaults' +################################### +# Test using streaming mode 'apply' +################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" ); -############################### -# Do INSERT after the PREPARE but before ROLLBACK PREPARED. -# 1. Table is deleted back to 2 rows which are replicated on subscriber. -# 2. Data is streamed using 2PC. -# 3. A single row INSERT is done which is after the PREPARE. -# 4. Then do a ROLLBACK PREPARED. -# -# Expect the 2PC data rolls back leaving only 3 rows on the subscriber -# (the original 2 + inserted 1). -############################### - -# First, delete the data except for 2 rows (will be replicated) -$node_publisher->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); - -# Then insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is in prepared state on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber'); - -# Insert a different record (now we are outside of the 2PC transaction) -# Note: the 2PC transaction still holds row locks so make sure this insert is for a separate primary key -$node_publisher->safe_psql('postgres', - "INSERT INTO test_tab VALUES (99999, 'foobar')"); - -# 2PC transaction gets aborted -$node_publisher->safe_psql('postgres', - "ROLLBACK PREPARED 'test_prepared_tab';"); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is aborted on subscriber, -# but the extra INSERT outside of the 2PC still was replicated -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3|3|3), 'check the outside insert was copied to subscriber'); - -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is aborted on subscriber'); - -############################### -# Do INSERT after the PREPARE but before COMMIT PREPARED. -# 1. Table is deleted back to 2 rows which are replicated on subscriber. -# 2. Data is streamed using 2PC. -# 3. A single row INSERT is done which is after the PREPARE. -# 4. Then do a COMMIT PREPARED. -# -# Expect 2PC data + the extra row are on the subscriber -# (the 3334 + inserted 1 = 3335). -############################### - -# First, delete the data except for 2 rows (will be replicated) -$node_publisher->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); - -# Then insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is in prepared state on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber'); - -# Insert a different record (now we are outside of the 2PC transaction) -# Note: the 2PC transaction still holds row locks so make sure this insert is for a separare primary key -$node_publisher->safe_psql('postgres', - "INSERT INTO test_tab VALUES (99999, 'foobar')"); - -# 2PC transaction gets committed -$node_publisher->safe_psql('postgres', - "COMMIT PREPARED 'test_prepared_tab';"); - -$node_publisher->wait_for_catchup($appname); +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = apply)"); -# check that transaction is committed on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3335|3335|3335), - 'Rows inserted by 2PC (as well as outside insert) have committed on subscriber, and extra columns contain local defaults' -); +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is committed on subscriber'); +test_streaming($node_publisher, $node_subscriber, $appname, 1); ############################### # check all the cleanup -- 2.23.0.windows.1 [application/octet-stream] v10-0003-The-v4-patch-0001-0003-in-thread-1.patch (17.2K, ../../OS3PR01MB62756BD9482EB6BB1CA4CD4D9EAA9@OS3PR01MB6275.jpnprd01.prod.outlook.com/4-v10-0003-The-v4-patch-0001-0003-in-thread-1.patch) download | inline diff: From d5fcde5bcd6cf093376bdc670c39f74936d1919a Mon Sep 17 00:00:00 2001 From: wangw <[email protected]> Date: Tue, 14 Jun 2022 11:21:25 +0800 Subject: [PATCH v10 3/5] The v4 patch(0001~0003) in thread [1] [1] - https://www.postgresql.org/message-id/OS0PR01MB5716377C85D4A164E6A7D45F94AB9%40OS0PR01MB5716.jpnprd01.prod.outlook.com --- src/backend/replication/logical/relation.c | 226 +++++++++++++-------- src/backend/replication/logical/worker.c | 25 ++- src/include/replication/logicalrelation.h | 1 + src/test/subscription/t/013_partition.pl | 85 +++++++- 4 files changed, 242 insertions(+), 95 deletions(-) diff --git a/src/backend/replication/logical/relation.c b/src/backend/replication/logical/relation.c index 80fb561a9a..f342396310 100644 --- a/src/backend/replication/logical/relation.c +++ b/src/backend/replication/logical/relation.c @@ -249,6 +249,72 @@ logicalrep_report_missing_attrs(LogicalRepRelation *remoterel, } } +/* + * Check if replica identity matches and mark the updatable flag. + * + * We allow for stricter replica identity (fewer columns) on subscriber as + * that will not stop us from finding unique tuple. IE, if publisher has + * identity (id,timestamp) and subscriber just (id) this will not be a + * problem, but in the opposite scenario it will. + * + * Don't throw any error here just mark the relation entry as not updatable, + * as replica identity is only for updates and deletes but inserts can be + * replicated even without it. + */ +static void +logicalrep_rel_mark_updatable(LogicalRepRelMapEntry *entry) +{ + Bitmapset *idkey; + LogicalRepRelation *remoterel = &entry->remoterel; + int i; + + entry->updatable = true; + + /* + * If it is a partitioned table, we don't check it, we will check its + * partition later. + */ + if (entry->localrel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + return; + + idkey = RelationGetIndexAttrBitmap(entry->localrel, + INDEX_ATTR_BITMAP_IDENTITY_KEY); + /* fallback to PK if no replica identity */ + if (idkey == NULL) + { + idkey = RelationGetIndexAttrBitmap(entry->localrel, + INDEX_ATTR_BITMAP_PRIMARY_KEY); + /* + * If no replica identity index and no PK, the published table + * must have replica identity FULL. + */ + if (idkey == NULL && remoterel->replident != REPLICA_IDENTITY_FULL) + entry->updatable = false; + } + + i = -1; + while ((i = bms_next_member(idkey, i)) >= 0) + { + int attnum = i + FirstLowInvalidHeapAttributeNumber; + + if (!AttrNumberIsForUserDefinedAttr(attnum)) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("logical replication target relation \"%s.%s\" uses " + "system columns in REPLICA IDENTITY index", + remoterel->nspname, remoterel->relname))); + + attnum = AttrNumberGetAttrOffset(attnum); + + if (entry->attrmap->attnums[attnum] < 0 || + !bms_is_member(entry->attrmap->attnums[attnum], remoterel->attkeys)) + { + entry->updatable = false; + break; + } + } +} + /* * Open the local relation associated with the remote one. * @@ -307,7 +373,6 @@ logicalrep_rel_open(LogicalRepRelId remoteid, LOCKMODE lockmode) if (!entry->localrelvalid) { Oid relid; - Bitmapset *idkey; TupleDesc desc; MemoryContext oldctx; int i; @@ -365,55 +430,8 @@ logicalrep_rel_open(LogicalRepRelId remoteid, LOCKMODE lockmode) /* be tidy */ bms_free(missingatts); - /* - * Check that replica identity matches. We allow for stricter replica - * identity (fewer columns) on subscriber as that will not stop us - * from finding unique tuple. IE, if publisher has identity - * (id,timestamp) and subscriber just (id) this will not be a problem, - * but in the opposite scenario it will. - * - * Don't throw any error here just mark the relation entry as not - * updatable, as replica identity is only for updates and deletes but - * inserts can be replicated even without it. - */ - entry->updatable = true; - idkey = RelationGetIndexAttrBitmap(entry->localrel, - INDEX_ATTR_BITMAP_IDENTITY_KEY); - /* fallback to PK if no replica identity */ - if (idkey == NULL) - { - idkey = RelationGetIndexAttrBitmap(entry->localrel, - INDEX_ATTR_BITMAP_PRIMARY_KEY); - - /* - * If no replica identity index and no PK, the published table - * must have replica identity FULL. - */ - if (idkey == NULL && remoterel->replident != REPLICA_IDENTITY_FULL) - entry->updatable = false; - } - - i = -1; - while ((i = bms_next_member(idkey, i)) >= 0) - { - int attnum = i + FirstLowInvalidHeapAttributeNumber; - - if (!AttrNumberIsForUserDefinedAttr(attnum)) - ereport(ERROR, - (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), - errmsg("logical replication target relation \"%s.%s\" uses " - "system columns in REPLICA IDENTITY index", - remoterel->nspname, remoterel->relname))); - - attnum = AttrNumberGetAttrOffset(attnum); - - if (entry->attrmap->attnums[attnum] < 0 || - !bms_is_member(entry->attrmap->attnums[attnum], remoterel->attkeys)) - { - entry->updatable = false; - break; - } - } + /* Check that replica identity matches. */ + logicalrep_rel_mark_updatable(entry); entry->localrelvalid = true; } @@ -436,22 +454,13 @@ logicalrep_rel_close(LogicalRepRelMapEntry *rel, LOCKMODE lockmode) rel->localrel = NULL; } -/* - * Partition cache: look up partition LogicalRepRelMapEntry's - * - * Unlike relation map cache, this is keyed by partition OID, not remote - * relation OID, because we only have to use this cache in the case where - * partitions are not directly mapped to any remote relation, such as when - * replication is occurring with one of their ancestors as target. - */ - /* * Relcache invalidation callback */ static void logicalrep_partmap_invalidate_cb(Datum arg, Oid reloid) { - LogicalRepRelMapEntry *entry; + LogicalRepPartMapEntry *entry; /* Just to be sure. */ if (LogicalRepPartMap == NULL) @@ -464,11 +473,11 @@ logicalrep_partmap_invalidate_cb(Datum arg, Oid reloid) hash_seq_init(&status, LogicalRepPartMap); /* TODO, use inverse lookup hashtable? */ - while ((entry = (LogicalRepRelMapEntry *) hash_seq_search(&status)) != NULL) + while ((entry = (LogicalRepPartMapEntry *) hash_seq_search(&status)) != NULL) { - if (entry->localreloid == reloid) + if (entry->relmapentry.localreloid == reloid) { - entry->localrelvalid = false; + entry->relmapentry.localrelvalid = false; hash_seq_term(&status); break; } @@ -481,8 +490,42 @@ logicalrep_partmap_invalidate_cb(Datum arg, Oid reloid) hash_seq_init(&status, LogicalRepPartMap); - while ((entry = (LogicalRepRelMapEntry *) hash_seq_search(&status)) != NULL) - entry->localrelvalid = false; + while ((entry = (LogicalRepPartMapEntry *) hash_seq_search(&status)) != NULL) + entry->relmapentry.localrelvalid = false; + } +} + +/* + * Reset the entries in the partition map that refer to remoterel + * + * Called when new relation mapping is sent by the publisher to update our + * expected view of incoming data from said publisher. + * + * Note that we don't update the remoterel information in the entry here, + * we will update the information in logicalrep_partition_open to avoid + * unnecessary work. + */ +void +logicalrep_partmap_reset_relmap(LogicalRepRelation *remoterel) +{ + HASH_SEQ_STATUS status; + LogicalRepPartMapEntry *part_entry; + LogicalRepRelMapEntry *entry; + + if (LogicalRepPartMap == NULL) + return; + + hash_seq_init(&status, LogicalRepPartMap); + while ((part_entry = (LogicalRepPartMapEntry *) hash_seq_search(&status)) != NULL) + { + entry = &part_entry->relmapentry; + + if (entry->remoterel.remoteid != remoterel->remoteid) + continue; + + logicalrep_relmap_free_entry(entry); + + memset(entry, 0, sizeof(LogicalRepRelMapEntry)); } } @@ -534,7 +577,6 @@ logicalrep_partition_open(LogicalRepRelMapEntry *root, Oid partOid = RelationGetRelid(partrel); AttrMap *attrmap = root->attrmap; bool found; - int i; MemoryContext oldctx; if (LogicalRepPartMap == NULL) @@ -545,31 +587,40 @@ logicalrep_partition_open(LogicalRepRelMapEntry *root, (void *) &partOid, HASH_ENTER, &found); - if (found) - return &part_entry->relmapentry; + entry = &part_entry->relmapentry; - memset(part_entry, 0, sizeof(LogicalRepPartMapEntry)); + if (found && entry->localrelvalid) + return entry; /* Switch to longer-lived context. */ oldctx = MemoryContextSwitchTo(LogicalRepPartMapContext); - part_entry->partoid = partOid; + if (!found) + { + memset(part_entry, 0, sizeof(LogicalRepPartMapEntry)); + part_entry->partoid = partOid; + } - /* Remote relation is copied as-is from the root entry. */ - entry = &part_entry->relmapentry; - entry->remoterel.remoteid = remoterel->remoteid; - entry->remoterel.nspname = pstrdup(remoterel->nspname); - entry->remoterel.relname = pstrdup(remoterel->relname); - entry->remoterel.natts = remoterel->natts; - entry->remoterel.attnames = palloc(remoterel->natts * sizeof(char *)); - entry->remoterel.atttyps = palloc(remoterel->natts * sizeof(Oid)); - for (i = 0; i < remoterel->natts; i++) + if (!entry->remoterel.remoteid) { - entry->remoterel.attnames[i] = pstrdup(remoterel->attnames[i]); - entry->remoterel.atttyps[i] = remoterel->atttyps[i]; + int i; + + /* Remote relation is copied as-is from the root entry. */ + entry = &part_entry->relmapentry; + entry->remoterel.remoteid = remoterel->remoteid; + entry->remoterel.nspname = pstrdup(remoterel->nspname); + entry->remoterel.relname = pstrdup(remoterel->relname); + entry->remoterel.natts = remoterel->natts; + entry->remoterel.attnames = palloc(remoterel->natts * sizeof(char *)); + entry->remoterel.atttyps = palloc(remoterel->natts * sizeof(Oid)); + for (i = 0; i < remoterel->natts; i++) + { + entry->remoterel.attnames[i] = pstrdup(remoterel->attnames[i]); + entry->remoterel.atttyps[i] = remoterel->atttyps[i]; + } + entry->remoterel.replident = remoterel->replident; + entry->remoterel.attkeys = bms_copy(remoterel->attkeys); } - entry->remoterel.replident = remoterel->replident; - entry->remoterel.attkeys = bms_copy(remoterel->attkeys); entry->localrel = partrel; entry->localreloid = partOid; @@ -594,7 +645,11 @@ logicalrep_partition_open(LogicalRepRelMapEntry *root, { AttrNumber root_attno = map->attnums[attno]; - entry->attrmap->attnums[attno] = attrmap->attnums[root_attno - 1]; + /* 0 means it's a dropped attribute */ + if (root_attno == 0) + entry->attrmap->attnums[attno] = -1; + else + entry->attrmap->attnums[attno] = attrmap->attnums[root_attno - 1]; } } else @@ -605,7 +660,8 @@ logicalrep_partition_open(LogicalRepRelMapEntry *root, attrmap->maplen * sizeof(AttrNumber)); } - entry->updatable = root->updatable; + /* Check that replica identity matches. */ + logicalrep_rel_mark_updatable(entry); entry->localrelvalid = true; diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c index 3f8dea0fc1..4e3bcf7c28 100644 --- a/src/backend/replication/logical/worker.c +++ b/src/backend/replication/logical/worker.c @@ -1803,6 +1803,11 @@ apply_handle_relation(StringInfo s) rel = logicalrep_read_rel(s); logicalrep_relmap_update(rel); + + /* + * Also reset all entries in the partition map that refer to remoterel. + */ + logicalrep_partmap_reset_relmap(rel); } /* @@ -2359,6 +2364,8 @@ apply_handle_tuple_routing(ApplyExecutionData *edata, TupleTableSlot *remoteslot_part; TupleConversionMap *map; MemoryContext oldctx; + LogicalRepRelMapEntry *part_entry; + AttrMap *attrmap = NULL; /* ModifyTableState is needed for ExecFindPartition(). */ edata->mtstate = mtstate = makeNode(ModifyTableState); @@ -2390,8 +2397,11 @@ apply_handle_tuple_routing(ApplyExecutionData *edata, remoteslot_part = table_slot_create(partrel, &estate->es_tupleTable); map = partrelinfo->ri_RootToPartitionMap; if (map != NULL) - remoteslot_part = execute_attr_map_slot(map->attrMap, remoteslot, + { + attrmap = map->attrMap; + remoteslot_part = execute_attr_map_slot(attrmap, remoteslot, remoteslot_part); + } else { remoteslot_part = ExecCopySlot(remoteslot_part, remoteslot); @@ -2399,6 +2409,14 @@ apply_handle_tuple_routing(ApplyExecutionData *edata, } MemoryContextSwitchTo(oldctx); + /* Check if we can do the update or delete on the leaf partition. */ + if(operation == CMD_UPDATE || operation == CMD_DELETE) + { + part_entry = logicalrep_partition_open(relmapentry, partrel, + attrmap); + check_relation_updatable(part_entry); + } + switch (operation) { case CMD_INSERT: @@ -2420,15 +2438,10 @@ apply_handle_tuple_routing(ApplyExecutionData *edata, * suitable partition. */ { - AttrMap *attrmap = map ? map->attrMap : NULL; - LogicalRepRelMapEntry *part_entry; TupleTableSlot *localslot; ResultRelInfo *partrelinfo_new; bool found; - part_entry = logicalrep_partition_open(relmapentry, partrel, - attrmap); - /* Get the matching local tuple from the partition. */ found = FindReplTupleInLocalRel(estate, partrel, &part_entry->remoterel, diff --git a/src/include/replication/logicalrelation.h b/src/include/replication/logicalrelation.h index 7bf8cd22bd..78cd7e77f5 100644 --- a/src/include/replication/logicalrelation.h +++ b/src/include/replication/logicalrelation.h @@ -38,6 +38,7 @@ typedef struct LogicalRepRelMapEntry } LogicalRepRelMapEntry; extern void logicalrep_relmap_update(LogicalRepRelation *remoterel); +extern void logicalrep_partmap_reset_relmap(LogicalRepRelation *remoterel); extern LogicalRepRelMapEntry *logicalrep_rel_open(LogicalRepRelId remoteid, LOCKMODE lockmode); diff --git a/src/test/subscription/t/013_partition.pl b/src/test/subscription/t/013_partition.pl index e7f4a94f19..df897c86db 100644 --- a/src/test/subscription/t/013_partition.pl +++ b/src/test/subscription/t/013_partition.pl @@ -800,9 +800,86 @@ ok( $logfile =~ qr/logical replication did not find row to be deleted in replication target relation "tab2_1"/, 'delete target row is missing in tab2_1'); -# No need for this until more tests are added. -# $node_subscriber1->append_conf('postgresql.conf', -# "log_min_messages = warning"); -# $node_subscriber1->reload; +$node_subscriber1->append_conf('postgresql.conf', + "log_min_messages = warning"); +$node_subscriber1->reload; + +# Test the case that target table on subscriber is a partitioned table and +# check that the changes are replicated correctly after changing the schema of +# table on subcriber. + +$node_publisher->safe_psql( + 'postgres', q{ + CREATE TABLE tab5 (a int NOT NULL, b int); + CREATE UNIQUE INDEX tab5_a_idx ON tab5 (a); + ALTER TABLE tab5 REPLICA IDENTITY USING INDEX tab5_a_idx;}); + +$node_subscriber2->safe_psql( + 'postgres', q{ + CREATE TABLE tab5 (a int NOT NULL, b int, c int) PARTITION BY LIST (a); + CREATE TABLE tab5_1 PARTITION OF tab5 DEFAULT; + CREATE UNIQUE INDEX tab5_a_idx ON tab5 (a); + ALTER TABLE tab5 REPLICA IDENTITY USING INDEX tab5_a_idx; + ALTER TABLE tab5_1 REPLICA IDENTITY USING INDEX tab5_1_a_idx;}); + +$node_subscriber2->safe_psql('postgres', + "ALTER SUBSCRIPTION sub2 REFRESH PUBLICATION"); + +$node_subscriber2->poll_query_until('postgres', $synced_query) + or die "Timed out while waiting for subscriber to synchronize data"; + +# Make partition map cache +$node_publisher->safe_psql('postgres', "INSERT INTO tab5 VALUES (1, 1)"); +$node_publisher->safe_psql('postgres', "UPDATE tab5 SET a = 2 WHERE a = 1"); + +$node_publisher->wait_for_catchup('sub2'); + +$result = $node_subscriber2->safe_psql('postgres', + "SELECT a, b FROM tab5 ORDER BY 1"); +is($result, qq(2|1), 'updates of tab5 replicated correctly'); + +# Change the column order of partition on subscriber +$node_subscriber2->safe_psql( + 'postgres', q{ + ALTER TABLE tab5 DETACH PARTITION tab5_1; + ALTER TABLE tab5_1 DROP COLUMN b; + ALTER TABLE tab5_1 ADD COLUMN b int; + ALTER TABLE tab5 ATTACH PARTITION tab5_1 DEFAULT}); + +$node_publisher->safe_psql('postgres', "UPDATE tab5 SET a = 3 WHERE a = 2"); + +$node_publisher->wait_for_catchup('sub2'); + +$result = $node_subscriber2->safe_psql('postgres', + "SELECT a, b, c FROM tab5 ORDER BY 1"); +is($result, qq(3|1|), 'updates of tab5 replicated correctly after altering table on subscriber'); + +# Change the column order of table on publisher +$node_publisher->safe_psql( + 'postgres', q{ + ALTER TABLE tab5 DROP COLUMN b, ADD COLUMN c INT; + ALTER TABLE tab5 ADD COLUMN b INT;}); + +$node_publisher->safe_psql('postgres', "UPDATE tab5 SET c = 1 WHERE a = 3"); + +$node_publisher->wait_for_catchup('sub2'); + +$result = $node_subscriber2->safe_psql('postgres', + "SELECT a, b, c FROM tab5 ORDER BY 1"); +is($result, qq(3||1), 'updates of tab5 replicated correctly after altering table on publisher'); + +# Alter REPLICA IDENTITY on subscriber. +# No REPLICA IDENTITY in the partitioned table on subscriber, but what we check +# is the partition, so it works fine. +$node_subscriber2->safe_psql('postgres', + "ALTER TABLE tab5 REPLICA IDENTITY NOTHING"); + +$node_publisher->safe_psql('postgres', "UPDATE tab5 SET a = 4 WHERE a = 3"); + +$node_publisher->wait_for_catchup('sub2'); + +$result = $node_subscriber2->safe_psql('postgres', + "SELECT a, b, c FROM tab5_1 ORDER BY 1"); +is($result, qq(4||1), 'updates of tab5 replicated correctly'); done_testing(); -- 2.23.0.windows.1 [application/octet-stream] v10-0004-Add-some-checks-before-using-apply-background-wo.patch (29.1K, ../../OS3PR01MB62756BD9482EB6BB1CA4CD4D9EAA9@OS3PR01MB6275.jpnprd01.prod.outlook.com/5-v10-0004-Add-some-checks-before-using-apply-background-wo.patch) download | inline diff: From 1a55a4e0bb1789a91732a872015f17493032a33f Mon Sep 17 00:00:00 2001 From: wangw <[email protected]> Date: Tue, 14 Jun 2022 11:23:52 +0800 Subject: [PATCH v10 4/5] Add some checks before using apply background worker to apply changes. If any of the following checks are violated, an error will be reported. 1. The unique columns between publisher and subscriber are difference. 2. There is any non-immutable function present in expression in subscriber's relation. Check from the following 3 items: a. The function in triggers; b. Column default value expressions and domain constraints; c. Constraint expressions. --- doc/src/sgml/ref/create_subscription.sgml | 4 + .../replication/logical/applybgwroker.c | 45 ++++ src/backend/replication/logical/proto.c | 62 ++++- src/backend/replication/logical/relation.c | 198 +++++++++++++++ src/backend/replication/logical/tablesync.c | 1 + src/backend/replication/logical/worker.c | 15 +- src/backend/utils/cache/typcache.c | 17 ++ src/include/replication/logicalproto.h | 1 + src/include/replication/logicalrelation.h | 15 ++ src/include/replication/worker_internal.h | 1 + src/include/utils/typcache.h | 2 + .../subscription/t/022_twophase_cascade.pl | 7 + .../subscription/t/032_streaming_apply.pl | 238 ++++++++++++++++++ 13 files changed, 598 insertions(+), 8 deletions(-) create mode 100644 src/test/subscription/t/032_streaming_apply.pl diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml index d4caae0222..8de1a23ce4 100644 --- a/doc/src/sgml/ref/create_subscription.sgml +++ b/doc/src/sgml/ref/create_subscription.sgml @@ -240,6 +240,10 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl the transaction is committed. Note that if an error happens when applying changes in a background worker, it might not report the finish LSN of the remote transaction in the server log. + To run in this mode, there are following two requirements. The first + is that the unique column should be the same between publisher and + subscriber; the second is that there should not be any non-immutable + function in subscriber-side replicated table. </para> </listitem> </varlistentry> diff --git a/src/backend/replication/logical/applybgwroker.c b/src/backend/replication/logical/applybgwroker.c index 1f52b155d7..7d3c61c2e9 100644 --- a/src/backend/replication/logical/applybgwroker.c +++ b/src/backend/replication/logical/applybgwroker.c @@ -763,3 +763,48 @@ apply_bgworker_subxact_info_add(TransactionId current_xid) MemoryContextSwitchTo(oldctx); } } + +/* + * Check if changes on this logical replication relation can be applied by + * apply background worker. + * + * Although we maintains the commit order by allowing only one process to + * commit at a time, our access order to the relation has changed. + * This could cause unexpected problems if the unique column on the replicated + * table is inconsistent with the publisher-side or contains non-immutable + * functions when applying transactions in the apply background worker. + */ +void +apply_bgworker_relation_check(LogicalRepRelMapEntry *rel) +{ + /* Check only we are in apply bgworker. */ + if (!am_apply_bgworker()) + return; + + /* + * If it is a partitioned table, we do not check it, we will check its + * partition later. + */ + if (rel->localrel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + return; + + /* + * If any unique index exist, check that they are same as remoterel. + */ + if (!rel->sameunique) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot replicate relation with different unique index"), + errhint("Please change the streaming option to 'on' instead of 'apply'."))); + + /* + * Check if there is any non-immutable function present in expression in + * this relation. + */ + if (rel->volatility == FUNCTION_NONIMMUTABLE) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot replicate relation. There is at least one non-immutable function"), + errhint("Please change the streaming option to 'on' instead of 'apply'."))); + +} diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c index a888038eb4..6af3302270 100644 --- a/src/backend/replication/logical/proto.c +++ b/src/backend/replication/logical/proto.c @@ -23,7 +23,8 @@ /* * Protocol message flags. */ -#define LOGICALREP_IS_REPLICA_IDENTITY 1 +#define LOGICALREP_IS_REPLICA_IDENTITY 0x0001 +#define LOGICALREP_IS_UNIQUE 0x0002 #define MESSAGE_TRANSACTIONAL (1<<0) #define TRUNCATE_CASCADE (1<<0) @@ -933,11 +934,55 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns) TupleDesc desc; int i; uint16 nliveatts = 0; - Bitmapset *idattrs = NULL; + Bitmapset *idattrs = NULL, + *attunique = NULL; bool replidentfull; desc = RelationGetDescr(rel); + if (rel->rd_rel->relhasindex) + { + List *indexoidlist = RelationGetIndexList(rel); + ListCell *indexoidscan; + + foreach(indexoidscan, indexoidlist) + { + Oid indexoid = lfirst_oid(indexoidscan); + Relation indexRel; + + /* Look up the description for index */ + indexRel = RelationIdGetRelation(indexoid); + + if (!RelationIsValid(indexRel)) + elog(ERROR, "could not open relation with OID %u", indexoid); + + if (indexRel->rd_index->indisunique) + { + int i; + + /* Add referenced attributes to idindexattrs */ + for (i = 0; i < indexRel->rd_index->indnatts; i++) + { + int attrnum = indexRel->rd_index->indkey.values[i]; + + /* + * We don't include non-key columns into idindexattrs + * bitmaps. See RelationGetIndexAttrBitmap. + */ + if (attrnum != 0) + { + if (i < indexRel->rd_index->indnkeyatts && + !bms_is_member(attrnum - FirstLowInvalidHeapAttributeNumber, attunique)) + attunique = bms_add_member(attunique, + attrnum - FirstLowInvalidHeapAttributeNumber); + } + } + } + RelationClose(indexRel); + } + list_free(indexoidlist); + } + /* send number of live attributes */ for (i = 0; i < desc->natts; i++) { @@ -976,6 +1021,10 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns) idattrs)) flags |= LOGICALREP_IS_REPLICA_IDENTITY; + if (bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber, + attunique)) + flags |= LOGICALREP_IS_UNIQUE; + pq_sendbyte(out, flags); /* attribute name */ @@ -1001,7 +1050,8 @@ logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel) int natts; char **attnames; Oid *atttyps; - Bitmapset *attkeys = NULL; + Bitmapset *attkeys = NULL, + *attunique = NULL; natts = pq_getmsgint(in, 2); attnames = palloc(natts * sizeof(char *)); @@ -1012,11 +1062,14 @@ logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel) { uint8 flags; - /* Check for replica identity column */ + /* Check for replica identity and unique column */ flags = pq_getmsgbyte(in); if (flags & LOGICALREP_IS_REPLICA_IDENTITY) attkeys = bms_add_member(attkeys, i); + if (flags & LOGICALREP_IS_UNIQUE) + attunique = bms_add_member(attunique, i); + /* attribute name */ attnames[i] = pstrdup(pq_getmsgstring(in)); @@ -1030,6 +1083,7 @@ logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel) rel->attnames = attnames; rel->atttyps = atttyps; rel->attkeys = attkeys; + rel->attunique = attunique; rel->natts = natts; } diff --git a/src/backend/replication/logical/relation.c b/src/backend/replication/logical/relation.c index f342396310..03fbc61209 100644 --- a/src/backend/replication/logical/relation.c +++ b/src/backend/replication/logical/relation.c @@ -19,12 +19,19 @@ #include "access/table.h" #include "catalog/namespace.h" +#include "catalog/pg_proc.h" #include "catalog/pg_subscription_rel.h" +#include "commands/trigger.h" #include "executor/executor.h" #include "nodes/makefuncs.h" +#include "optimizer/optimizer.h" #include "replication/logicalrelation.h" #include "replication/worker_internal.h" +#include "rewrite/rewriteHandler.h" #include "utils/inval.h" +#include "utils/lsyscache.h" +#include "utils/syscache.h" +#include "utils/typcache.h" static MemoryContext LogicalRepRelMapContext = NULL; @@ -91,6 +98,23 @@ logicalrep_relmap_invalidate_cb(Datum arg, Oid reloid) } } +/* + * Reset the flag volatility of all existing entry in the relation map cache. + */ +static void +logicalrep_relmap_reset_volatility_cb(Datum arg, int cacheid, uint32 hashvalue) +{ + HASH_SEQ_STATUS hash_seq; + LogicalRepRelMapEntry *entry; + + if (LogicalRepRelMap == NULL) + return; + + hash_seq_init(&hash_seq, LogicalRepRelMap); + while ((entry = hash_seq_search(&hash_seq)) != NULL) + entry->volatility = FUNCTION_UNKNOWN; +} + /* * Initialize the relation map cache. */ @@ -116,6 +140,9 @@ logicalrep_relmap_init(void) /* Watch for invalidation events. */ CacheRegisterRelcacheCallback(logicalrep_relmap_invalidate_cb, (Datum) 0); + CacheRegisterSyscacheCallback(PROCOID, + logicalrep_relmap_reset_volatility_cb, + (Datum) 0); } /* @@ -142,6 +169,7 @@ logicalrep_relmap_free_entry(LogicalRepRelMapEntry *entry) pfree(remoterel->atttyps); } bms_free(remoterel->attkeys); + bms_free(remoterel->attunique); if (entry->attrmap) pfree(entry->attrmap); @@ -190,6 +218,7 @@ logicalrep_relmap_update(LogicalRepRelation *remoterel) } entry->remoterel.replident = remoterel->replident; entry->remoterel.attkeys = bms_copy(remoterel->attkeys); + entry->remoterel.attunique = bms_copy(remoterel->attunique); MemoryContextSwitchTo(oldctx); } @@ -315,6 +344,162 @@ logicalrep_rel_mark_updatable(LogicalRepRelMapEntry *entry) } } +/* + * Helper function to mark flags for apply background worker when opening the + * local table. + */ +static void +logicalrep_rel_mark_apply_bgworker(LogicalRepRelMapEntry *entry) +{ + Bitmapset *ukey; + int i; + TupleDesc tupdesc; + int attnum; + List *fkeys = NIL; + + /* + * Check whether the unique index of publisher and subscriber are + * consistent. + */ + entry->sameunique = true; + ukey = RelationGetIndexAttrBitmap(entry->localrel, + INDEX_ATTR_BITMAP_KEY); + + if (ukey) + { + i = -1; + while ((i = bms_next_member(ukey, i)) >= 0) + { + int attnum = i + FirstLowInvalidHeapAttributeNumber; + + attnum = AttrNumberGetAttrOffset(attnum); + + if (entry->attrmap->attnums[attnum] < 0 || + !bms_is_member(entry->attrmap->attnums[attnum], entry->remoterel.attunique)) + { + entry->sameunique = false; + break; + } + } + } + + /* + * Check whether there is any non-immutable function in the local table. + * + * a. The function in triggers; + * b. Column default value expressions and domain constraints; + * c. Constraint expressions; + * d. Foreign keys. + */ + if (entry->volatility != FUNCTION_UNKNOWN) + return; + + /* Initialize the flag. */ + entry->volatility = FUNCTION_IMMUTABLE; + + /* Check the trigger functions. */ + if (entry->localrel->trigdesc != NULL) + { + for (i = 0; i < entry->localrel->trigdesc->numtriggers; i++) + { + Trigger *trig = entry->localrel->trigdesc->triggers + i; + + if (trig->tgenabled != TRIGGER_FIRES_ALWAYS && + trig->tgenabled != TRIGGER_FIRES_ON_REPLICA) + continue; + + if (func_volatile(trig->tgfoid) != PROVOLATILE_IMMUTABLE) + { + entry->volatility = FUNCTION_NONIMMUTABLE; + return; + } + } + } + + /* Check the columns. */ + tupdesc = RelationGetDescr(entry->localrel); + for (attnum = 0; attnum < tupdesc->natts; attnum++) + { + Form_pg_attribute att = TupleDescAttr(tupdesc, attnum); + + /* We don't need info for dropped or generated attributes */ + if (att->attisdropped || att->attgenerated) + continue; + + /* + * We don't need to check columns that only exist on the + * subscriber + */ + if (entry->attrmap->attnums[attnum] < 0) + continue; + + if (att->atthasdef) + { + Node *defaultexpr; + + defaultexpr = build_column_default(entry->localrel, attnum + 1); + if (contain_mutable_functions(defaultexpr)) + { + entry->volatility = FUNCTION_NONIMMUTABLE; + return; + } + } + + /* + * If the column is of a DOMAIN type, determine whether + * that domain has any CHECK expressions that are not + * immutable. + */ + if (get_typtype(att->atttypid) == TYPTYPE_DOMAIN) + { + List *domain_constraints; + ListCell *lc; + + domain_constraints = GetDomainConstraints(att->atttypid); + + foreach(lc, domain_constraints) + { + DomainConstraintState *con = (DomainConstraintState *) lfirst(lc); + + if (con->check_expr && contain_mutable_functions((Node *) con->check_expr)) + { + entry->volatility = FUNCTION_NONIMMUTABLE; + return; + } + } + } + } + + + /* Check the constraints. */ + tupdesc = RelationGetDescr(entry->localrel); + + if (tupdesc->constr) + { + ConstrCheck *check = tupdesc->constr->check; + + /* + * Determine if there are any CHECK constraints which + * contains non-immutable function. + */ + for (i = 0; i < tupdesc->constr->num_check; i++) + { + Expr *check_expr = stringToNode(check[i].ccbin); + + if (contain_mutable_functions((Node *) check_expr)) + { + entry->volatility = FUNCTION_NONIMMUTABLE; + return; + } + } + } + + /* Check the foreign keys. */ + fkeys = RelationGetFKeyList(entry->localrel); + if (fkeys) + entry->volatility = FUNCTION_NONIMMUTABLE; +} + /* * Open the local relation associated with the remote one. * @@ -433,6 +618,12 @@ logicalrep_rel_open(LogicalRepRelId remoteid, LOCKMODE lockmode) /* Check that replica identity matches. */ logicalrep_rel_mark_updatable(entry); + /* + * Set flags to check later whether changes could be applied in the + * apply background worker. + */ + logicalrep_rel_mark_apply_bgworker(entry); + entry->localrelvalid = true; } @@ -620,6 +811,7 @@ logicalrep_partition_open(LogicalRepRelMapEntry *root, } entry->remoterel.replident = remoterel->replident; entry->remoterel.attkeys = bms_copy(remoterel->attkeys); + entry->remoterel.attunique = bms_copy(remoterel->attunique); } entry->localrel = partrel; @@ -663,6 +855,12 @@ logicalrep_partition_open(LogicalRepRelMapEntry *root, /* Check that replica identity matches. */ logicalrep_rel_mark_updatable(entry); + /* + * Set flags to check later whether changes could be applied in the apply + * background worker. + */ + logicalrep_rel_mark_apply_bgworker(entry); + entry->localrelvalid = true; /* state and statelsn are left set to 0. */ diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c index 8ffba7e2e5..3cdbf8b457 100644 --- a/src/backend/replication/logical/tablesync.c +++ b/src/backend/replication/logical/tablesync.c @@ -884,6 +884,7 @@ fetch_remote_table_info(char *nspname, char *relname, lrel->attnames = palloc0(MaxTupleAttributeNumber * sizeof(char *)); lrel->atttyps = palloc0(MaxTupleAttributeNumber * sizeof(Oid)); lrel->attkeys = NULL; + lrel->attunique = NULL; /* * Store the columns as a list of names. Ignore those that are not diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c index 4e3bcf7c28..1b83a74685 100644 --- a/src/backend/replication/logical/worker.c +++ b/src/backend/replication/logical/worker.c @@ -1920,6 +1920,8 @@ apply_handle_insert(StringInfo s) /* Set relation for error callback */ apply_error_callback_arg.rel = rel; + apply_bgworker_relation_check(rel); + /* Initialize the executor state. */ edata = create_edata_for_relation(rel); estate = edata->estate; @@ -2056,6 +2058,8 @@ apply_handle_update(StringInfo s) /* Check if we can do the update. */ check_relation_updatable(rel); + apply_bgworker_relation_check(rel); + /* Initialize the executor state. */ edata = create_edata_for_relation(rel); estate = edata->estate; @@ -2224,6 +2228,8 @@ apply_handle_delete(StringInfo s) /* Check if we can do the delete. */ check_relation_updatable(rel); + apply_bgworker_relation_check(rel); + /* Initialize the executor state. */ edata = create_edata_for_relation(rel); estate = edata->estate; @@ -2409,13 +2415,14 @@ apply_handle_tuple_routing(ApplyExecutionData *edata, } MemoryContextSwitchTo(oldctx); + part_entry = logicalrep_partition_open(relmapentry, partrel, + attrmap); + + apply_bgworker_relation_check(part_entry); + /* Check if we can do the update or delete on the leaf partition. */ if(operation == CMD_UPDATE || operation == CMD_DELETE) - { - part_entry = logicalrep_partition_open(relmapentry, partrel, - attrmap); check_relation_updatable(part_entry); - } switch (operation) { diff --git a/src/backend/utils/cache/typcache.c b/src/backend/utils/cache/typcache.c index 808f9ebd0d..b248899d82 100644 --- a/src/backend/utils/cache/typcache.c +++ b/src/backend/utils/cache/typcache.c @@ -2540,6 +2540,23 @@ compare_values_of_enum(TypeCacheEntry *tcache, Oid arg1, Oid arg2) return 0; } +/* + * GetDomainConstraints --- get DomainConstraintState list of specified domain type + */ +List * +GetDomainConstraints(Oid type_id) +{ + TypeCacheEntry *typentry; + List *constraints = NIL; + + typentry = lookup_type_cache(type_id, TYPECACHE_DOMAIN_CONSTR_INFO); + + if(typentry->domainData != NULL) + constraints = typentry->domainData->constraints; + + return constraints; +} + /* * Load (or re-load) the enumData member of the typcache entry. */ diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h index 7bba77c9e7..a503eb62c2 100644 --- a/src/include/replication/logicalproto.h +++ b/src/include/replication/logicalproto.h @@ -108,6 +108,7 @@ typedef struct LogicalRepRelation char replident; /* replica identity */ char relkind; /* remote relation kind */ Bitmapset *attkeys; /* Bitmap of key columns */ + Bitmapset *attunique; /* Bitmap of unique columns */ } LogicalRepRelation; /* Type mapping info */ diff --git a/src/include/replication/logicalrelation.h b/src/include/replication/logicalrelation.h index 78cd7e77f5..75b3a86c7d 100644 --- a/src/include/replication/logicalrelation.h +++ b/src/include/replication/logicalrelation.h @@ -15,6 +15,17 @@ #include "access/attmap.h" #include "replication/logicalproto.h" +/* + * States to determine volatility of the function in expressions in one + * relation. + */ +typedef enum RelFuncVolatility +{ + FUNCTION_UNKNOWN = 0, /* initializing */ + FUNCTION_IMMUTABLE, /* all functions are immutable function */ + FUNCTION_NONIMMUTABLE /* at least one non-immutable function */ +} RelFuncVolatility; + typedef struct LogicalRepRelMapEntry { LogicalRepRelation remoterel; /* key is remoterel.remoteid */ @@ -31,6 +42,10 @@ typedef struct LogicalRepRelMapEntry Relation localrel; /* relcache entry (NULL when closed) */ AttrMap *attrmap; /* map of local attributes to remote ones */ bool updatable; /* Can apply updates/deletes? */ + bool sameunique; /* Is the unique column of local and remote + consistent? */ + RelFuncVolatility volatility; /* all functions in localrel are + immutable function? */ /* Sync state. */ char state; diff --git a/src/include/replication/worker_internal.h b/src/include/replication/worker_internal.h index 08e63de013..dd49d8e2ec 100644 --- a/src/include/replication/worker_internal.h +++ b/src/include/replication/worker_internal.h @@ -196,6 +196,7 @@ extern void apply_bgworker_free(ApplyBgworkerState *wstate); extern void apply_bgworker_check_status(void); extern void apply_bgworker_set_status(ApplyBgworkerStatus status); extern void apply_bgworker_subxact_info_add(TransactionId current_xid); +extern void apply_bgworker_relation_check(LogicalRepRelMapEntry *rel); static inline bool am_tablesync_worker(void) diff --git a/src/include/utils/typcache.h b/src/include/utils/typcache.h index 431ad7f1b3..ed7c2e7f48 100644 --- a/src/include/utils/typcache.h +++ b/src/include/utils/typcache.h @@ -199,6 +199,8 @@ extern uint64 assign_record_type_identifier(Oid type_id, int32 typmod); extern int compare_values_of_enum(TypeCacheEntry *tcache, Oid arg1, Oid arg2); +extern List *GetDomainConstraints(Oid type_id); + extern size_t SharedRecordTypmodRegistryEstimate(void); extern void SharedRecordTypmodRegistryInit(SharedRecordTypmodRegistry *, diff --git a/src/test/subscription/t/022_twophase_cascade.pl b/src/test/subscription/t/022_twophase_cascade.pl index b52af74e35..81791c8d3a 100644 --- a/src/test/subscription/t/022_twophase_cascade.pl +++ b/src/test/subscription/t/022_twophase_cascade.pl @@ -39,6 +39,13 @@ sub test_streaming ALTER SUBSCRIPTION tap_sub_C SET (streaming = $streaming_mode)"); + if ($streaming_mode eq 'apply') + { + $node_C->safe_psql( + 'postgres', " + ALTER TABLE test_tab ALTER c DROP DEFAULT"); + } + # Wait for subscribers to finish initialization $node_A->poll_query_until( diff --git a/src/test/subscription/t/032_streaming_apply.pl b/src/test/subscription/t/032_streaming_apply.pl new file mode 100644 index 0000000000..776f1390fa --- /dev/null +++ b/src/test/subscription/t/032_streaming_apply.pl @@ -0,0 +1,238 @@ +# Copyright (c) 2022, PostgreSQL Global Development Group + +# Test the restrictions of streaming mode "apply" in logical replication + +use strict; +use warnings; +use PostgreSQL::Test::Cluster; +use PostgreSQL::Test::Utils; +use Test::More; + +my $offset = 0; + +# Create publisher node +my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); +$node_publisher->init(allows_streaming => 'logical'); +$node_publisher->append_conf('postgresql.conf', + 'logical_decoding_work_mem = 64kB'); +$node_publisher->start; + +# Create subscriber node +my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber'); +$node_subscriber->init; +$node_subscriber->start; + +# Setup structure on publisher +$node_publisher->safe_psql('postgres', + "CREATE TABLE test_tab (a int primary key, b varchar)"); + +# Setup structure on subscriber +$node_subscriber->safe_psql('postgres', + "CREATE TABLE test_tab (a int primary key, b varchar)"); + +# Setup logical replication +my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres'; +$node_publisher->safe_psql('postgres', + "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); + +my $appname = 'tap_sub'; +$node_subscriber->safe_psql( + 'postgres', " + CREATE SUBSCRIPTION tap_sub + CONNECTION '$publisher_connstr application_name=$appname' + PUBLICATION tap_pub + WITH (streaming = apply, copy_data = false)"); + +$node_publisher->wait_for_catchup($appname); + +# It is not allowed that the unique index on the publisher and the subscriber +# is different. Check the error reported by background worker in this case. +$node_subscriber->safe_psql('postgres', + "CREATE UNIQUE INDEX test_tab_b_idx ON test_tab (b)"); + +# Check that a background worker starts if "streaming" option is +# specified as "apply". We have to look for the DEBUG1 log messages +# about that, so temporarily bump up the log verbosity. +$node_subscriber->append_conf('postgresql.conf', "log_min_messages = debug1"); +$node_subscriber->reload; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); +$node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); +$node_subscriber->reload; + +$node_subscriber->wait_for_log( + qr/ERROR: cannot replicate relation with different unique index/, + $offset); + +# Drop the unique index on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', "DROP INDEX test_tab_b_idx"); + +$node_publisher->wait_for_catchup($appname); + +my $result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); +is($result, qq(5000), 'data replicated to subscriber after dropping index'); + +# Triggers which execute non-immutable function are not allowed on the +# subscriber side. Check the error reported by background worker in this case. +$node_subscriber->safe_psql( + 'postgres', qq{ +CREATE FUNCTION trigger_func() RETURNS TRIGGER AS \$\$ + BEGIN + RETURN NULL; + END +\$\$ language plpgsql; +CREATE TRIGGER insert_trig +BEFORE INSERT ON test_tab +FOR EACH ROW EXECUTE PROCEDURE trigger_func(); +ALTER TABLE test_tab ENABLE REPLICA TRIGGER insert_trig; +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', "DELETE FROM test_tab"); + +$node_subscriber->wait_for_log( + qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, + $offset); + +# Drop the trigger on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "DROP TRIGGER insert_trig ON test_tab"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); +is($result, qq(0), 'data replicated to subscriber after dropping trigger'); + +# It is not allowed that column default value expression contains a +# non-immutable function on the subscriber side. Check the error reported by +# background worker in this case. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab ALTER COLUMN b SET DEFAULT random()"); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log( + qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, + $offset); + +# Drop default value on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab ALTER COLUMN b DROP DEFAULT"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); +is($result, qq(5000), + 'data replicated to subscriber after dropping default value expression'); + +# It is not allowed that domain constraint expression contains a non-immutable +# function on the subscriber side. Check the error reported by background +# worker in this case. +$node_subscriber->safe_psql( + 'postgres', qq{ +CREATE DOMAIN test_domain AS int CHECK(VALUE > random()); +ALTER TABLE test_tab ALTER COLUMN a TYPE test_domain; +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', "DELETE FROM test_tab"); + +$node_subscriber->wait_for_log( + qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, + $offset); + +# Drop domain constraint expression on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab ALTER COLUMN a TYPE int"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); +is($result, qq(0), + 'data replicated to subscriber after dropping domain constraint expression' +); + +# It is not allowed that constraint expression contains a non-immutable function +# on the subscriber side. Check the error reported by background worker in this +# case. +$node_subscriber->safe_psql( + 'postgres', qq{ +ALTER TABLE test_tab ADD CONSTRAINT test_tab_con check (a > random()); +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log( + qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, + $offset); + +# Drop constraint on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab DROP CONSTRAINT test_tab_con"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); +is($result, qq(5000), + 'data replicated to subscriber after dropping constraint expression'); + +# It is not allowed that foreign key on the subscriber side. Check the error +# reported by background worker in this case. +$node_publisher->safe_psql('postgres', "DELETE FROM test_tab"); +$node_publisher->wait_for_catchup($appname); +$node_subscriber->safe_psql( + 'postgres', qq{ +CREATE TABLE test_tab_f (a int primary key); +ALTER TABLE test_tab ADD CONSTRAINT test_tabfk FOREIGN KEY(a) REFERENCES test_tab_f(a); +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log( + qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, + $offset); + +# Drop the foreign key constraint on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab DROP CONSTRAINT test_tabfk"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); +is($result, qq(5000), + 'data replicated to subscriber after dropping the foreign key'); + +$node_subscriber->stop; +$node_publisher->stop; + +done_testing(); -- 2.23.0.windows.1 [application/octet-stream] v10-0005-Retry-to-apply-streaming-xact-only-in-apply-work.patch (20.4K, ../../OS3PR01MB62756BD9482EB6BB1CA4CD4D9EAA9@OS3PR01MB6275.jpnprd01.prod.outlook.com/6-v10-0005-Retry-to-apply-streaming-xact-only-in-apply-work.patch) download | inline diff: From 56c7537e5b8e5044866d7fe7a1a6d691b2afa30d Mon Sep 17 00:00:00 2001 From: wangw <[email protected]> Date: Fri, 10 Jun 2022 15:04:52 +0800 Subject: [PATCH v10 5/5] Retry to apply streaming xact only in apply worker. --- doc/src/sgml/catalogs.sgml | 11 +++ doc/src/sgml/ref/create_subscription.sgml | 4 + src/backend/catalog/pg_subscription.c | 1 + src/backend/catalog/system_views.sql | 4 +- src/backend/commands/subscriptioncmds.c | 1 + .../replication/logical/applybgwroker.c | 13 ++- src/backend/replication/logical/worker.c | 89 ++++++++++++++++++ src/bin/pg_dump/pg_dump.c | 5 +- src/include/catalog/pg_subscription.h | 4 + .../subscription/t/032_streaming_apply.pl | 91 ++++++++++--------- 10 files changed, 174 insertions(+), 49 deletions(-) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index 5cb39b18bd..928ad02ace 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -7907,6 +7907,17 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>subretry</structfield> <type>bool</type> + </para> + <para> + If true, the subscription will not try to apply streaming transaction + in <literal>apply</literal> mode. See + <xref linkend="sql-createsubscription"/> for more information. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>subconninfo</structfield> <type>text</type> diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml index 8de1a23ce4..70fbf81668 100644 --- a/doc/src/sgml/ref/create_subscription.sgml +++ b/doc/src/sgml/ref/create_subscription.sgml @@ -244,6 +244,10 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl is that the unique column should be the same between publisher and subscriber; the second is that there should not be any non-immutable function in subscriber-side replicated table. + When applying a streaming transaction, if either requirement is not + met, the background worker will exit with an error. And when retrying + later, we will try to apply this transaction in <literal>on</literal> + mode. </para> </listitem> </varlistentry> diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c index add51caadf..1824390d7d 100644 --- a/src/backend/catalog/pg_subscription.c +++ b/src/backend/catalog/pg_subscription.c @@ -71,6 +71,7 @@ GetSubscription(Oid subid, bool missing_ok) sub->stream = subform->substream; sub->twophasestate = subform->subtwophasestate; sub->disableonerr = subform->subdisableonerr; + sub->retry = subform->subretry; /* Get conninfo */ datum = SysCacheGetAttr(SUBSCRIPTIONOID, diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql index fedaed533b..10f4dd6785 100644 --- a/src/backend/catalog/system_views.sql +++ b/src/backend/catalog/system_views.sql @@ -1298,8 +1298,8 @@ REVOKE ALL ON pg_replication_origin_status FROM public; -- All columns of pg_subscription except subconninfo are publicly readable. REVOKE ALL ON pg_subscription FROM public; GRANT SELECT (oid, subdbid, subskiplsn, subname, subowner, subenabled, - subbinary, substream, subtwophasestate, subdisableonerr, subslotname, - subsynccommit, subpublications) + subbinary, substream, subtwophasestate, subdisableonerr, + subretry, subslotname, subsynccommit, subpublications) ON pg_subscription TO public; CREATE VIEW pg_stat_subscription_stats AS diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c index 4080bba987..38697184d9 100644 --- a/src/backend/commands/subscriptioncmds.c +++ b/src/backend/commands/subscriptioncmds.c @@ -663,6 +663,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt, LOGICALREP_TWOPHASE_STATE_PENDING : LOGICALREP_TWOPHASE_STATE_DISABLED); values[Anum_pg_subscription_subdisableonerr - 1] = BoolGetDatum(opts.disableonerr); + values[Anum_pg_subscription_subretry - 1] = BoolGetDatum(false); values[Anum_pg_subscription_subconninfo - 1] = CStringGetTextDatum(conninfo); if (opts.slot_name) diff --git a/src/backend/replication/logical/applybgwroker.c b/src/backend/replication/logical/applybgwroker.c index 7d3c61c2e9..02b1ef5e62 100644 --- a/src/backend/replication/logical/applybgwroker.c +++ b/src/backend/replication/logical/applybgwroker.c @@ -103,6 +103,13 @@ apply_bgworker_find_or_start(TransactionId xid, bool start) if (start && !XLogRecPtrIsInvalid(MySubscription->skiplsn)) return NULL; + if (start && MySubscription->retry) + { + elog(DEBUG1, "retry to apply an streaming transaction in apply " + "background worker"); + return NULL; + } + /* * For streaming transactions that are being applied in apply background * worker, we cannot decide whether to apply the change for a relation @@ -794,8 +801,7 @@ apply_bgworker_relation_check(LogicalRepRelMapEntry *rel) if (!rel->sameunique) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("cannot replicate relation with different unique index"), - errhint("Please change the streaming option to 'on' instead of 'apply'."))); + errmsg("cannot replicate relation with different unique index"))); /* * Check if there is any non-immutable function present in expression in @@ -804,7 +810,6 @@ apply_bgworker_relation_check(LogicalRepRelMapEntry *rel) if (rel->volatility == FUNCTION_NONIMMUTABLE) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("cannot replicate relation. There is at least one non-immutable function"), - errhint("Please change the streaming option to 'on' instead of 'apply'."))); + errmsg("cannot replicate relation. There is at least one non-immutable function"))); } diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c index 1b83a74685..b82f973626 100644 --- a/src/backend/replication/logical/worker.c +++ b/src/backend/replication/logical/worker.c @@ -374,6 +374,8 @@ static void clear_subscription_skip_lsn(XLogRecPtr finish_lsn); static inline void set_apply_error_context_xact(TransactionId xid, XLogRecPtr lsn); static inline void reset_apply_error_context_info(void); +static void set_subscription_retry(bool retry); + /* * Should this worker apply changes for given relation. * @@ -890,6 +892,9 @@ apply_handle_commit(StringInfo s) apply_handle_commit_internal(&commit_data); + /* Set the flag that we will not retry later. */ + set_subscription_retry(false); + /* Check the status of apply background worker if any. */ apply_bgworker_check_status(); @@ -1001,6 +1006,9 @@ apply_handle_prepare(StringInfo s) in_remote_transaction = false; + /* Set the flag that we will not retry later. */ + set_subscription_retry(false); + /* Check the status of apply background worker if any. */ apply_bgworker_check_status(); @@ -1054,6 +1062,9 @@ apply_handle_commit_prepared(StringInfo s) store_flush_position(prepare_data.end_lsn); in_remote_transaction = false; + /* Set the flag that we will not retry later. */ + set_subscription_retry(false); + /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(prepare_data.end_lsn); @@ -1109,6 +1120,9 @@ apply_handle_rollback_prepared(StringInfo s) store_flush_position(rollback_data.rollback_end_lsn); in_remote_transaction = false; + /* Set the flag that we will not retry later. */ + set_subscription_retry(false); + /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(rollback_data.rollback_end_lsn); @@ -1195,6 +1209,9 @@ apply_handle_stream_prepare(StringInfo s) /* unlink the files with serialized changes and subxact info. */ stream_cleanup_files(MyLogicalRepWorker->subid, prepare_data.xid); } + + /* Set the flag that we will not retry later. */ + set_subscription_retry(false); } in_remote_transaction = false; @@ -1615,6 +1632,9 @@ apply_handle_stream_abort(StringInfo s) */ else serialize_stream_abort(xid, subxid); + + /* Set the flag that we will not retry later. */ + set_subscription_retry(false); } } @@ -2788,6 +2808,9 @@ apply_handle_stream_commit(StringInfo s) /* unlink the files with serialized changes and subxact info */ stream_cleanup_files(MyLogicalRepWorker->subid, xid); } + + /* Set the flag that we will not retry later. */ + set_subscription_retry(false); } /* Check the status of apply background worker if any. */ @@ -3856,6 +3879,9 @@ start_table_sync(XLogRecPtr *origin_startpos, char **myslotname) } PG_CATCH(); { + /* Set the flag that we will retry later. */ + set_subscription_retry(true); + if (MySubscription->disableonerr) DisableSubscriptionAndExit(); else @@ -3894,6 +3920,9 @@ start_apply(XLogRecPtr origin_startpos) } PG_CATCH(); { + /* Set the flag that we will retry later. */ + set_subscription_retry(true); + if (MySubscription->disableonerr) DisableSubscriptionAndExit(); else @@ -4395,3 +4424,63 @@ reset_apply_error_context_info(void) apply_error_callback_arg.remote_attnum = -1; set_apply_error_context_xact(InvalidTransactionId, InvalidXLogRecPtr); } + +/* + * Set subretry of pg_subscription catalog. + * + * If retry is true, subscriber is about to exit with an error. Otherwise, it + * means that the changes was applied successfully. + */ +static void +set_subscription_retry(bool retry) +{ + Relation rel; + HeapTuple tup; + bool started_tx = false; + bool nulls[Natts_pg_subscription]; + bool replaces[Natts_pg_subscription]; + Datum values[Natts_pg_subscription]; + + if (MySubscription->retry == retry || + am_apply_bgworker()) + return; + + if (!IsTransactionState()) + { + StartTransactionCommand(); + started_tx = true; + } + + /* Look up the subscription in the catalog */ + rel = table_open(SubscriptionRelationId, RowExclusiveLock); + tup = SearchSysCacheCopy1(SUBSCRIPTIONOID, + ObjectIdGetDatum(MySubscription->oid)); + + if (!HeapTupleIsValid(tup)) + elog(ERROR, "subscription \"%s\" does not exist", MySubscription->name); + + LockSharedObject(SubscriptionRelationId, MySubscription->oid, 0, + AccessShareLock); + + /* Form a new tuple. */ + memset(values, 0, sizeof(values)); + memset(nulls, false, sizeof(nulls)); + memset(replaces, false, sizeof(replaces)); + + /* reset subretry */ + values[Anum_pg_subscription_subretry - 1] = BoolGetDatum(retry); + replaces[Anum_pg_subscription_subretry - 1] = true; + + tup = heap_modify_tuple(tup, RelationGetDescr(rel), values, nulls, + replaces); + + /* Update the catalog. */ + CatalogTupleUpdate(rel, &tup->t_self, tup); + + /* Cleanup. */ + heap_freetuple(tup); + table_close(rel, NoLock); + + if (started_tx) + CommitTransactionCommand(); +} diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 6f8b30abb0..9b3ffe0888 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -4472,8 +4472,9 @@ getSubscriptions(Archive *fout) ntups = PQntuples(res); /* - * Get subscription fields. We don't include subskiplsn in the dump as - * after restoring the dump this value may no longer be relevant. + * Get subscription fields. We don't include subskiplsn and subretry in + * the dump as after restoring the dump this value may no longer be + * relevant. */ i_tableoid = PQfnumber(res, "tableoid"); i_oid = PQfnumber(res, "oid"); diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h index 9b394a45fe..dc7597bf80 100644 --- a/src/include/catalog/pg_subscription.h +++ b/src/include/catalog/pg_subscription.h @@ -76,6 +76,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW bool subdisableonerr; /* True if a worker error should cause the * subscription to be disabled */ + bool subretry; /* True if the previous apply change failed. */ + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* Connection string to the publisher */ text subconninfo BKI_FORCE_NOT_NULL; @@ -115,6 +117,8 @@ typedef struct Subscription bool disableonerr; /* Indicates if the subscription should be * automatically disabled if a worker error * occurs */ + bool retry; /* Indicates if the previous apply change + * failed. */ char *conninfo; /* Connection string to the publisher */ char *slotname; /* Name of the replication slot */ char *synccommit; /* Synchronous commit setting for worker */ diff --git a/src/test/subscription/t/032_streaming_apply.pl b/src/test/subscription/t/032_streaming_apply.pl index 776f1390fa..51483b04a1 100644 --- a/src/test/subscription/t/032_streaming_apply.pl +++ b/src/test/subscription/t/032_streaming_apply.pl @@ -46,7 +46,9 @@ $node_subscriber->safe_psql( $node_publisher->wait_for_catchup($appname); # It is not allowed that the unique index on the publisher and the subscriber -# is different. Check the error reported by background worker in this case. +# is different. Check the error reported by background worker in this case. And +# after retrying in apply worker, we check if the data is replicated +# successfully. $node_subscriber->safe_psql('postgres', "CREATE UNIQUE INDEX test_tab_b_idx ON test_tab (b)"); @@ -69,17 +71,20 @@ $node_subscriber->wait_for_log( qr/ERROR: cannot replicate relation with different unique index/, $offset); -# Drop the unique index on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', "DROP INDEX test_tab_b_idx"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); my $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); -is($result, qq(5000), 'data replicated to subscriber after dropping index'); +is($result, qq(5000), 'data replicated to subscribers after retrying'); + +# Drop the unique index on the subscriber. +$node_subscriber->safe_psql('postgres', "DROP INDEX test_tab_b_idx"); # Triggers which execute non-immutable function are not allowed on the # subscriber side. Check the error reported by background worker in this case. +# And after retrying in apply worker, we check if the data is replicated +# successfully. $node_subscriber->safe_psql( 'postgres', qq{ CREATE FUNCTION trigger_func() RETURNS TRIGGER AS \$\$ @@ -102,19 +107,21 @@ $node_subscriber->wait_for_log( qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, $offset); -# Drop the trigger on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "DROP TRIGGER insert_trig ON test_tab"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); -is($result, qq(0), 'data replicated to subscriber after dropping trigger'); +is($result, qq(0), 'data replicated to subscribers after retrying'); + +# Drop the trigger on the subscriber. +$node_subscriber->safe_psql('postgres', + "DROP TRIGGER insert_trig ON test_tab"); # It is not allowed that column default value expression contains a # non-immutable function on the subscriber side. Check the error reported by -# background worker in this case. +# background worker in this case. And after retrying in apply worker, we check +# if the data is replicated successfully. $node_subscriber->safe_psql('postgres', "ALTER TABLE test_tab ALTER COLUMN b SET DEFAULT random()"); @@ -129,20 +136,21 @@ $node_subscriber->wait_for_log( qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, $offset); -# Drop default value on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "ALTER TABLE test_tab ALTER COLUMN b DROP DEFAULT"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); -is($result, qq(5000), - 'data replicated to subscriber after dropping default value expression'); +is($result, qq(5000), 'data replicated to subscribers after retrying'); + +# Drop default value on the subscriber. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab ALTER COLUMN b DROP DEFAULT"); # It is not allowed that domain constraint expression contains a non-immutable # function on the subscriber side. Check the error reported by background -# worker in this case. +# worker in this case. And after retrying in apply worker, we check if the data +# is replicated successfully. $node_subscriber->safe_psql( 'postgres', qq{ CREATE DOMAIN test_domain AS int CHECK(VALUE > random()); @@ -158,21 +166,21 @@ $node_subscriber->wait_for_log( qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, $offset); -# Drop domain constraint expression on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "ALTER TABLE test_tab ALTER COLUMN a TYPE int"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); -is($result, qq(0), - 'data replicated to subscriber after dropping domain constraint expression' -); +is($result, qq(0), 'data replicated to subscribers after retrying'); + +# Drop domain constraint expression on the subscriber. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab ALTER COLUMN a TYPE int"); -# It is not allowed that constraint expression contains a non-immutable function -# on the subscriber side. Check the error reported by background worker in this -# case. +# It is not allowed that constraint expression contains a non-immutable +# function on the subscriber side. Check the error reported by background +# worker in this case. And after retrying in apply worker, we check if the data +# is replicated successfully. $node_subscriber->safe_psql( 'postgres', qq{ ALTER TABLE test_tab ADD CONSTRAINT test_tab_con check (a > random()); @@ -189,19 +197,20 @@ $node_subscriber->wait_for_log( qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, $offset); -# Drop constraint on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "ALTER TABLE test_tab DROP CONSTRAINT test_tab_con"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); -is($result, qq(5000), - 'data replicated to subscriber after dropping constraint expression'); +is($result, qq(5000), 'data replicated to subscribers after retrying'); + +# Drop constraint on the subscriber. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab DROP CONSTRAINT test_tab_con"); # It is not allowed that foreign key on the subscriber side. Check the error -# reported by background worker in this case. +# reported by background worker in this case. And after retrying in apply +# worker, we check if the data is replicated successfully. $node_publisher->safe_psql('postgres', "DELETE FROM test_tab"); $node_publisher->wait_for_catchup($appname); $node_subscriber->safe_psql( @@ -221,16 +230,16 @@ $node_subscriber->wait_for_log( qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, $offset); -# Drop the foreign key constraint on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "ALTER TABLE test_tab DROP CONSTRAINT test_tabfk"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); -is($result, qq(5000), - 'data replicated to subscriber after dropping the foreign key'); +is($result, qq(5000), 'data replicated to subscribers after retrying'); + +# Drop the foreign key constraint on the subscriber. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab DROP CONSTRAINT test_tabfk"); $node_subscriber->stop; $node_publisher->stop; -- 2.23.0.windows.1 ^ permalink raw reply [nested|flat] 66+ messages in thread
* RE: Perform streaming logical transactions by background workers and parallel apply @ 2022-06-15 08:26 [email protected] <[email protected]> parent: [email protected] <[email protected]> 1 sibling, 1 reply; 66+ messages in thread From: [email protected] @ 2022-06-15 08:26 UTC (permalink / raw) To: Amit Kapila <[email protected]>; Masahiko Sawada <[email protected]>; +Cc: Peter Smith <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; PostgreSQL Hackers <[email protected]> On Tues, Jun 14, 2022 11:17 AM I wrote: > Attach the new patches. > ...... > 3. Adding a check for the partition table when trying to apply changes in the > apply background worker. (see patch 0004) > In additional, the partition cache map on subscriber have several bugs (see > thread [2]). Because patch 0004 is developed based on the patches in [2], so I > merged the patches(v4-0001~v4-0003) in [2] into a temporary patch 0003 here. > After the patches in [2] is committed, I will delete patch 0003 and rebase > patch 0004. I added some test cases for this (see patch 0004). In patch 0005, I made corresponding adjustments according to these test cases. I also slightly modified the comments about the check for unique index. (see patch 0004) Also rebased the temporary patch 0003 because the first patch in thread [1] is committed (see commit 5a97b132 in HEAD) . Attach the new patches. Only changed patches 0004, 0005. [1] - https://www.postgresql.org/message-id/OSZPR01MB6310F46CD425A967E4AEF736FDA49%40OSZPR01MB6310.jpnprd0... Regards, Wang wei Attachments: [application/octet-stream] v11-0001-Perform-streaming-logical-transactions-by-backgr.patch (98.2K, ../../OS3PR01MB6275C4679CFE3414340A259B9EAD9@OS3PR01MB6275.jpnprd01.prod.outlook.com/2-v11-0001-Perform-streaming-logical-transactions-by-backgr.patch) download | inline diff: From 850f741aeb31c509ae5c9bcd49233795fb83f708 Mon Sep 17 00:00:00 2001 From: "houzj.fnst" <[email protected]> Date: Wed, 20 Apr 2022 16:45:07 +0800 Subject: [PATCH v11 1/5] Perform streaming logical transactions by background workers Currently, for large transactions, the publisher sends the data in multiple streams (changes divided into chunks depending upon logical_decoding_work_mem), and then on the subscriber-side, the apply worker writes the changes into temporary files and once it receives the commit, it reads from the file and applies the entire transaction. To improve the performance of such transactions, we can instead allow them to be applied via background workers. In this approach, we assign a new apply background worker (if available) as soon as the xact's first stream is received and the main apply worker will send changes to this new worker via shared memory. The apply background worker will directly apply the change instead of writing it to temporary files. We keep this worker assigned till the transaction commit is received and also wait for the worker to finish at commit. This preserves commit ordering and avoids writing to and reading from file in most cases. We still need to spill if there is no worker available. We also need to allow stream_stop to complete by the apply background worker to avoid deadlocks because T-1's current stream of changes can update rows in conflicting order with T-2's next stream of changes. This patch also extends the SUBSCRIPTION 'streaming' option so that the user can control whether to apply the streaming transaction in an apply background worker or spill the change to disk. User can set the streaming option to 'on/off', 'apply'. For now, 'apply' means the streaming will be applied via a apply background worker if available. 'on' means the streaming transaction will be spilled to disk. --- doc/src/sgml/catalogs.sgml | 10 +- doc/src/sgml/config.sgml | 25 + doc/src/sgml/logical-replication.sgml | 7 + doc/src/sgml/protocol.sgml | 19 + doc/src/sgml/ref/create_subscription.sgml | 24 +- src/backend/access/transam/xact.c | 13 + src/backend/commands/subscriptioncmds.c | 66 +- src/backend/postmaster/bgworker.c | 3 + src/backend/replication/logical/Makefile | 3 +- .../replication/logical/applybgwroker.c | 765 ++++++++++++++++++ src/backend/replication/logical/decode.c | 10 +- src/backend/replication/logical/launcher.c | 112 ++- src/backend/replication/logical/origin.c | 26 +- src/backend/replication/logical/proto.c | 35 +- .../replication/logical/reorderbuffer.c | 10 +- src/backend/replication/logical/tablesync.c | 10 +- src/backend/replication/logical/worker.c | 645 +++++++++++---- src/backend/replication/pgoutput/pgoutput.c | 2 +- src/backend/utils/activity/wait_event.c | 3 + src/backend/utils/misc/guc.c | 12 + src/backend/utils/misc/postgresql.conf.sample | 1 + src/bin/pg_dump/pg_dump.c | 6 +- src/include/catalog/pg_subscription.h | 17 +- src/include/replication/logicallauncher.h | 1 + src/include/replication/logicalproto.h | 19 +- src/include/replication/logicalworker.h | 1 + src/include/replication/origin.h | 2 +- src/include/replication/reorderbuffer.h | 7 +- src/include/replication/worker_internal.h | 104 ++- src/include/utils/wait_event.h | 1 + src/test/regress/expected/subscription.out | 2 +- src/tools/pgindent/typedefs.list | 5 + 32 files changed, 1751 insertions(+), 215 deletions(-) create mode 100644 src/backend/replication/logical/applybgwroker.c diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index c00c93dd7b..5cb39b18bd 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -7873,11 +7873,15 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l <row> <entry role="catalog_table_entry"><para role="column_definition"> - <structfield>substream</structfield> <type>bool</type> + <structfield>substream</structfield> <type>char</type> </para> <para> - If true, the subscription will allow streaming of in-progress - transactions + Controls how to handle the streaming of in-progress transactions: + <literal>f</literal> = disallow streaming of in-progress transactions, + <literal>t</literal> = spill the changes of in-progress transactions to + disk and apply at once after the transaction is committed on the + publisher, + <literal>a</literal> = apply changes directly using a background worker </para></entry> </row> diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml index 5b7ce6531d..ce7f6827f2 100644 --- a/doc/src/sgml/config.sgml +++ b/doc/src/sgml/config.sgml @@ -4970,6 +4970,31 @@ ANY <replaceable class="parameter">num_sync</replaceable> ( <replaceable class=" </listitem> </varlistentry> + <varlistentry id="guc-max-apply-bgworkers-per-subscription" xreflabel="max_apply_bgworkers_per_subscription"> + <term><varname>max_apply_bgworkers_per_subscription</varname> (<type>integer</type>) + <indexterm> + <primary><varname>max_apply_bgworkers_per_subscription</varname> configuration parameter</primary> + </indexterm> + </term> + <listitem> + <para> + Maximum number of apply background workers per subscription. This + parameter controls the amount of parallelism of the streaming of + in-progress transactions if we set subscription option + <literal>streaming</literal> to <literal>apply</literal>. + </para> + <para> + The apply background workers are taken from the pool defined by + <varname>max_logical_replication_workers</varname>. + </para> + <para> + The default value is 3. This parameter can only be set in the + <filename>postgresql.conf</filename> file or on the server command + line. + </para> + </listitem> + </varlistentry> + </variablelist> </sect2> diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml index 145ea71d61..cca6c97d79 100644 --- a/doc/src/sgml/logical-replication.sgml +++ b/doc/src/sgml/logical-replication.sgml @@ -948,6 +948,13 @@ CONTEXT: processing remote data for replication origin "pg_16395" during "INSER might not violate any constraint. This can easily make the subscriber inconsistent. </para> + + <para> + Setting streaming mode to <literal>apply</literal> could export invalid LSN + as finish LSN of failed transaction. Changing the streaming mode and making + the same conflict writes the finish LSN of the failed transaction in the + server log if required. + </para> </sect1> <sect1 id="logical-replication-restrictions"> diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml index bb30340892..34be1b2698 100644 --- a/doc/src/sgml/protocol.sgml +++ b/doc/src/sgml/protocol.sgml @@ -6809,6 +6809,25 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;" </listitem> </varlistentry> + <varlistentry> + <term>Int64 (XLogRecPtr)</term> + <listitem> + <para> + The LSN of the abort. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Int64 (TimestampTz)</term> + <listitem> + <para> + Abort timestamp of the transaction. The value is in number + of microseconds since PostgreSQL epoch (2000-01-01). + </para> + </listitem> + </varlistentry> + <varlistentry> <term>Int32 (TransactionId)</term> <listitem> diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml index 35b39c28da..d4caae0222 100644 --- a/doc/src/sgml/ref/create_subscription.sgml +++ b/doc/src/sgml/ref/create_subscription.sgml @@ -217,13 +217,29 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl </varlistentry> <varlistentry> - <term><literal>streaming</literal> (<type>boolean</type>)</term> + <term><literal>streaming</literal> (<type>enum</type>)</term> <listitem> <para> Specifies whether to enable streaming of in-progress transactions - for this subscription. By default, all transactions - are fully decoded on the publisher and only then sent to the - subscriber as a whole. + for this subscription. The default value is <literal>off</literal>, + all transactions are fully decoded on the publisher and only then + sent to the subscriber as a whole. + </para> + + <para> + If set to <literal>on</literal>, the changes of transaction are + written to temporary files and then applied at once after the + transaction is committed on the publisher. + </para> + + <para> + If set to <literal>apply</literal> incoming + changes are directly applied via one of the background workers, if + available. If no background worker is free to handle streaming + transaction then the changes are written to a file and applied after + the transaction is committed. Note that if an error happens when + applying changes in a background worker, it might not report the + finish LSN of the remote transaction in the server log. </para> </listitem> </varlistentry> diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c index 47d80b0d25..678540ba25 100644 --- a/src/backend/access/transam/xact.c +++ b/src/backend/access/transam/xact.c @@ -1711,6 +1711,7 @@ RecordTransactionAbort(bool isSubXact) int nchildren; TransactionId *children; TimestampTz xact_time; + bool replorigin; /* * If we haven't been assigned an XID, nobody will care whether we aborted @@ -1741,6 +1742,13 @@ RecordTransactionAbort(bool isSubXact) elog(PANIC, "cannot abort transaction %u, it was already committed", xid); + /* + * Are we using the replication origins feature? Or, in other words, + * are we replaying remote actions? + */ + replorigin = (replorigin_session_origin != InvalidRepOriginId && + replorigin_session_origin != DoNotReplicateId); + /* Fetch the data we need for the abort record */ nrels = smgrGetPendingDeletes(false, &rels); nchildren = xactGetCommittedChildren(&children); @@ -1765,6 +1773,11 @@ RecordTransactionAbort(bool isSubXact) MyXactFlags, InvalidTransactionId, NULL); + if (replorigin) + /* Move LSNs forward for this replication origin */ + replorigin_session_advance(replorigin_session_origin_lsn, + XactLastRecEnd); + /* * Report the latest async abort LSN, so that the WAL writer knows to * flush this abort. There's nothing to be gained by delaying this, since diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c index 83e6eae855..4080bba987 100644 --- a/src/backend/commands/subscriptioncmds.c +++ b/src/backend/commands/subscriptioncmds.c @@ -83,7 +83,7 @@ typedef struct SubOpts bool copy_data; bool refresh; bool binary; - bool streaming; + char streaming; bool twophase; bool disableonerr; XLogRecPtr lsn; @@ -95,6 +95,62 @@ static List *merge_publications(List *oldpublist, List *newpublist, bool addpub, static void ReportSlotConnectionError(List *rstates, Oid subid, char *slotname, char *err); +/* + * Extract the streaming mode value from a DefElem. This is like + * defGetBoolean() but also accepts the special value of "apply". + */ +static char +defGetStreamingMode(DefElem *def) +{ + /* + * If no parameter given, assume "true" is meant. + */ + if (def->arg == NULL) + return SUBSTREAM_ON; + + /* + * Allow 0, 1, "true", "false", "on", "off" or "apply". + */ + switch (nodeTag(def->arg)) + { + case T_Integer: + switch (intVal(def->arg)) + { + case 0: + return SUBSTREAM_OFF; + case 1: + return SUBSTREAM_ON; + default: + /* otherwise, error out below */ + break; + } + break; + default: + { + char *sval = defGetString(def); + + /* + * The set of strings accepted here should match up with the + * grammar's opt_boolean_or_string production. + */ + if (pg_strcasecmp(sval, "false") == 0 || + pg_strcasecmp(sval, "off") == 0) + return SUBSTREAM_OFF; + if (pg_strcasecmp(sval, "true") == 0 || + pg_strcasecmp(sval, "on") == 0) + return SUBSTREAM_ON; + if (pg_strcasecmp(sval, "apply") == 0) + return SUBSTREAM_APPLY; + } + break; + } + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s requires a Boolean value or \"apply\"", + def->defname))); + return SUBSTREAM_OFF; /* keep compiler quiet */ +} + /* * Common option parsing function for CREATE and ALTER SUBSCRIPTION commands. * @@ -132,7 +188,7 @@ parse_subscription_options(ParseState *pstate, List *stmt_options, if (IsSet(supported_opts, SUBOPT_BINARY)) opts->binary = false; if (IsSet(supported_opts, SUBOPT_STREAMING)) - opts->streaming = false; + opts->streaming = SUBSTREAM_OFF; if (IsSet(supported_opts, SUBOPT_TWOPHASE_COMMIT)) opts->twophase = false; if (IsSet(supported_opts, SUBOPT_DISABLE_ON_ERR)) @@ -233,7 +289,7 @@ parse_subscription_options(ParseState *pstate, List *stmt_options, errorConflictingDefElem(defel, pstate); opts->specified_opts |= SUBOPT_STREAMING; - opts->streaming = defGetBoolean(defel); + opts->streaming = defGetStreamingMode(defel); } else if (strcmp(defel->defname, "two_phase") == 0) { @@ -601,7 +657,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt, values[Anum_pg_subscription_subowner - 1] = ObjectIdGetDatum(owner); values[Anum_pg_subscription_subenabled - 1] = BoolGetDatum(opts.enabled); values[Anum_pg_subscription_subbinary - 1] = BoolGetDatum(opts.binary); - values[Anum_pg_subscription_substream - 1] = BoolGetDatum(opts.streaming); + values[Anum_pg_subscription_substream - 1] = CharGetDatum(opts.streaming); values[Anum_pg_subscription_subtwophasestate - 1] = CharGetDatum(opts.twophase ? LOGICALREP_TWOPHASE_STATE_PENDING : @@ -1060,7 +1116,7 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt, if (IsSet(opts.specified_opts, SUBOPT_STREAMING)) { values[Anum_pg_subscription_substream - 1] = - BoolGetDatum(opts.streaming); + CharGetDatum(opts.streaming); replaces[Anum_pg_subscription_substream - 1] = true; } diff --git a/src/backend/postmaster/bgworker.c b/src/backend/postmaster/bgworker.c index 40601aefd9..40ccb8993c 100644 --- a/src/backend/postmaster/bgworker.c +++ b/src/backend/postmaster/bgworker.c @@ -128,6 +128,9 @@ static const struct }, { "ApplyWorkerMain", ApplyWorkerMain + }, + { + "ApplyBgworkerMain", ApplyBgworkerMain } }; diff --git a/src/backend/replication/logical/Makefile b/src/backend/replication/logical/Makefile index c4e2fdeb71..a24a41969e 100644 --- a/src/backend/replication/logical/Makefile +++ b/src/backend/replication/logical/Makefile @@ -26,6 +26,7 @@ OBJS = \ reorderbuffer.o \ snapbuild.o \ tablesync.o \ - worker.o + worker.o \ + applybgwroker.o include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/replication/logical/applybgwroker.c b/src/backend/replication/logical/applybgwroker.c new file mode 100644 index 0000000000..1f52b155d7 --- /dev/null +++ b/src/backend/replication/logical/applybgwroker.c @@ -0,0 +1,765 @@ +/*------------------------------------------------------------------------- + * applybgwroker.c + * Support routines for applying xact by apply background worker + * + * Copyright (c) 2016-2022, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/replication/logical/applybgwroker.c + * + * This file contains routines that are intended to support setting up, using, + * and tearing down a ApplyBgworkerState. + * Refer to the comments in file header of logical/worker.c to see more + * informations about apply background worker. + * + *------------------------------------------------------------------------- + */ + +#include "postgres.h" + +#include "libpq/pqformat.h" +#include "mb/pg_wchar.h" +#include "pgstat.h" +#include "postmaster/interrupt.h" +#include "replication/logicallauncher.h" +#include "replication/logicalworker.h" +#include "replication/origin.h" +#include "replication/walreceiver.h" +#include "replication/worker_internal.h" +#include "storage/ipc.h" +#include "storage/procarray.h" +#include "tcop/tcopprot.h" +#include "utils/inval.h" +#include "utils/memutils.h" +#include "utils/resowner.h" +#include "utils/syscache.h" + +#define PG_LOGICAL_APPLY_SHM_MAGIC 0x79fb2447 // TODO Consider change + +/* + * DSM keys for apply background worker. Unlike other parallel execution code, + * since we don't need to worry about DSM keys conflicting with plan_node_id we + * can use small integers. + */ +#define APPLY_BGWORKER_KEY_SHARED 1 +#define APPLY_BGWORKER_KEY_MQ 2 + +/* + * entry for a hash table we use to map from xid to our apply background worker + * state. + */ +typedef struct ApplyBgworkerEntry +{ + TransactionId xid; + ApplyBgworkerState *wstate; +} ApplyBgworkerEntry; + +/* Apply background workers hash table (initialized on first use) */ +static HTAB *ApplyWorkersHash = NULL; +static List *ApplyWorkersFreeList = NIL; +static List *ApplyWorkersList = NIL; + +/* + * Fields to record the share informations between main apply worker and apply + * background worker. + */ +volatile ApplyBgworkerShared *MyParallelState = NULL; + +List *subxactlist = NIL; + +/* apply background worker setup */ +static ApplyBgworkerState *apply_bgworker_setup(void); +static void apply_bgworker_setup_dsm(ApplyBgworkerState *wstate); + +/* + * Look up worker inside ApplyWorkersHash for requested xid. + * + * If start flag is true, try to start a new worker if not found in hash table. + */ +ApplyBgworkerState * +apply_bgworker_find_or_start(TransactionId xid, bool start) +{ + bool found; + ApplyBgworkerState *wstate; + ApplyBgworkerEntry *entry = NULL; + + if (!TransactionIdIsValid(xid)) + return NULL; + + /* + * We don't start new background worker if we are not in streaming apply + * mode. + */ + if (MySubscription->stream != SUBSTREAM_APPLY) + return NULL; + + /* + * We don't start new background worker if user has set skiplsn as it's + * possible that user want to skip the streaming transaction. For + * streaming transaction, we need to spill the transaction to disk so that + * we can get the last LSN of the transaction to judge whether to skip + * before starting to apply the change. + */ + if (start && !XLogRecPtrIsInvalid(MySubscription->skiplsn)) + return NULL; + + /* + * For streaming transactions that are being applied in apply background + * worker, we cannot decide whether to apply the change for a relation + * that is not in the READY state (see should_apply_changes_for_rel) as we + * won't know remote_final_lsn by that time. So, we don't start new apply + * background worker in this case. + */ + if (start && !AllTablesyncsReady()) + return NULL; + + /* First time through, initialize apply workers hashtable */ + if (ApplyWorkersHash == NULL) + { + HASHCTL ctl; + + MemSet(&ctl, 0, sizeof(ctl)); + ctl.keysize = sizeof(TransactionId); + ctl.entrysize = sizeof(ApplyBgworkerEntry); + ctl.hcxt = ApplyContext; + + ApplyWorkersHash = hash_create("logical apply workers hash", 8, &ctl, + HASH_ELEM | HASH_BLOBS | HASH_CONTEXT); + } + + /* + * Find entry for requested transaction. + */ + entry = hash_search(ApplyWorkersHash, &xid, start ? HASH_ENTER : HASH_FIND, + &found); + if (found) + { + entry->wstate->pstate->status = APPLY_BGWORKER_BUSY; + return entry->wstate; + } + else if (!start) + return NULL; + + /* + * Now, we try to get a apply background worker. If there is at least one + * worker in the idle list, then take one. Otherwise, we try to start a + * new apply background worker. + */ + if (list_length(ApplyWorkersFreeList) > 0) + { + wstate = (ApplyBgworkerState *) llast(ApplyWorkersFreeList); + ApplyWorkersFreeList = list_delete_last(ApplyWorkersFreeList); + } + else + { + wstate = apply_bgworker_setup(); + + if (wstate == NULL) + { + /* + * If the apply background worker cannot be launched, remove entry + * in hash table. + */ + hash_search(ApplyWorkersHash, &xid, HASH_REMOVE, &found); + return NULL; + } + } + + /* Fill up the hash entry */ + wstate->pstate->status = APPLY_BGWORKER_BUSY; + wstate->pstate->server_version = walrcv_server_version(LogRepWorkerWalRcvConn); + wstate->pstate->stream_xid = xid; + entry->wstate = wstate; + entry->xid = xid; + + return wstate; +} + +/* + * Add the worker to the free list and remove the entry from hash table. + */ +void +apply_bgworker_free(ApplyBgworkerState *wstate) +{ + MemoryContext oldctx; + TransactionId xid = wstate->pstate->stream_xid; + + Assert(wstate->pstate->status == APPLY_BGWORKER_FINISHED); + + oldctx = MemoryContextSwitchTo(ApplyContext); + + hash_search(ApplyWorkersHash, &xid, HASH_REMOVE, NULL); + + elog(DEBUG1, "adding finished apply worker #%u for xid %u to the idle list", + wstate->pstate->n, wstate->pstate->stream_xid); + + ApplyWorkersFreeList = lappend(ApplyWorkersFreeList, wstate); + + MemoryContextSwitchTo(oldctx); +} + +/* Apply Background Worker main loop */ +static void +LogicalApplyBgwLoop(shm_mq_handle *mqh, volatile ApplyBgworkerShared *pst) +{ + shm_mq_result shmq_res; + PGPROC *registrant; + ErrorContextCallback errcallback; + XLogRecPtr last_received = InvalidXLogRecPtr; + + registrant = BackendPidGetProc(MyBgworkerEntry->bgw_notify_pid); + SetLatch(®istrant->procLatch); + + /* + * Push apply error context callback. Fields will be filled during + * applying a change. + */ + errcallback.callback = apply_error_callback; + errcallback.previous = error_context_stack; + error_context_stack = &errcallback; + + for (;;) + { + void *data; + Size len; + int c; + StringInfoData s; + MemoryContext oldctx; + XLogRecPtr start_lsn; + XLogRecPtr end_lsn; + TimestampTz send_time; + + CHECK_FOR_INTERRUPTS(); + + /* Ensure we are reading the data into our memory context. */ + oldctx = MemoryContextSwitchTo(ApplyMessageContext); + + shmq_res = shm_mq_receive(mqh, &len, &data, false); + + if (shmq_res != SHM_MQ_SUCCESS) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("lost connection to the main apply worker"))); + + if (len == 0) + break; + + s.cursor = 0; + s.maxlen = -1; + s.data = (char *) data; + s.len = len; + + /* + * We use first byte of message for additional communication between + * main Logical replication worker and apply bgworkers, so if it + * differs from 'w', then process it first. + */ + c = pq_getmsgbyte(&s); + switch (c) + { + /* End message of streaming chunk */ + case LOGICAL_REP_MSG_STREAM_STOP: + elog(DEBUG1, "[Apply BGW #%u] ended processing streaming chunk," + "waiting on shm_mq_receive", pst->n); + + apply_bgworker_set_status(APPLY_BGWORKER_READY); + + SetLatch(®istrant->procLatch); + + in_streamed_transaction = false; + pgstat_report_activity(STATE_IDLE, NULL); + continue; + case 'w': + break; + default: + elog(ERROR, "[Apply BGW #%u] unexpected message \"%c\"", + pst->n, c); + break; + } + + start_lsn = pq_getmsgint64(&s); + end_lsn = pq_getmsgint64(&s); + send_time = pq_getmsgint64(&s); + + if (last_received < start_lsn) + last_received = start_lsn; + + if (last_received < end_lsn) + last_received = end_lsn; + + /* + * TO IMPROVE: Do we need to display the apply background worker's + * information in pg_stat_replication ? + */ + UpdateWorkerStats(last_received, send_time, false); + + apply_dispatch(&s); + + if (ConfigReloadPending) + { + ConfigReloadPending = false; + ProcessConfigFile(PGC_SIGHUP); + } + + MemoryContextSwitchTo(oldctx); + MemoryContextReset(ApplyMessageContext); + } + + MemoryContextSwitchTo(TopMemoryContext); + MemoryContextReset(ApplyContext); + + /* Pop the error context stack */ + error_context_stack = errcallback.previous; + + elog(DEBUG1, "[Apply BGW #%u] exiting", pst->n); + + /* Signal main process that we are done. */ + SetLatch(®istrant->procLatch); +} + +/* + * Set the exit status so that the main apply worker can realize we have + * shutdown. + */ +static void +ApplyBgwShutdown(int code, Datum arg) +{ + SpinLockAcquire(&MyParallelState->mutex); + MyParallelState->status = APPLY_BGWORKER_EXIT; + SpinLockRelease(&MyParallelState->mutex); + + dsm_detach((dsm_segment *) DatumGetPointer(arg)); +} + +/* + * Apply Background Worker entry point + */ +void +ApplyBgworkerMain(Datum main_arg) +{ + volatile ApplyBgworkerShared *pst; + + dsm_handle handle; + dsm_segment *seg; + shm_toc *toc; + shm_mq *mq; + shm_mq_handle *mqh; + MemoryContext oldcontext; + RepOriginId originid; + int worker_slot = DatumGetInt32(main_arg); + char originname[NAMEDATALEN]; + + /* Attach to slot */ + logicalrep_worker_attach(worker_slot); + + MemoryContextSwitchTo(TopMemoryContext); + + /* Load the subscription into persistent memory context. */ + ApplyContext = AllocSetContextCreate(TopMemoryContext, + "ApplyContext", + ALLOCSET_DEFAULT_SIZES); + + /* + * Init the ApplyMessageContext which we clean up after each replication + * protocol message. + */ + ApplyMessageContext = AllocSetContextCreate(ApplyContext, + "ApplyMessageContext", + ALLOCSET_DEFAULT_SIZES); + + /* Setup signal handling */ + pqsignal(SIGHUP, SignalHandlerForConfigReload); + pqsignal(SIGTERM, die); + BackgroundWorkerUnblockSignals(); + + /* + * Connect to the dynamic shared memory segment. + * + * The backend that registered this worker passed us the ID of a shared + * memory segment to which we must attach for further instructions. In + * order to attach to dynamic shared memory, we need a resource owner. + * Once we've mapped the segment in our address space, attach to the table + * of contents so we can locate the various data structures we'll need to + * find within the segment. + */ + CurrentResourceOwner = ResourceOwnerCreate(NULL, "Logical apply worker"); + memcpy(&handle, MyBgworkerEntry->bgw_extra, sizeof(dsm_handle)); + seg = dsm_attach(handle); + if (seg == NULL) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("unable to map dynamic shared memory segment"))); + toc = shm_toc_attach(PG_LOGICAL_APPLY_SHM_MAGIC, dsm_segment_address(seg)); + if (toc == NULL) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("bad magic number in dynamic shared memory segment"))); + + before_shmem_exit(ApplyBgwShutdown, PointerGetDatum(seg)); + + /* Look up the parallel state. */ + pst = shm_toc_lookup(toc, APPLY_BGWORKER_KEY_SHARED, false); + MyParallelState = pst; + + /* + * Attach to the message queue. + */ + mq = shm_toc_lookup(toc, APPLY_BGWORKER_KEY_MQ, false); + shm_mq_set_receiver(mq, MyProc); + mqh = shm_mq_attach(mq, seg, NULL); + + /* Run as replica session replication role. */ + SetConfigOption("session_replication_role", "replica", + PGC_SUSET, PGC_S_OVERRIDE); + + /* Connect to our database. */ + BackgroundWorkerInitializeConnectionByOid(MyLogicalRepWorker->dbid, + MyLogicalRepWorker->userid, + 0); + + /* + * Set always-secure search path, so malicious users can't redirect user + * code (e.g. pg_index.indexprs). + */ + SetConfigOption("search_path", "", PGC_SUSET, PGC_S_OVERRIDE); + + /* + * Set the client encoding to the database encoding, since that is what + * the leader will expect. + */ + SetClientEncoding(GetDatabaseEncoding()); + + stream_xid = pst->stream_xid; + + StartTransactionCommand(); + oldcontext = MemoryContextSwitchTo(ApplyContext); + + MySubscription = GetSubscription(MyLogicalRepWorker->subid, true); + if (!MySubscription) + { + ereport(LOG, + (errmsg("logical replication apply worker for subscription %u will not " + "start because the subscription was removed during startup", + MyLogicalRepWorker->subid))); + proc_exit(0); + } + + MySubscriptionValid = true; + MemoryContextSwitchTo(oldcontext); + + /* Setup synchronous commit according to the user's wishes */ + SetConfigOption("synchronous_commit", MySubscription->synccommit, + PGC_BACKEND, PGC_S_OVERRIDE); + + /* Keep us informed about subscription changes. */ + CacheRegisterSyscacheCallback(SUBSCRIPTIONOID, + subscription_change_cb, + (Datum) 0); + + CommitTransactionCommand(); + + /* Setup replication origin tracking. */ + StartTransactionCommand(); + snprintf(originname, sizeof(originname), "pg_%u", MySubscription->oid); + originid = replorigin_by_name(originname, true); + if (!OidIsValid(originid)) + originid = replorigin_create(originname); + + /* + * The apply background worker doesn't need to monopolize this replication + * origin which was already acquired by its leader process. + */ + replorigin_session_setup(originid, false); + replorigin_session_origin = originid; + CommitTransactionCommand(); + + /* + * Allocate the origin name in long-lived context for error context + * message. + */ + apply_error_callback_arg.origin_name = MemoryContextStrdup(ApplyContext, + originname); + + /* + * Indicate that we're fully initialized and ready to begin the main part + * of the apply operation. + */ + apply_bgworker_set_status(APPLY_BGWORKER_ATTACHED); + + elog(DEBUG1, "[Apply BGW #%u] started", pst->n); + + LogicalApplyBgwLoop(mqh, pst); + + /* + * We're done. Explicitly detach the shared memory segment so that we + * don't get a resource leak warning at commit time. This will fire any + * on_dsm_detach callbacks we've registered, as well. Once that's done, + * we can go ahead and exit. + */ + dsm_detach(seg); + proc_exit(0); +} + +/* + * Set up a dynamic shared memory segment. + * + * We set up a control region that contains a ApplyBgworkerShared, + * plus one region per message queue. There are as many message queues as + * the number of workers. + */ +static void +apply_bgworker_setup_dsm(ApplyBgworkerState *wstate) +{ + shm_toc_estimator e; + Size segsize; + dsm_segment *seg; + shm_toc *toc; + ApplyBgworkerShared *pst; + shm_mq *mq; + int64 queue_size = 160000000; /* 16 MB for now */ + + /* + * Estimate how much shared memory we need. + * + * Because the TOC machinery may choose to insert padding of oddly-sized + * requests, we must estimate each chunk separately. + * + * We need one key to register the location of the header, and we need + * another key to track the location of the message queue. + */ + shm_toc_initialize_estimator(&e); + shm_toc_estimate_chunk(&e, sizeof(ApplyBgworkerShared)); + shm_toc_estimate_chunk(&e, (Size) queue_size); + + shm_toc_estimate_keys(&e, 1 + 1); + segsize = shm_toc_estimate(&e); + + /* Create the shared memory segment and establish a table of contents. */ + seg = dsm_create(shm_toc_estimate(&e), 0); + toc = shm_toc_create(PG_LOGICAL_APPLY_SHM_MAGIC, dsm_segment_address(seg), + segsize); + + /* Set up the header region. */ + pst = shm_toc_allocate(toc, sizeof(ApplyBgworkerShared)); + SpinLockInit(&pst->mutex); + pst->status = APPLY_BGWORKER_BUSY; + pst->server_version = walrcv_server_version(LogRepWorkerWalRcvConn); + pst->stream_xid = stream_xid; + pst->n = list_length(ApplyWorkersList) + 1; + + shm_toc_insert(toc, APPLY_BGWORKER_KEY_SHARED, pst); + + /* Set up one message queue per worker, plus one. */ + mq = shm_mq_create(shm_toc_allocate(toc, (Size) queue_size), + (Size) queue_size); + shm_toc_insert(toc, APPLY_BGWORKER_KEY_MQ, mq); + shm_mq_set_sender(mq, MyProc); + + /* Attach the queues. */ + wstate->mq_handle = shm_mq_attach(mq, seg, NULL); + + /* Return results to caller. */ + wstate->dsm_seg = seg; + wstate->pstate = pst; +} + +/* + * Start apply background worker process and allocate shared memory for it. + */ +static ApplyBgworkerState * +apply_bgworker_setup(void) +{ + MemoryContext oldcontext; + bool launched; + ApplyBgworkerState *wstate; + int napplyworkers; + + elog(DEBUG1, "setting up apply worker #%u", list_length(ApplyWorkersList) + 1); + + /* Check If there are free worker slot(s) */ + LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); + napplyworkers = logicalrep_apply_background_worker_count(MyLogicalRepWorker->subid); + LWLockRelease(LogicalRepWorkerLock); + if (napplyworkers >= max_apply_bgworkers_per_subscription) + return NULL; + + oldcontext = MemoryContextSwitchTo(ApplyContext); + + wstate = (ApplyBgworkerState *) palloc0(sizeof(ApplyBgworkerState)); + + /* Setup shared memory */ + apply_bgworker_setup_dsm(wstate); + + launched = logicalrep_worker_launch(MyLogicalRepWorker->dbid, + MySubscription->oid, + MySubscription->name, + MyLogicalRepWorker->userid, + InvalidOid, + dsm_segment_handle(wstate->dsm_seg)); + + if (launched) + { + /* Wait for worker to attach. */ + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_ATTACHED); + + ApplyWorkersList = lappend(ApplyWorkersList, wstate); + } + else + { + pfree(wstate); + wstate = NULL; + } + + MemoryContextSwitchTo(oldcontext); + + return wstate; +} + +/* + * Send the data to the specified apply background worker via shared-memory queue. + */ +void +apply_bgworker_send_data(ApplyBgworkerState *wstate, Size nbytes, const void *data) +{ + shm_mq_result result; + + result = shm_mq_send(wstate->mq_handle, nbytes, data, false, true); + + if (result != SHM_MQ_SUCCESS) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("could not send tuples to shared-memory queue"))); +} + +/* + * Wait until the status of apply background worker reaches the + * 'wait_for_status' + */ +void +apply_bgworker_wait_for(ApplyBgworkerState *wstate, + ApplyBgworkerStatus wait_for_status) +{ + for (;;) + { + char status; + + SpinLockAcquire(&wstate->pstate->mutex); + status = wstate->pstate->status; + SpinLockRelease(&wstate->pstate->mutex); + + /* Done if already in correct status. */ + if (status == wait_for_status) + break; + + /* If any workers (or the postmaster) have died, we have failed. */ + if (status == APPLY_BGWORKER_EXIT) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("background worker %u failed to apply transaction %u", + wstate->pstate->n, wstate->pstate->stream_xid))); + + /* Wait to be signalled. */ + WaitLatch(MyLatch, WL_LATCH_SET | WL_EXIT_ON_PM_DEATH, 0, + WAIT_EVENT_LOGICAL_APPLY_WORKER_STATE_CHANGE); + + /* Reset the latch so we don't spin. */ + ResetLatch(MyLatch); + + /* An interrupt may have occurred while we were waiting. */ + CHECK_FOR_INTERRUPTS(); + } +} + +/* + * Check the status of workers and report an error if any apply background + * worker has exited unexpectedly. + * + * Exit if any relation is not in the READY state and if any worker is handling + * the streaming transaction at the same time. Because for streaming + * transactions that is being applied in apply background worker, we cannot + * decide whether to apply the change for a relation that is not in the READY + * state (see should_apply_changes_for_rel) as we won't know remote_final_lsn + * by that time. + */ +void +apply_bgworker_check_status(void) +{ + ListCell *lc; + + if (am_apply_bgworker() || MySubscription->stream != SUBSTREAM_APPLY) + return; + + foreach(lc, ApplyWorkersList) + { + ApplyBgworkerState *wstate = (ApplyBgworkerState *) lfirst(lc); + + /* + * We don't lock here as in the worst case we will just detect the + * failure of worker a bit later. + */ + if (wstate->pstate->status == APPLY_BGWORKER_EXIT) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("background worker %u exited unexpectedly", + wstate->pstate->n))); + } + + if (list_length(ApplyWorkersFreeList) != list_length(ApplyWorkersList) && + !AllTablesyncsReady()) + { + ereport(LOG, + (errmsg("logical replication apply workers for subscription \"%s\" will restart", + MySubscription->name), + errdetail("Cannot handle streamed replication transaction by apply " + "bgworkers until all tables are synchronized"))); + + proc_exit(0); + } +} + +/* Set the apply background worker status */ +void +apply_bgworker_set_status(ApplyBgworkerStatus status) +{ + if (!am_apply_bgworker()) + return; + + elog(DEBUG1, "[Apply BGW #%u] set status to %d", MyParallelState->n, status); + + SpinLockAcquire(&MyParallelState->mutex); + MyParallelState->status = status; + SpinLockRelease(&MyParallelState->mutex); +} + +/* + * Define a savepoint for a subxact in apply background worker if needed. + * + * Inside apply background worker we can figure out that new subtransaction was + * started if new change arrived with different xid. In that case we can define + * named savepoint, so that we were able to commit/rollback it separately + * later. + * Special case is if the first change comes from subtransaction, then + * we check that current_xid differs from stream_xid. + */ +void +apply_bgworker_subxact_info_add(TransactionId current_xid) +{ + if (current_xid != stream_xid && + !list_member_int(subxactlist, (int) current_xid)) + { + MemoryContext oldctx; + char spname[MAXPGPATH]; + + snprintf(spname, MAXPGPATH, "savepoint_for_xid_%u", current_xid); + + elog(DEBUG1, "[Apply BGW #%u] defining savepoint %s", + MyParallelState->n, spname); + + DefineSavepoint(spname); + CommitTransactionCommand(); + + oldctx = MemoryContextSwitchTo(ApplyContext); + subxactlist = lappend_int(subxactlist, (int) current_xid); + MemoryContextSwitchTo(oldctx); + } +} diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c index aa2427ba73..00fdde0830 100644 --- a/src/backend/replication/logical/decode.c +++ b/src/backend/replication/logical/decode.c @@ -651,9 +651,10 @@ DecodeCommit(LogicalDecodingContext *ctx, XLogRecordBuffer *buf, { for (i = 0; i < parsed->nsubxacts; i++) { - ReorderBufferForget(ctx->reorder, parsed->subxacts[i], buf->origptr); + ReorderBufferForget(ctx->reorder, parsed->subxacts[i], buf->origptr, + commit_time); } - ReorderBufferForget(ctx->reorder, xid, buf->origptr); + ReorderBufferForget(ctx->reorder, xid, buf->origptr, commit_time); return; } @@ -821,10 +822,11 @@ DecodeAbort(LogicalDecodingContext *ctx, XLogRecordBuffer *buf, for (i = 0; i < parsed->nsubxacts; i++) { ReorderBufferAbort(ctx->reorder, parsed->subxacts[i], - buf->record->EndRecPtr); + buf->record->EndRecPtr, abort_time); } - ReorderBufferAbort(ctx->reorder, xid, buf->record->EndRecPtr); + ReorderBufferAbort(ctx->reorder, xid, buf->record->EndRecPtr, + abort_time); } /* update the decoding stats */ diff --git a/src/backend/replication/logical/launcher.c b/src/backend/replication/logical/launcher.c index 2bdab53e19..dd68651314 100644 --- a/src/backend/replication/logical/launcher.c +++ b/src/backend/replication/logical/launcher.c @@ -54,6 +54,7 @@ int max_logical_replication_workers = 4; int max_sync_workers_per_subscription = 2; +int max_apply_bgworkers_per_subscription = 3; LogicalRepWorker *MyLogicalRepWorker = NULL; @@ -73,6 +74,7 @@ static void logicalrep_launcher_onexit(int code, Datum arg); static void logicalrep_worker_onexit(int code, Datum arg); static void logicalrep_worker_detach(void); static void logicalrep_worker_cleanup(LogicalRepWorker *worker); +static void logicalrep_worker_stop_internal(LogicalRepWorker *worker); static bool on_commit_launcher_wakeup = false; @@ -223,6 +225,10 @@ logicalrep_worker_find(Oid subid, Oid relid, bool only_running) { LogicalRepWorker *w = &LogicalRepCtx->workers[i]; + /* We only need main apply worker or table sync worker here */ + if (w->subworker) + continue; + if (w->in_use && w->subid == subid && w->relid == relid && (!only_running || w->proc)) { @@ -259,11 +265,11 @@ logicalrep_workers_find(Oid subid, bool only_running) } /* - * Start new apply background worker, if possible. + * Start new background worker, if possible. */ -void +bool logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, Oid userid, - Oid relid) + Oid relid, dsm_handle subworker_dsm) { BackgroundWorker bgw; BackgroundWorkerHandle *bgw_handle; @@ -273,6 +279,10 @@ logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, Oid userid, LogicalRepWorker *worker = NULL; int nsyncworkers; TimestampTz now; + bool is_subworker = (subworker_dsm != DSM_HANDLE_INVALID); + + /* We don't support table sync in subworker */ + Assert(!(is_subworker && OidIsValid(relid))); ereport(DEBUG1, (errmsg_internal("starting logical replication worker for subscription \"%s\"", @@ -350,7 +360,7 @@ retry: if (OidIsValid(relid) && nsyncworkers >= max_sync_workers_per_subscription) { LWLockRelease(LogicalRepWorkerLock); - return; + return false; } /* @@ -364,7 +374,7 @@ retry: (errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED), errmsg("out of logical replication worker slots"), errhint("You might need to increase max_logical_replication_workers."))); - return; + return false; } /* Prepare the worker slot. */ @@ -379,6 +389,7 @@ retry: worker->relstate = SUBREL_STATE_UNKNOWN; worker->relstate_lsn = InvalidXLogRecPtr; worker->stream_fileset = NULL; + worker->subworker = is_subworker; worker->last_lsn = InvalidXLogRecPtr; TIMESTAMP_NOBEGIN(worker->last_send_time); TIMESTAMP_NOBEGIN(worker->last_recv_time); @@ -396,19 +407,30 @@ retry: BGWORKER_BACKEND_DATABASE_CONNECTION; bgw.bgw_start_time = BgWorkerStart_RecoveryFinished; snprintf(bgw.bgw_library_name, BGW_MAXLEN, "postgres"); - snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyWorkerMain"); + + if (!is_subworker) + snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyWorkerMain"); + else + snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyBgworkerMain"); + if (OidIsValid(relid)) snprintf(bgw.bgw_name, BGW_MAXLEN, "logical replication worker for subscription %u sync %u", subid, relid); - else + else if (!is_subworker) snprintf(bgw.bgw_name, BGW_MAXLEN, "logical replication worker for subscription %u", subid); + else + snprintf(bgw.bgw_name, BGW_MAXLEN, + "logical replication background apply worker for subscription %u ", subid); snprintf(bgw.bgw_type, BGW_MAXLEN, "logical replication worker"); bgw.bgw_restart_time = BGW_NEVER_RESTART; bgw.bgw_notify_pid = MyProcPid; bgw.bgw_main_arg = Int32GetDatum(slot); + if (is_subworker) + memcpy(bgw.bgw_extra, &subworker_dsm, sizeof(dsm_handle)); + if (!RegisterDynamicBackgroundWorker(&bgw, &bgw_handle)) { /* Failed to start worker, so clean up the worker slot. */ @@ -421,11 +443,13 @@ retry: (errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED), errmsg("out of background worker slots"), errhint("You might need to increase max_worker_processes."))); - return; + return false; } /* Now wait until it attaches. */ WaitForReplicationWorkerAttach(worker, generation, bgw_handle); + + return true; } /* @@ -436,7 +460,6 @@ void logicalrep_worker_stop(Oid subid, Oid relid) { LogicalRepWorker *worker; - uint16 generation; LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); @@ -449,6 +472,22 @@ logicalrep_worker_stop(Oid subid, Oid relid) return; } + logicalrep_worker_stop_internal(worker); + + LWLockRelease(LogicalRepWorkerLock); +} + +/* + * Workhorse for logicalrep_worker_stop() and logicalrep_worker_detach(). Stop + * the worker and wait for it to die. + */ +static void +logicalrep_worker_stop_internal(LogicalRepWorker *worker) +{ + uint16 generation; + + Assert(LWLockHeldByMe(LogicalRepWorkerLock)); + /* * Remember which generation was our worker so we can check if what we see * is still the same one. @@ -485,10 +524,7 @@ logicalrep_worker_stop(Oid subid, Oid relid) * different, meaning that a different worker has taken the slot. */ if (!worker->in_use || worker->generation != generation) - { - LWLockRelease(LogicalRepWorkerLock); return; - } /* Worker has assigned proc, so it has started. */ if (worker->proc) @@ -522,8 +558,6 @@ logicalrep_worker_stop(Oid subid, Oid relid) LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); } - - LWLockRelease(LogicalRepWorkerLock); } /* @@ -599,6 +633,29 @@ logicalrep_worker_attach(int slot) static void logicalrep_worker_detach(void) { + /* + * If we are the main apply worker, stop all the apply background workers + * we started before. + */ + if (!MyLogicalRepWorker->subworker) + { + List *workers; + ListCell *lc; + + LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); + + workers = logicalrep_workers_find(MyLogicalRepWorker->subid, true); + foreach(lc, workers) + { + LogicalRepWorker *w = (LogicalRepWorker *) lfirst(lc); + + if (w->subworker) + logicalrep_worker_stop_internal(w); + } + + LWLockRelease(LogicalRepWorkerLock); + } + /* Block concurrent access. */ LWLockAcquire(LogicalRepWorkerLock, LW_EXCLUSIVE); @@ -621,6 +678,7 @@ logicalrep_worker_cleanup(LogicalRepWorker *worker) worker->userid = InvalidOid; worker->subid = InvalidOid; worker->relid = InvalidOid; + worker->subworker = false; } /* @@ -679,6 +737,30 @@ logicalrep_sync_worker_count(Oid subid) return res; } +/* + * Count the number of registered (not necessarily running) apply background + * worker for a subscription. + */ +int +logicalrep_apply_background_worker_count(Oid subid) +{ + int i; + int res = 0; + + Assert(LWLockHeldByMe(LogicalRepWorkerLock)); + + /* Search for attached worker for a given subscription id. */ + for (i = 0; i < max_logical_replication_workers; i++) + { + LogicalRepWorker *w = &LogicalRepCtx->workers[i]; + + if (w->subid == subid && w->subworker) + res++; + } + + return res; +} + /* * ApplyLauncherShmemSize * Compute space needed for replication launcher shared memory @@ -868,7 +950,7 @@ ApplyLauncherMain(Datum main_arg) wait_time = wal_retrieve_retry_interval; logicalrep_worker_launch(sub->dbid, sub->oid, sub->name, - sub->owner, InvalidOid); + sub->owner, InvalidOid, DSM_HANDLE_INVALID); } } diff --git a/src/backend/replication/logical/origin.c b/src/backend/replication/logical/origin.c index 21937ab2d3..9273011b0a 100644 --- a/src/backend/replication/logical/origin.c +++ b/src/backend/replication/logical/origin.c @@ -1063,12 +1063,21 @@ ReplicationOriginExitCleanup(int code, Datum arg) * array doesn't have to be searched when calling * replorigin_session_advance(). * - * Obviously only one such cached origin can exist per process and the current + * Normally only one such cached origin can exist per process and the current * cached value can only be set again after the previous value is torn down * with replorigin_session_reset(). + * + * However, If must_acquire is false, we allow process to get the slot which is + * already acquired by other process. It's safe because 1) The only caller + * (apply background workers) will maintain the commit order by allowing only + * one process to commit at a time, so no two workers will be operating on the + * same origin at the same time (see comments in logical/worker.c). 2) Even + * though we try to advance the session's origin concurrently, it's safe to do + * so as we change/advance the session_origin LSNs under replicate_state + * LWLock. */ void -replorigin_session_setup(RepOriginId node) +replorigin_session_setup(RepOriginId node, bool must_acquire) { static bool registered_cleanup; int i; @@ -1110,7 +1119,7 @@ replorigin_session_setup(RepOriginId node) if (curstate->roident != node) continue; - else if (curstate->acquired_by != 0) + else if (curstate->acquired_by != 0 && must_acquire) { ereport(ERROR, (errcode(ERRCODE_OBJECT_IN_USE), @@ -1141,7 +1150,14 @@ replorigin_session_setup(RepOriginId node) Assert(session_replication_state->roident != InvalidRepOriginId); - session_replication_state->acquired_by = MyProcPid; + if (must_acquire) + session_replication_state->acquired_by = MyProcPid; + else if (session_replication_state->acquired_by == 0) + ereport(ERROR, + (errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED), + errmsg("could not find correct replication state slot for replication origin with OID %u for apply background worker", + node), + errhint("There is no replication state slot set by its main apply worker."))); LWLockRelease(ReplicationOriginLock); @@ -1321,7 +1337,7 @@ pg_replication_origin_session_setup(PG_FUNCTION_ARGS) name = text_to_cstring((text *) DatumGetPointer(PG_GETARG_DATUM(0))); origin = replorigin_by_name(name, false); - replorigin_session_setup(origin); + replorigin_session_setup(origin, true); replorigin_session_origin = origin; diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c index ff8513e2d2..a888038eb4 100644 --- a/src/backend/replication/logical/proto.c +++ b/src/backend/replication/logical/proto.c @@ -1166,28 +1166,47 @@ logicalrep_read_stream_commit(StringInfo in, LogicalRepCommitData *commit_data) */ void logicalrep_write_stream_abort(StringInfo out, TransactionId xid, - TransactionId subxid) + ReorderBufferTXN *txn, XLogRecPtr abort_lsn) { pq_sendbyte(out, LOGICAL_REP_MSG_STREAM_ABORT); - Assert(TransactionIdIsValid(xid) && TransactionIdIsValid(subxid)); + Assert(TransactionIdIsValid(xid) && TransactionIdIsValid(txn->xid)); /* transaction ID */ pq_sendint32(out, xid); - pq_sendint32(out, subxid); + pq_sendint32(out, txn->xid); + pq_sendint64(out, abort_lsn); + pq_sendint64(out, txn->xact_time.abort_time); } /* * Read STREAM ABORT from the output stream. */ void -logicalrep_read_stream_abort(StringInfo in, TransactionId *xid, - TransactionId *subxid) +logicalrep_read_stream_abort(StringInfo in, + LogicalRepStreamAbortData *abort_data, + bool include_abort_lsn) { - Assert(xid && subxid); + Assert(abort_data); + + abort_data->xid = pq_getmsgint(in, 4); + abort_data->subxid = pq_getmsgint(in, 4); - *xid = pq_getmsgint(in, 4); - *subxid = pq_getmsgint(in, 4); + /* + * If the version of the publisher is lower than the version of the + * subscriber, it may not support sending these two fields, so only take + * these fields when include_abort_lsn is true. + */ + if (include_abort_lsn) + { + abort_data->abort_lsn = pq_getmsgint64(in); + abort_data->abort_time = pq_getmsgint64(in); + } + else + { + abort_data->abort_lsn = InvalidXLogRecPtr; + abort_data->abort_time = 0; + } } /* diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c index 8da5f9089c..3d2bbdb38d 100644 --- a/src/backend/replication/logical/reorderbuffer.c +++ b/src/backend/replication/logical/reorderbuffer.c @@ -2826,7 +2826,8 @@ ReorderBufferFinishPrepared(ReorderBuffer *rb, TransactionId xid, * disk. */ void -ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn) +ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn, + TimestampTz abort_time) { ReorderBufferTXN *txn; @@ -2837,6 +2838,8 @@ ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn) if (txn == NULL) return; + txn->xact_time.abort_time = abort_time; + /* For streamed transactions notify the remote node about the abort. */ if (rbtxn_is_streamed(txn)) { @@ -2911,7 +2914,8 @@ ReorderBufferAbortOld(ReorderBuffer *rb, TransactionId oldestRunningXid) * to this xid might re-create the transaction incompletely. */ void -ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn) +ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn, + TimestampTz abort_time) { ReorderBufferTXN *txn; @@ -2922,6 +2926,8 @@ ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn) if (txn == NULL) return; + txn->xact_time.abort_time = abort_time; + /* For streamed transactions notify the remote node about the abort. */ if (rbtxn_is_streamed(txn)) rb->stream_abort(rb, txn, lsn); diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c index 670c6fcada..8ffba7e2e5 100644 --- a/src/backend/replication/logical/tablesync.c +++ b/src/backend/replication/logical/tablesync.c @@ -568,7 +568,8 @@ process_syncing_tables_for_apply(XLogRecPtr current_lsn) MySubscription->oid, MySubscription->name, MyLogicalRepWorker->userid, - rstate->relid); + rstate->relid, + DSM_HANDLE_INVALID); hentry->last_start_time = now; } } @@ -589,6 +590,9 @@ process_syncing_tables_for_apply(XLogRecPtr current_lsn) void process_syncing_tables(XLogRecPtr current_lsn) { + if (MyLogicalRepWorker->subworker) + return; + if (am_tablesync_worker()) process_syncing_tables_for_sync(current_lsn); else @@ -1273,7 +1277,7 @@ LogicalRepSyncTableStart(XLogRecPtr *origin_startpos) * time this tablesync was launched. */ originid = replorigin_by_name(originname, false); - replorigin_session_setup(originid); + replorigin_session_setup(originid, true); replorigin_session_origin = originid; *origin_startpos = replorigin_session_get_progress(false); @@ -1384,7 +1388,7 @@ LogicalRepSyncTableStart(XLogRecPtr *origin_startpos) true /* go backward */ , true /* WAL log */ ); UnlockRelationOid(ReplicationOriginRelationId, RowExclusiveLock); - replorigin_session_setup(originid); + replorigin_session_setup(originid, true); replorigin_session_origin = originid; } else diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c index fc210a9e7b..3f8dea0fc1 100644 --- a/src/backend/replication/logical/worker.c +++ b/src/backend/replication/logical/worker.c @@ -22,8 +22,25 @@ * STREAMED TRANSACTIONS * --------------------- * Streamed transactions (large transactions exceeding a memory limit on the - * upstream) are not applied immediately, but instead, the data is written - * to temporary files and then applied at once when the final commit arrives. + * upstream) are applied via one of two approaches. + * + * 1) Separate background workers + * + * Assign a new apply background worker (if available) as soon as the xact's + * first stream is received and the main apply worker will send changes to this + * new worker via shared memory. We keep this worker assigned till the + * transaction commit is received and also wait for the worker to finish at + * commit. This preserves commit ordering and avoids writing to and reading + * from file in most cases. We still need to spill if there is no worker + * available. We also need to allow stream_stop to complete by the background + * worker to avoid deadlocks because T-1's current stream of changes can update + * rows in conflicting order with T-2's next stream of changes. + * + * 2) Write to temporary files and apply when the final commit arrives + * + * If no worker is available to handle streamed transaction, the data is + * written to temporary files and then applied at once when the final commit + * arrives. * * Unlike the regular (non-streamed) case, handling streamed transactions has * to handle aborts of both the toplevel transaction and subtransactions. This @@ -219,20 +236,7 @@ typedef struct ApplyExecutionData PartitionTupleRouting *proute; /* partition routing info */ } ApplyExecutionData; -/* Struct for saving and restoring apply errcontext information */ -typedef struct ApplyErrorCallbackArg -{ - LogicalRepMsgType command; /* 0 if invalid */ - LogicalRepRelMapEntry *rel; - - /* Remote node information */ - int remote_attnum; /* -1 if invalid */ - TransactionId remote_xid; - XLogRecPtr finish_lsn; - char *origin_name; -} ApplyErrorCallbackArg; - -static ApplyErrorCallbackArg apply_error_callback_arg = +ApplyErrorCallbackArg apply_error_callback_arg = { .command = 0, .rel = NULL, @@ -242,7 +246,7 @@ static ApplyErrorCallbackArg apply_error_callback_arg = .origin_name = NULL, }; -static MemoryContext ApplyMessageContext = NULL; +MemoryContext ApplyMessageContext = NULL; MemoryContext ApplyContext = NULL; /* per stream context for streaming transactions */ @@ -251,27 +255,38 @@ static MemoryContext LogicalStreamingContext = NULL; WalReceiverConn *LogRepWorkerWalRcvConn = NULL; Subscription *MySubscription = NULL; -static bool MySubscriptionValid = false; +bool MySubscriptionValid = false; bool in_remote_transaction = false; static XLogRecPtr remote_final_lsn = InvalidXLogRecPtr; /* fields valid only when processing streamed transaction */ -static bool in_streamed_transaction = false; +bool in_streamed_transaction = false; + +TransactionId stream_xid = InvalidTransactionId; +static ApplyBgworkerState *stream_apply_worker = NULL; + +/* check if we are applying the transaction in apply background worker */ +#define apply_bgworker_active() (in_streamed_transaction && stream_apply_worker != NULL) -static TransactionId stream_xid = InvalidTransactionId; +/* + * The number of changes during one streaming block (only for apply background + * workers) + */ +static uint32 nchanges = 0; /* * We enable skipping all data modification changes (INSERT, UPDATE, etc.) for * the subscription if the remote transaction's finish LSN matches the subskiplsn. * Once we start skipping changes, we don't stop it until we skip all changes of * the transaction even if pg_subscription is updated and MySubscription->skiplsn - * gets changed or reset during that. Also, in streaming transaction cases, we - * don't skip receiving and spooling the changes since we decide whether or not + * gets changed or reset during that. Also, in streaming transaction cases (streaming = on), + * we don't skip receiving and spooling the changes since we decide whether or not * to skip applying the changes when starting to apply changes. The subskiplsn is * cleared after successfully skipping the transaction or applying non-empty * transaction. The latter prevents the mistakenly specified subskiplsn from - * being left. + * being left. Note that we cannot skip the streaming transaction in apply mode, + * because we cannot get the finish LSN before applying the changes. */ static XLogRecPtr skip_xact_finish_lsn = InvalidXLogRecPtr; #define is_skipping_changes() (unlikely(!XLogRecPtrIsInvalid(skip_xact_finish_lsn))) @@ -324,9 +339,6 @@ static void maybe_reread_subscription(void); static void DisableSubscriptionAndExit(void); -/* prototype needed because of stream_commit */ -static void apply_dispatch(StringInfo s); - static void apply_handle_commit_internal(LogicalRepCommitData *commit_data); static void apply_handle_insert_internal(ApplyExecutionData *edata, ResultRelInfo *relinfo, @@ -359,7 +371,6 @@ static void stop_skipping_changes(void); static void clear_subscription_skip_lsn(XLogRecPtr finish_lsn); /* Functions for apply error callback */ -static void apply_error_callback(void *arg); static inline void set_apply_error_context_xact(TransactionId xid, XLogRecPtr lsn); static inline void reset_apply_error_context_info(void); @@ -426,41 +437,76 @@ end_replication_step(void) } /* - * Handle streamed transactions. + * Handle streamed transactions for both main apply worker and apply background + * worker. * - * If in streaming mode (receiving a block of streamed transaction), we - * simply redirect it to a file for the proper toplevel transaction. + * In streaming case (receiving a block of streamed transaction), for + * SUBSTREAM_ON mode, we simply redirect it to a file for the proper toplevel + * transaction, and for SUBSTREAM_APPLY mode, we send the changes to background + * apply worker (LOGICAL_REP_MSG_RELATION or LOGICAL_REP_MSG_TYPE changes will + * also be applied in main apply worker). * - * Returns true for streamed transactions, false otherwise (regular mode). + * For non-streamed transactions, returns false; + * For streamed transactions, returns true if in main apply worker (except we + * apply streamed transaction in "apply" mode and address + * LOGICAL_REP_MSG_RELATION or LOGICAL_REP_MSG_TYPE changes), false otherwise. */ static bool handle_streamed_transaction(LogicalRepMsgType action, StringInfo s) { - TransactionId xid; + TransactionId current_xid = InvalidTransactionId; /* not in streaming mode */ - if (!in_streamed_transaction) + if (!(in_streamed_transaction || am_apply_bgworker())) return false; - Assert(stream_fd != NULL); Assert(TransactionIdIsValid(stream_xid)); /* * We should have received XID of the subxact as the first part of the * message, so extract it. */ - xid = pq_getmsgint(s, 4); + current_xid = pq_getmsgint(s, 4); - if (!TransactionIdIsValid(xid)) + if (!TransactionIdIsValid(current_xid)) ereport(ERROR, (errcode(ERRCODE_PROTOCOL_VIOLATION), errmsg_internal("invalid transaction ID in streamed replication transaction"))); - /* Add the new subxact to the array (unless already there). */ - subxact_info_add(xid); + if (am_apply_bgworker()) + { + /* define a savepoint for a subxact if needed. */ + apply_bgworker_subxact_info_add(current_xid); - /* write the change to the current file */ - stream_write_change(action, s); + return false; + } + else if (apply_bgworker_active()) + { + /* + * If we decided to apply the changes of this transaction in an apply + * background worker, pass the data to the worker. + */ + apply_bgworker_send_data(stream_apply_worker, s->len, s->data); + nchanges += 1; + + /* + * XXX The publisher side doesn't always send relation/type update + * messages after the streaming transaction, so also update the + * relation/type in main apply worker here. See function + * cleanup_rel_sync_cache. + */ + if (action == LOGICAL_REP_MSG_RELATION || + action == LOGICAL_REP_MSG_TYPE) + return false; + } + else + { + /* Add the new subxact to the array (unless already there). */ + subxact_info_add(current_xid); + + /* write the change to the current file */ + stream_write_change(action, s); + } return true; } @@ -844,6 +890,9 @@ apply_handle_commit(StringInfo s) apply_handle_commit_internal(&commit_data); + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); + /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(commit_data.end_lsn); @@ -898,7 +947,9 @@ apply_handle_prepare_internal(LogicalRepPreparedTxnData *prepare_data) * BeginTransactionBlock is necessary to balance the EndTransactionBlock * called within the PrepareTransactionBlock below. */ - BeginTransactionBlock(); + if (!IsTransactionBlock()) + BeginTransactionBlock(); + CommitTransactionCommand(); /* Completes the preceding Begin command. */ /* @@ -950,6 +1001,9 @@ apply_handle_prepare(StringInfo s) in_remote_transaction = false; + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); + /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(prepare_data.end_lsn); @@ -1064,10 +1118,6 @@ apply_handle_rollback_prepared(StringInfo s) /* * Handle STREAM PREPARE. - * - * Logic is in two parts: - * 1. Replay all the spooled operations - * 2. Mark the transaction as prepared */ static void apply_handle_stream_prepare(StringInfo s) @@ -1088,24 +1138,70 @@ apply_handle_stream_prepare(StringInfo s) logicalrep_read_stream_prepare(s, &prepare_data); set_apply_error_context_xact(prepare_data.xid, prepare_data.prepare_lsn); - elog(DEBUG1, "received prepare for streamed transaction %u", prepare_data.xid); + if (am_apply_bgworker()) + { + /* Mark the transaction as prepared. */ + apply_handle_prepare_internal(&prepare_data); - /* Replay all the spooled operations. */ - apply_spooled_messages(prepare_data.xid, prepare_data.prepare_lsn); + CommitTransactionCommand(); - /* Mark the transaction as prepared. */ - apply_handle_prepare_internal(&prepare_data); + pgstat_report_stat(false); - CommitTransactionCommand(); + list_free(subxactlist); + subxactlist = NIL; - pgstat_report_stat(false); + apply_bgworker_set_status(APPLY_BGWORKER_FINISHED); + } + else + { + ApplyBgworkerState *wstate = apply_bgworker_find_or_start(prepare_data.xid, false); - store_flush_position(prepare_data.end_lsn); + elog(DEBUG1, "received prepare for streamed transaction %u", + prepare_data.xid); + + /* + * This is the main apply worker. Check if we are processing this + * transaction in a apply background worker. + */ + if (wstate) + { + apply_bgworker_send_data(wstate, s->len, s->data); + + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_FINISHED); + apply_bgworker_free(wstate); + + pgstat_report_stat(false); + store_flush_position(prepare_data.end_lsn); + } + else + { + /* + * This is the main apply worker and the transaction has been + * serialized to file, replay all the spooled operations. + */ + apply_spooled_messages(prepare_data.xid, prepare_data.prepare_lsn); + + /* Mark the transaction as prepared. */ + apply_handle_prepare_internal(&prepare_data); + + CommitTransactionCommand(); + + pgstat_report_stat(false); + + store_flush_position(prepare_data.end_lsn); + + in_remote_transaction = false; + + /* unlink the files with serialized changes and subxact info. */ + stream_cleanup_files(MyLogicalRepWorker->subid, prepare_data.xid); + } + } in_remote_transaction = false; + stream_apply_worker = NULL; - /* unlink the files with serialized changes and subxact info. */ - stream_cleanup_files(MyLogicalRepWorker->subid, prepare_data.xid); + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(prepare_data.end_lsn); @@ -1155,15 +1251,6 @@ apply_handle_stream_start(StringInfo s) (errcode(ERRCODE_PROTOCOL_VIOLATION), errmsg_internal("duplicate STREAM START message"))); - /* - * Start a transaction on stream start, this transaction will be committed - * on the stream stop unless it is a tablesync worker in which case it - * will be committed after processing all the messages. We need the - * transaction for handling the buffile, used for serializing the - * streaming data and subxact info. - */ - begin_replication_step(); - /* notify handle methods we're processing a remote transaction */ in_streamed_transaction = true; @@ -1177,36 +1264,92 @@ apply_handle_stream_start(StringInfo s) set_apply_error_context_xact(stream_xid, InvalidXLogRecPtr); + if (am_apply_bgworker()) + { + /* + * Make sure the handle apply_dispatch methods are aware we're in a + * remote transaction. + */ + in_remote_transaction = true; + + /* If we are in a apply background worker, begin the transaction */ + AcceptInvalidationMessages(); + maybe_reread_subscription(); + + StartTransactionCommand(); + BeginTransactionBlock(); + CommitTransactionCommand(); + + return; + } + /* - * Initialize the worker's stream_fileset if we haven't yet. This will be - * used for the entire duration of the worker so create it in a permanent - * context. We create this on the very first streaming message from any - * transaction and then use it for this and other streaming transactions. - * Now, we could create a fileset at the start of the worker as well but - * then we won't be sure that it will ever be used. + * This is the main apply worker. Check if there is any free apply + * background worker we can use to process this transaction. */ - if (MyLogicalRepWorker->stream_fileset == NULL) + stream_apply_worker = apply_bgworker_find_or_start(stream_xid, first_segment); + + if (stream_apply_worker) { - MemoryContext oldctx; + /* + * If we have found a free worker or if we are already applying this + * transaction in an apply background worker, then we pass the data to + * that worker. + */ + if (first_segment) + apply_bgworker_send_data(stream_apply_worker, s->len, s->data); - oldctx = MemoryContextSwitchTo(ApplyContext); + nchanges = 0; + elog(DEBUG1, "starting streaming of xid %u", stream_xid); + } - MyLogicalRepWorker->stream_fileset = palloc(sizeof(FileSet)); - FileSetInit(MyLogicalRepWorker->stream_fileset); + /* + * If no worker is available for the first stream start, we start to + * serialize all the changes of the transaction. + */ + else + { + /* + * Start a transaction on stream start, this transaction will be + * committed on the stream stop unless it is a tablesync worker in + * which case it will be committed after processing all the messages. + * We need the transaction for handling the buffile, used for + * serializing the streaming data and subxact info. + */ + begin_replication_step(); - MemoryContextSwitchTo(oldctx); - } + /* + * Initialize the worker's stream_fileset if we haven't yet. This will + * be used for the entire duration of the worker so create it in a + * permanent context. We create this on the very first streaming + * message from any transaction and then use it for this and other + * streaming transactions. Now, we could create a fileset at the start + * of the worker as well but then we won't be sure that it will ever + * be used. + */ + if (MyLogicalRepWorker->stream_fileset == NULL) + { + MemoryContext oldctx; - /* open the spool file for this transaction */ - stream_open_file(MyLogicalRepWorker->subid, stream_xid, first_segment); + oldctx = MemoryContextSwitchTo(ApplyContext); - /* if this is not the first segment, open existing subxact file */ - if (!first_segment) - subxact_info_read(MyLogicalRepWorker->subid, stream_xid); + MyLogicalRepWorker->stream_fileset = palloc(sizeof(FileSet)); + FileSetInit(MyLogicalRepWorker->stream_fileset); - pgstat_report_activity(STATE_RUNNING, NULL); + MemoryContextSwitchTo(oldctx); + } - end_replication_step(); + /* open the spool file for this transaction */ + stream_open_file(MyLogicalRepWorker->subid, stream_xid, first_segment); + + /* if this is not the first segment, open existing subxact file */ + if (!first_segment) + subxact_info_read(MyLogicalRepWorker->subid, stream_xid); + + end_replication_step(); + } + + pgstat_report_activity(STATE_RUNNING, NULL); } /* @@ -1220,44 +1363,47 @@ apply_handle_stream_stop(StringInfo s) (errcode(ERRCODE_PROTOCOL_VIOLATION), errmsg_internal("STREAM STOP message without STREAM START"))); - /* - * Close the file with serialized changes, and serialize information about - * subxacts for the toplevel transaction. - */ - subxact_info_write(MyLogicalRepWorker->subid, stream_xid); - stream_close_file(); + if (apply_bgworker_active()) + { + char action = LOGICAL_REP_MSG_STREAM_STOP; - /* We must be in a valid transaction state */ - Assert(IsTransactionState()); + apply_bgworker_send_data(stream_apply_worker, 1, &action); + apply_bgworker_wait_for(stream_apply_worker, APPLY_BGWORKER_READY); - /* Commit the per-stream transaction */ - CommitTransactionCommand(); + elog(DEBUG1, "stopped streaming of xid %u, %u changes streamed", stream_xid, nchanges); + } + else + { + /* + * Close the file with serialized changes, and serialize information + * about subxacts for the toplevel transaction. + */ + subxact_info_write(MyLogicalRepWorker->subid, stream_xid); + stream_close_file(); - in_streamed_transaction = false; + /* We must be in a valid transaction state */ + Assert(IsTransactionState()); + + /* Commit the per-stream transaction */ + CommitTransactionCommand(); + + /* Reset per-stream context */ + MemoryContextReset(LogicalStreamingContext); + } - /* Reset per-stream context */ - MemoryContextReset(LogicalStreamingContext); + in_streamed_transaction = false; + stream_apply_worker = NULL; pgstat_report_activity(STATE_IDLE, NULL); reset_apply_error_context_info(); } /* - * Handle STREAM abort message. + * Handle STREAM ABORT message when the transaction was spilled to disk. */ static void -apply_handle_stream_abort(StringInfo s) +serialize_stream_abort(TransactionId xid, TransactionId subxid) { - TransactionId xid; - TransactionId subxid; - - if (in_streamed_transaction) - ereport(ERROR, - (errcode(ERRCODE_PROTOCOL_VIOLATION), - errmsg_internal("STREAM ABORT message without STREAM STOP"))); - - logicalrep_read_stream_abort(s, &xid, &subxid); - /* * If the two XIDs are the same, it's in fact abort of toplevel xact, so * just delete the files with serialized info. @@ -1339,8 +1485,137 @@ apply_handle_stream_abort(StringInfo s) end_replication_step(); CommitTransactionCommand(); } +} - reset_apply_error_context_info(); +/* + * Handle STREAM ABORT message. + */ +static void +apply_handle_stream_abort(StringInfo s) +{ + TransactionId xid; + TransactionId subxid; + LogicalRepStreamAbortData abort_data; + bool include_abort_lsn = false; + + if (in_streamed_transaction) + ereport(ERROR, + (errcode(ERRCODE_PROTOCOL_VIOLATION), + errmsg_internal("STREAM ABORT message without STREAM STOP"))); + + /* Check whether the publisher sends abort_lsn and abort_time. */ + if (am_apply_bgworker()) + include_abort_lsn = MyParallelState->server_version >= 150000; + + logicalrep_read_stream_abort(s, &abort_data, include_abort_lsn); + + xid = abort_data.xid; + subxid = abort_data.subxid; + + if (am_apply_bgworker()) + { + elog(DEBUG1, "[Apply BGW #%u] aborting current transaction xid=%u, subxid=%u", + MyParallelState->n, GetCurrentTransactionIdIfAny(), + GetCurrentSubTransactionId()); + + /* + * Update origin state so we can restart streaming from correct + * position in case of crash. + */ + if (include_abort_lsn) + { + replorigin_session_origin_lsn = abort_data.abort_lsn; + replorigin_session_origin_timestamp = abort_data.abort_time; + } + + /* + * If the two XIDs are the same, it's in fact abort of toplevel xact, + * so just free the subxactlist. + */ + if (subxid == xid) + { + set_apply_error_context_xact(subxid, InvalidXLogRecPtr); + + AbortCurrentTransaction(); + + EndTransactionBlock(false); + CommitTransactionCommand(); + + in_remote_transaction = false; + + list_free(subxactlist); + subxactlist = NIL; + + apply_bgworker_set_status(APPLY_BGWORKER_FINISHED); + } + else + { + /* + * OK, so it's a subxact. Rollback to the savepoint. + * + * We also need to read the subxactlist, determine the offset + * tracked for the subxact, and truncate the list. + */ + int i; + bool found = false; + char spname[MAXPGPATH]; + + set_apply_error_context_xact(subxid, InvalidXLogRecPtr); + snprintf(spname, MAXPGPATH, "savepoint_for_xid_%u", subxid); + + elog(DEBUG1, "[Apply BGW #%u] rolling back to savepoint %s", + MyParallelState->n, spname); + + for (i = list_length(subxactlist) - 1; i >= 0; i--) + { + xid = (TransactionId) list_nth_int(subxactlist, i); + if (xid == subxid) + { + found = true; + break; + } + } + + if (found) + { + RollbackToSavepoint(spname); + CommitTransactionCommand(); + subxactlist = list_truncate(subxactlist, i + 1); + } + + apply_bgworker_set_status(APPLY_BGWORKER_READY); + } + + reset_apply_error_context_info(); + } + else + { + ApplyBgworkerState *wstate = apply_bgworker_find_or_start(xid, false); + + /* + * This is the main apply worker. Check if we are processing this + * transaction in a apply background worker. + */ + if (wstate) + { + apply_bgworker_send_data(wstate, s->len, s->data); + + if (subxid == xid) + { + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_FINISHED); + apply_bgworker_free(wstate); + } + else + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_READY); + } + + /* + * We are in main apply worker and the transaction has been serialized + * to file. + */ + else + serialize_stream_abort(xid, subxid); + } } /* @@ -1462,40 +1737,6 @@ apply_spooled_messages(TransactionId xid, XLogRecPtr lsn) return; } -/* - * Handle STREAM COMMIT message. - */ -static void -apply_handle_stream_commit(StringInfo s) -{ - TransactionId xid; - LogicalRepCommitData commit_data; - - if (in_streamed_transaction) - ereport(ERROR, - (errcode(ERRCODE_PROTOCOL_VIOLATION), - errmsg_internal("STREAM COMMIT message without STREAM STOP"))); - - xid = logicalrep_read_stream_commit(s, &commit_data); - set_apply_error_context_xact(xid, commit_data.commit_lsn); - - elog(DEBUG1, "received commit for streamed transaction %u", xid); - - apply_spooled_messages(xid, commit_data.commit_lsn); - - apply_handle_commit_internal(&commit_data); - - /* unlink the files with serialized changes and subxact info */ - stream_cleanup_files(MyLogicalRepWorker->subid, xid); - - /* Process any tables that are being synchronized in parallel. */ - process_syncing_tables(commit_data.end_lsn); - - pgstat_report_activity(STATE_IDLE, NULL); - - reset_apply_error_context_info(); -} - /* * Helper function for apply_handle_commit and apply_handle_stream_commit. */ @@ -2445,11 +2686,105 @@ apply_handle_truncate(StringInfo s) end_replication_step(); } +/* + * Handle STREAM COMMIT message. + */ +static void +apply_handle_stream_commit(StringInfo s) +{ + LogicalRepCommitData commit_data; + TransactionId xid; + + if (in_streamed_transaction) + ereport(ERROR, + (errcode(ERRCODE_PROTOCOL_VIOLATION), + errmsg_internal("STREAM COMMIT message without STREAM STOP"))); + + xid = logicalrep_read_stream_commit(s, &commit_data); + set_apply_error_context_xact(xid, commit_data.commit_lsn); + + if (am_apply_bgworker()) + { + /* + * Update origin state so we can restart streaming from correct + * position in case of crash. + */ + replorigin_session_origin_lsn = commit_data.end_lsn; + replorigin_session_origin_timestamp = commit_data.committime; + + CommitTransactionCommand(); + EndTransactionBlock(false); + CommitTransactionCommand(); + + in_remote_transaction = false; + + pgstat_report_stat(false); + + list_free(subxactlist); + subxactlist = NIL; + + apply_bgworker_set_status(APPLY_BGWORKER_FINISHED); + } + else + { + /* + * This is the main apply worker. Check if we are processing this + * transaction in an apply background worker. + */ + ApplyBgworkerState *wstate = apply_bgworker_find_or_start(xid, false); + + elog(DEBUG1, "received commit for streamed transaction %u", xid); + + if (wstate) + { + /* Send commit message */ + apply_bgworker_send_data(wstate, s->len, s->data); + + /* Wait for apply background worker to finish */ + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_FINISHED); + + pgstat_report_stat(false); + store_flush_position(commit_data.end_lsn); + stop_skipping_changes(); + + apply_bgworker_free(wstate); + + /* + * The transaction is either non-empty or skipped, so we clear the + * subskiplsn. + */ + clear_subscription_skip_lsn(commit_data.commit_lsn); + } + else + { + /* + * This is the main apply worker and the transaction has been + * serialized to file, replay all the spooled operations. + */ + apply_spooled_messages(xid, commit_data.commit_lsn); + + apply_handle_commit_internal(&commit_data); + + /* unlink the files with serialized changes and subxact info */ + stream_cleanup_files(MyLogicalRepWorker->subid, xid); + } + } + + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); + + /* Process any tables that are being synchronized in parallel. */ + process_syncing_tables(commit_data.end_lsn); + + pgstat_report_activity(STATE_IDLE, NULL); + + reset_apply_error_context_info(); +} /* * Logical replication protocol message dispatcher. */ -static void +void apply_dispatch(StringInfo s) { LogicalRepMsgType action = pq_getmsgbyte(s); @@ -2618,6 +2953,10 @@ store_flush_position(XLogRecPtr remote_lsn) { FlushPosition *flushpos; + /* We only need to collect the LSN in main apply worker */ + if (am_apply_bgworker()) + return; + /* Need to do this in permanent context */ MemoryContextSwitchTo(ApplyContext); @@ -2632,7 +2971,7 @@ store_flush_position(XLogRecPtr remote_lsn) /* Update statistics of the worker. */ -static void +void UpdateWorkerStats(XLogRecPtr last_lsn, TimestampTz send_time, bool reply) { MyLogicalRepWorker->last_lsn = last_lsn; @@ -2794,6 +3133,9 @@ LogicalRepApplyLoop(XLogRecPtr last_received) AcceptInvalidationMessages(); maybe_reread_subscription(); + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); + /* Process any table synchronization changes. */ process_syncing_tables(last_received); } @@ -3095,7 +3437,7 @@ maybe_reread_subscription(void) /* * Callback from subscription syscache invalidation. */ -static void +void subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue) { MySubscriptionValid = false; @@ -3691,7 +4033,7 @@ ApplyWorkerMain(Datum main_arg) originid = replorigin_by_name(originname, true); if (!OidIsValid(originid)) originid = replorigin_create(originname); - replorigin_session_setup(originid); + replorigin_session_setup(originid, true); replorigin_session_origin = originid; origin_startpos = replorigin_session_get_progress(false); CommitTransactionCommand(); @@ -3738,7 +4080,7 @@ ApplyWorkerMain(Datum main_arg) options.proto.logical.publication_names = MySubscription->publications; options.proto.logical.binary = MySubscription->binary; - options.proto.logical.streaming = MySubscription->stream; + options.proto.logical.streaming = (MySubscription->stream != SUBSTREAM_OFF); options.proto.logical.twophase = false; if (!am_tablesync_worker()) @@ -3896,7 +4238,8 @@ clear_subscription_skip_lsn(XLogRecPtr finish_lsn) XLogRecPtr myskiplsn = MySubscription->skiplsn; bool started_tx = false; - if (likely(XLogRecPtrIsInvalid(myskiplsn))) + if (likely(XLogRecPtrIsInvalid(myskiplsn)) || + am_apply_bgworker()) return; if (!IsTransactionState()) @@ -3968,7 +4311,7 @@ clear_subscription_skip_lsn(XLogRecPtr finish_lsn) } /* Error callback to give more context info about the change being applied */ -static void +void apply_error_callback(void *arg) { ApplyErrorCallbackArg *errarg = &apply_error_callback_arg; diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c index 8deae57143..6ebfa24baf 100644 --- a/src/backend/replication/pgoutput/pgoutput.c +++ b/src/backend/replication/pgoutput/pgoutput.c @@ -1833,7 +1833,7 @@ pgoutput_stream_abort(struct LogicalDecodingContext *ctx, Assert(rbtxn_is_streamed(toptxn)); OutputPluginPrepareWrite(ctx, true); - logicalrep_write_stream_abort(ctx->out, toptxn->xid, txn->xid); + logicalrep_write_stream_abort(ctx->out, toptxn->xid, txn, abort_lsn); OutputPluginWrite(ctx, true); cleanup_rel_sync_cache(toptxn->xid, false); diff --git a/src/backend/utils/activity/wait_event.c b/src/backend/utils/activity/wait_event.c index 87c15b9c6f..ba781e6f08 100644 --- a/src/backend/utils/activity/wait_event.c +++ b/src/backend/utils/activity/wait_event.c @@ -388,6 +388,9 @@ pgstat_get_wait_ipc(WaitEventIPC w) case WAIT_EVENT_HASH_GROW_BUCKETS_REINSERT: event_name = "HashGrowBucketsReinsert"; break; + case WAIT_EVENT_LOGICAL_APPLY_WORKER_STATE_CHANGE: + event_name = "LogicalApplyWorkerStateChange"; + break; case WAIT_EVENT_LOGICAL_SYNC_DATA: event_name = "LogicalSyncData"; break; diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c index a7cc49898b..519efff1d2 100644 --- a/src/backend/utils/misc/guc.c +++ b/src/backend/utils/misc/guc.c @@ -3220,6 +3220,18 @@ static struct config_int ConfigureNamesInt[] = NULL, NULL, NULL }, + { + {"max_apply_bgworkers_per_subscription", + PGC_SIGHUP, + REPLICATION_SUBSCRIBERS, + gettext_noop("Maximum number of apply backgrand workers per subscription."), + NULL, + }, + &max_apply_bgworkers_per_subscription, + 3, 0, MAX_BACKENDS, + NULL, NULL, NULL + }, + { {"log_rotation_age", PGC_SIGHUP, LOGGING_WHERE, gettext_noop("Sets the amount of time to wait before forcing " diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample index 48ad80cf2e..b71340549d 100644 --- a/src/backend/utils/misc/postgresql.conf.sample +++ b/src/backend/utils/misc/postgresql.conf.sample @@ -360,6 +360,7 @@ #max_logical_replication_workers = 4 # taken from max_worker_processes # (change requires restart) #max_sync_workers_per_subscription = 2 # taken from max_logical_replication_workers +#max_apply_bgworkers_per_subscription = 3 # taken from max_logical_replication_workers #------------------------------------------------------------------------------ diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 7cc9c72e49..6f8b30abb0 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -4450,7 +4450,7 @@ getSubscriptions(Archive *fout) if (fout->remoteVersion >= 140000) appendPQExpBufferStr(query, " s.substream,\n"); else - appendPQExpBufferStr(query, " false AS substream,\n"); + appendPQExpBufferStr(query, " 'f' AS substream,\n"); if (fout->remoteVersion >= 150000) appendPQExpBufferStr(query, @@ -4580,8 +4580,10 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo) if (strcmp(subinfo->subbinary, "t") == 0) appendPQExpBufferStr(query, ", binary = true"); - if (strcmp(subinfo->substream, "f") != 0) + if (strcmp(subinfo->substream, "t") == 0) appendPQExpBufferStr(query, ", streaming = on"); + else if (strcmp(subinfo->substream, "a") == 0) + appendPQExpBufferStr(query, ", streaming = apply"); if (strcmp(subinfo->subtwophasestate, two_phase_disabled) != 0) appendPQExpBufferStr(query, ", two_phase = on"); diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h index d1260f590c..9b394a45fe 100644 --- a/src/include/catalog/pg_subscription.h +++ b/src/include/catalog/pg_subscription.h @@ -68,7 +68,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW bool subbinary; /* True if the subscription wants the * publisher to send data in binary */ - bool substream; /* Stream in-progress transactions. */ + char substream; /* Stream in-progress transactions. + * See SUBSTREAM_xxx constants. */ char subtwophasestate; /* Stream two-phase transactions */ @@ -109,7 +110,7 @@ typedef struct Subscription bool enabled; /* Indicates if the subscription is enabled */ bool binary; /* Indicates if the subscription wants data in * binary format */ - bool stream; /* Allow streaming in-progress transactions. */ + char stream; /* Allow streaming in-progress transactions. */ char twophasestate; /* Allow streaming two-phase transactions */ bool disableonerr; /* Indicates if the subscription should be * automatically disabled if a worker error @@ -120,6 +121,18 @@ typedef struct Subscription List *publications; /* List of publication names to subscribe to */ } Subscription; +/* Disallow streaming in-progress transactions */ +#define SUBSTREAM_OFF 'f' + +/* + * Streaming transactions are written to a temporary file and applied only + * after the transaction is committed on upstream. + */ +#define SUBSTREAM_ON 't' + +/* Streaming transactions are applied immediately via a background worker */ +#define SUBSTREAM_APPLY 'a' + extern Subscription *GetSubscription(Oid subid, bool missing_ok); extern void FreeSubscription(Subscription *sub); extern void DisableSubscription(Oid subid); diff --git a/src/include/replication/logicallauncher.h b/src/include/replication/logicallauncher.h index f1e2821e25..ac8ef94381 100644 --- a/src/include/replication/logicallauncher.h +++ b/src/include/replication/logicallauncher.h @@ -14,6 +14,7 @@ extern PGDLLIMPORT int max_logical_replication_workers; extern PGDLLIMPORT int max_sync_workers_per_subscription; +extern PGDLLIMPORT int max_apply_bgworkers_per_subscription; extern void ApplyLauncherRegister(void); extern void ApplyLauncherMain(Datum main_arg); diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h index a771ab8ff3..7bba77c9e7 100644 --- a/src/include/replication/logicalproto.h +++ b/src/include/replication/logicalproto.h @@ -175,6 +175,17 @@ typedef struct LogicalRepRollbackPreparedTxnData char gid[GIDSIZE]; } LogicalRepRollbackPreparedTxnData; +/* + * Transaction protocol information for stream abort. + */ +typedef struct LogicalRepStreamAbortData +{ + TransactionId xid; + TransactionId subxid; + XLogRecPtr abort_lsn; + TimestampTz abort_time; +} LogicalRepStreamAbortData; + extern void logicalrep_write_begin(StringInfo out, ReorderBufferTXN *txn); extern void logicalrep_read_begin(StringInfo in, LogicalRepBeginData *begin_data); @@ -246,9 +257,11 @@ extern void logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn extern TransactionId logicalrep_read_stream_commit(StringInfo out, LogicalRepCommitData *commit_data); extern void logicalrep_write_stream_abort(StringInfo out, TransactionId xid, - TransactionId subxid); -extern void logicalrep_read_stream_abort(StringInfo in, TransactionId *xid, - TransactionId *subxid); + ReorderBufferTXN *txn, + XLogRecPtr abort_lsn); +extern void logicalrep_read_stream_abort(StringInfo in, + LogicalRepStreamAbortData *abort_data, + bool include_abort_lsn); extern char *logicalrep_message_type(LogicalRepMsgType action); #endif /* LOGICAL_PROTO_H */ diff --git a/src/include/replication/logicalworker.h b/src/include/replication/logicalworker.h index cd1b6e8afc..6a1af7f13c 100644 --- a/src/include/replication/logicalworker.h +++ b/src/include/replication/logicalworker.h @@ -13,6 +13,7 @@ #define LOGICALWORKER_H extern void ApplyWorkerMain(Datum main_arg); +extern void ApplyBgworkerMain(Datum main_arg); extern bool IsLogicalWorker(void); diff --git a/src/include/replication/origin.h b/src/include/replication/origin.h index 2d1b5e5c28..c7389b40a7 100644 --- a/src/include/replication/origin.h +++ b/src/include/replication/origin.h @@ -53,7 +53,7 @@ extern XLogRecPtr replorigin_get_progress(RepOriginId node, bool flush); extern void replorigin_session_advance(XLogRecPtr remote_commit, XLogRecPtr local_commit); -extern void replorigin_session_setup(RepOriginId node); +extern void replorigin_session_setup(RepOriginId node, bool must_acquire); extern void replorigin_session_reset(void); extern XLogRecPtr replorigin_session_get_progress(bool flush); diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h index 4a01f877e5..aba1640f4d 100644 --- a/src/include/replication/reorderbuffer.h +++ b/src/include/replication/reorderbuffer.h @@ -301,6 +301,7 @@ typedef struct ReorderBufferTXN { TimestampTz commit_time; TimestampTz prepare_time; + TimestampTz abort_time; } xact_time; /* @@ -647,9 +648,11 @@ extern void ReorderBufferFinishPrepared(ReorderBuffer *rb, TransactionId xid, extern void ReorderBufferAssignChild(ReorderBuffer *, TransactionId, TransactionId, XLogRecPtr commit_lsn); extern void ReorderBufferCommitChild(ReorderBuffer *, TransactionId, TransactionId, XLogRecPtr commit_lsn, XLogRecPtr end_lsn); -extern void ReorderBufferAbort(ReorderBuffer *, TransactionId, XLogRecPtr lsn); +extern void ReorderBufferAbort(ReorderBuffer *, TransactionId, XLogRecPtr lsn, + TimestampTz abort_time); extern void ReorderBufferAbortOld(ReorderBuffer *, TransactionId xid); -extern void ReorderBufferForget(ReorderBuffer *, TransactionId, XLogRecPtr lsn); +extern void ReorderBufferForget(ReorderBuffer *, TransactionId, XLogRecPtr lsn, + TimestampTz abort_time); extern void ReorderBufferInvalidate(ReorderBuffer *, TransactionId, XLogRecPtr lsn); extern void ReorderBufferSetBaseSnapshot(ReorderBuffer *, TransactionId, XLogRecPtr lsn, struct SnapshotData *snap); diff --git a/src/include/replication/worker_internal.h b/src/include/replication/worker_internal.h index 901845abc2..08e63de013 100644 --- a/src/include/replication/worker_internal.h +++ b/src/include/replication/worker_internal.h @@ -17,8 +17,11 @@ #include "access/xlogdefs.h" #include "catalog/pg_subscription.h" #include "datatype/timestamp.h" +#include "replication/logicalrelation.h" #include "storage/fileset.h" #include "storage/lock.h" +#include "storage/shm_mq.h" +#include "storage/shm_toc.h" #include "storage/spin.h" @@ -60,6 +63,9 @@ typedef struct LogicalRepWorker */ FileSet *stream_fileset; + /* Indicates if this slot is used for an apply background worker. */ + bool subworker; + /* Stats. */ XLogRecPtr last_lsn; TimestampTz last_send_time; @@ -68,9 +74,70 @@ typedef struct LogicalRepWorker TimestampTz reply_time; } LogicalRepWorker; +/* Struct for saving and restoring apply errcontext information */ +typedef struct ApplyErrorCallbackArg +{ + LogicalRepMsgType command; /* 0 if invalid */ + LogicalRepRelMapEntry *rel; + + /* Remote node information */ + int remote_attnum; /* -1 if invalid */ + TransactionId remote_xid; + XLogRecPtr finish_lsn; + char *origin_name; +} ApplyErrorCallbackArg; + +/* + * Status for apply background worker. + */ +typedef enum ApplyBgworkerStatus +{ + APPLY_BGWORKER_ATTACHED = 0, + APPLY_BGWORKER_READY, + APPLY_BGWORKER_BUSY, + APPLY_BGWORKER_FINISHED, + APPLY_BGWORKER_EXIT +} ApplyBgworkerStatus; + +/* + * Shared information among apply workers. + */ +typedef struct ApplyBgworkerShared +{ + slock_t mutex; + + /* Status for apply background worker. */ + ApplyBgworkerStatus status; + + /* server version of publisher. */ + int server_version; + + TransactionId stream_xid; + uint32 n; /* id of apply background worker */ +} ApplyBgworkerShared; + +/* + * Struct for maintaining an apply background worker. + */ +typedef struct ApplyBgworkerState +{ + shm_mq_handle *mq_handle; + dsm_segment *dsm_seg; + ApplyBgworkerShared volatile *pstate; +} ApplyBgworkerState; + /* Main memory context for apply worker. Permanent during worker lifetime. */ extern PGDLLIMPORT MemoryContext ApplyContext; +extern PGDLLIMPORT MemoryContext ApplyMessageContext; + +extern PGDLLIMPORT ApplyErrorCallbackArg apply_error_callback_arg; + +extern PGDLLIMPORT bool MySubscriptionValid; + +extern PGDLLIMPORT volatile ApplyBgworkerShared *MyParallelState; +extern PGDLLIMPORT List *subxactlist; + /* libpqreceiver connection */ extern PGDLLIMPORT struct WalReceiverConn *LogRepWorkerWalRcvConn; @@ -79,18 +146,22 @@ extern PGDLLIMPORT Subscription *MySubscription; extern PGDLLIMPORT LogicalRepWorker *MyLogicalRepWorker; extern PGDLLIMPORT bool in_remote_transaction; +extern PGDLLIMPORT bool in_streamed_transaction; +extern PGDLLIMPORT TransactionId stream_xid; extern void logicalrep_worker_attach(int slot); extern LogicalRepWorker *logicalrep_worker_find(Oid subid, Oid relid, bool only_running); extern List *logicalrep_workers_find(Oid subid, bool only_running); -extern void logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, - Oid userid, Oid relid); +extern bool logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, + Oid userid, Oid relid, + dsm_handle subworker_dsm); extern void logicalrep_worker_stop(Oid subid, Oid relid); extern void logicalrep_worker_wakeup(Oid subid, Oid relid); extern void logicalrep_worker_wakeup_ptr(LogicalRepWorker *worker); extern int logicalrep_sync_worker_count(Oid subid); +extern int logicalrep_apply_background_worker_count(Oid subid); extern void ReplicationOriginNameForTablesync(Oid suboid, Oid relid, char *originname, int szorgname); @@ -103,10 +174,39 @@ extern void process_syncing_tables(XLogRecPtr current_lsn); extern void invalidate_syncing_table_states(Datum arg, int cacheid, uint32 hashvalue); +extern void UpdateWorkerStats(XLogRecPtr last_lsn, TimestampTz send_time, + bool reply); + +/* prototype needed because of stream_commit */ +extern void apply_dispatch(StringInfo s); + +/* Function for apply error callback */ +extern void apply_error_callback(void *arg); + +extern void subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue); + +/* apply background worker setup and interactions */ +extern ApplyBgworkerState *apply_bgworker_find_or_start(TransactionId xid, + bool start); +extern void apply_bgworker_wait_for(ApplyBgworkerState *wstate, + ApplyBgworkerStatus wait_for_status); +extern void apply_bgworker_send_data(ApplyBgworkerState *wstate, Size nbytes, + const void *data); +extern void apply_bgworker_free(ApplyBgworkerState *wstate); +extern void apply_bgworker_check_status(void); +extern void apply_bgworker_set_status(ApplyBgworkerStatus status); +extern void apply_bgworker_subxact_info_add(TransactionId current_xid); + static inline bool am_tablesync_worker(void) { return OidIsValid(MyLogicalRepWorker->relid); } +static inline bool +am_apply_bgworker(void) +{ + return MyLogicalRepWorker->subworker; +} + #endif /* WORKER_INTERNAL_H */ diff --git a/src/include/utils/wait_event.h b/src/include/utils/wait_event.h index b578e2ec75..c2d2a114d7 100644 --- a/src/include/utils/wait_event.h +++ b/src/include/utils/wait_event.h @@ -105,6 +105,7 @@ typedef enum WAIT_EVENT_HASH_GROW_BUCKETS_ALLOCATE, WAIT_EVENT_HASH_GROW_BUCKETS_ELECT, WAIT_EVENT_HASH_GROW_BUCKETS_REINSERT, + WAIT_EVENT_LOGICAL_APPLY_WORKER_STATE_CHANGE, WAIT_EVENT_LOGICAL_SYNC_DATA, WAIT_EVENT_LOGICAL_SYNC_STATE_CHANGE, WAIT_EVENT_MQ_INTERNAL, diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out index 7fcfad1591..f769835af0 100644 --- a/src/test/regress/expected/subscription.out +++ b/src/test/regress/expected/subscription.out @@ -197,7 +197,7 @@ ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); DROP SUBSCRIPTION regress_testsub; -- fail - streaming must be boolean CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = foo); -ERROR: streaming requires a Boolean value +ERROR: streaming requires a Boolean value or "apply" -- now it works CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = true); WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 4fb746930a..664d4b8b1a 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -116,6 +116,10 @@ Append AppendPath AppendRelInfo AppendState +ApplyBgworkerEntry +ApplyBgworkerShared +ApplyBgworkerState +ApplyBgworkerStatus ApplyErrorCallbackArg ApplyExecutionData ApplySubXactData @@ -1483,6 +1487,7 @@ LogicalRepRelId LogicalRepRelMapEntry LogicalRepRelation LogicalRepRollbackPreparedTxnData +LogicalRepStreamAbortData LogicalRepTupleData LogicalRepTyp LogicalRepWorker -- 2.23.0.windows.1 [application/octet-stream] v11-0002-Test-streaming-apply-option-in-tap-test.patch (68.9K, ../../OS3PR01MB6275C4679CFE3414340A259B9EAD9@OS3PR01MB6275.jpnprd01.prod.outlook.com/3-v11-0002-Test-streaming-apply-option-in-tap-test.patch) download | inline diff: From f30f619f8dd2b3d443d630d5b988808e5ac51193 Mon Sep 17 00:00:00 2001 From: "shiy.fnst" <[email protected]> Date: Fri, 13 May 2022 14:50:30 +0800 Subject: [PATCH v11 2/5] Test streaming apply option in tap test Change all TAP tests using the SUBSCRIPTION "streaming" option, so they now test both 'on' and 'apply' values. --- src/test/subscription/t/015_stream.pl | 199 ++++--- src/test/subscription/t/016_stream_subxact.pl | 119 +++-- src/test/subscription/t/017_stream_ddl.pl | 188 ++++--- .../t/018_stream_subxact_abort.pl | 195 ++++--- .../t/019_stream_subxact_ddl_abort.pl | 110 +++- .../subscription/t/022_twophase_cascade.pl | 363 +++++++------ .../subscription/t/023_twophase_stream.pl | 498 ++++++++++-------- 7 files changed, 1035 insertions(+), 637 deletions(-) diff --git a/src/test/subscription/t/015_stream.pl b/src/test/subscription/t/015_stream.pl index 6561b189de..b3d84b1c7c 100644 --- a/src/test/subscription/t/015_stream.pl +++ b/src/test/subscription/t/015_stream.pl @@ -8,6 +8,116 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" option +# so the same code can be run both for the streaming=on and streaming=apply +# cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_apply) = @_; + + # Interleave a pair of transactions, each exceeding the 64kB limit. + my $in = ''; + my $out = ''; + + my $timer = IPC::Run::timeout($PostgreSQL::Test::Utils::timeout_default); + + my $h = $node_publisher->background_psql('postgres', \$in, \$out, $timer, + on_error_stop => 0); + + # Check that a background worker starts if "streaming" option is + # specified as "apply". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_apply) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + $in .= q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + }; + $h->pump_nb; + + if ($is_apply) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001, 9999) s(i); + DELETE FROM test_tab WHERE a > 5000; + COMMIT; + }); + + $in .= q{ + COMMIT; + \q + }; + $h->finish; # errors make the next test fail, so ignore them here + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), + 'check extra columns contain local defaults'); + + # Test the streaming in binary mode + $node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET (binary = on)"); + + # Insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001, 10000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + COMMIT; + }); + + $node_publisher->wait_for_catchup($appname); + + $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(6667|6667|6667), + 'check extra columns contain local defaults'); + + # Change the local values of the extra columns on the subscriber, + # update publisher, and check that subscriber retains the expected + # values. This is to ensure that non-streaming transactions behave + # properly after a streaming transaction. + $node_subscriber->safe_psql('postgres', + "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'" + ); + $node_publisher->safe_psql('postgres', + "UPDATE test_tab SET b = md5(a::text)"); + + $node_publisher->wait_for_catchup($appname); + + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab" + ); + is($result, qq(6667|6667|6667), + 'check extra columns contain locally changed data'); + + # Cleanup the test data + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE (a > 2)"); + $node_publisher->wait_for_catchup($appname); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -37,6 +147,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -54,82 +168,25 @@ my $result = "SELECT count(*), count(c), count(d = 999) FROM test_tab"); is($result, qq(2|2|2), 'check initial data was copied to subscriber'); -# Interleave a pair of transactions, each exceeding the 64kB limit. -my $in = ''; -my $out = ''; - -my $timer = IPC::Run::timeout($PostgreSQL::Test::Utils::timeout_default); - -my $h = $node_publisher->background_psql('postgres', \$in, \$out, $timer, - on_error_stop => 0); - -$in .= q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -}; -$h->pump_nb; - -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001, 9999) s(i); -DELETE FROM test_tab WHERE a > 5000; -COMMIT; -}); - -$in .= q{ -COMMIT; -\q -}; -$h->finish; # errors make the next test fail, so ignore them here - -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), 'check extra columns contain local defaults'); - -# Test the streaming in binary mode -$node_subscriber->safe_psql('postgres', - "ALTER SUBSCRIPTION tap_sub SET (binary = on)"); - -# Insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001, 10000) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -COMMIT; -}); - -$node_publisher->wait_for_catchup($appname); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(6667|6667|6667), 'check extra columns contain local defaults'); +################################### +# Test using streaming mode 'apply' +################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" +); -# Change the local values of the extra columns on the subscriber, -# update publisher, and check that subscriber retains the expected -# values. This is to ensure that non-streaming transactions behave -# properly after a streaming transaction. $node_subscriber->safe_psql('postgres', - "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'" -); -$node_publisher->safe_psql('postgres', - "UPDATE test_tab SET b = md5(a::text)"); + "ALTER SUBSCRIPTION tap_sub SET(streaming = apply, binary = off)"); -$node_publisher->wait_for_catchup($appname); +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab" -); -is($result, qq(6667|6667|6667), - 'check extra columns contain locally changed data'); +test_streaming($node_publisher, $node_subscriber, $appname, 1); $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/016_stream_subxact.pl b/src/test/subscription/t/016_stream_subxact.pl index f27f1694f2..b5b6dc379f 100644 --- a/src/test/subscription/t/016_stream_subxact.pl +++ b/src/test/subscription/t/016_stream_subxact.pl @@ -8,6 +8,72 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" option +# so the same code can be run both for the streaming=on and streaming=apply +# cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_apply) = @_; + + # Check that a background worker starts if "streaming" option is + # specified as "apply". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_apply) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + # Insert, update and delete enough rows to exceed 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series( 3, 500) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501, 1000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001, 1500) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + SAVEPOINT s3; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501, 2000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + SAVEPOINT s4; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001, 2500) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + COMMIT; + }); + + if ($is_apply) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(1667|1667|1667), + 'check data was copied to subscriber in streaming mode and extra columns contain local defaults' + ); + + # Cleanup the test data + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE (a > 2)"); + $node_publisher->wait_for_catchup($appname); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -37,6 +103,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -54,41 +124,26 @@ my $result = "SELECT count(*), count(c), count(d = 999) FROM test_tab"); is($result, qq(2|2|2), 'check initial data was copied to subscriber'); -# Insert, update and delete enough rows to exceed 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series( 3, 500) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501, 1000) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001, 1500) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -SAVEPOINT s3; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501, 2000) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -SAVEPOINT s4; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001, 2500) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -COMMIT; -}); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(1667|1667|1667), - 'check data was copied to subscriber in streaming mode and extra columns contain local defaults' +################################### +# Test using streaming mode 'apply' +################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" ); +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = apply)"); + +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; + +test_streaming($node_publisher, $node_subscriber, $appname, 1); + $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/017_stream_ddl.pl b/src/test/subscription/t/017_stream_ddl.pl index 0bce63b716..7e5dc18cd2 100644 --- a/src/test/subscription/t/017_stream_ddl.pl +++ b/src/test/subscription/t/017_stream_ddl.pl @@ -8,6 +8,111 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" option +# so the same code can be run both for the streaming=on and streaming=apply +# cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_apply) = @_; + + # a small (non-streamed) transaction with DDL and DML + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab VALUES (3, md5(3::text)); + ALTER TABLE test_tab ADD COLUMN c INT; + SAVEPOINT s1; + INSERT INTO test_tab VALUES (4, md5(4::text), -4); + COMMIT; + }); + + # Check that a background worker starts if "streaming" option is + # specified as "apply". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_apply) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + # large (streamed) transaction with DDL and DML + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(5, 1000) s(i); + ALTER TABLE test_tab ADD COLUMN d INT; + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001, 2000) s(i); + COMMIT; + }); + + if ($is_apply) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + # a small (non-streamed) transaction with DDL and DML + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab VALUES (2001, md5(2001::text), -2001, 2*2001); + ALTER TABLE test_tab ADD COLUMN e INT; + SAVEPOINT s1; + INSERT INTO test_tab VALUES (2002, md5(2002::text), -2002, 2*2002, -3*2002); + COMMIT; + }); + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d), count(e) FROM test_tab"); + is($result, qq(2002|1999|1002|1), + 'check data was copied to subscriber in streaming mode and extra columns contain local defaults' + ); + + # A large (streamed) transaction with DDL and DML. One of the DDL is performed + # after DML to ensure that we invalidate the schema sent for test_tab so that + # the next transaction has to send the schema again. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(2003,5000) s(i); + ALTER TABLE test_tab ADD COLUMN f INT; + COMMIT; + }); + + # A small transaction that won't get streamed. This is just to ensure that we + # send the schema again to reflect the last column added in the previous test. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i, 4*i FROM generate_series(5001,5005) s(i); + COMMIT; + }); + + $node_publisher->wait_for_catchup($appname); + + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d), count(e), count(f) FROM test_tab" + ); + is($result, qq(5005|5002|4005|3004|5), + 'check data was copied to subscriber for both streaming and non-streaming transactions' + ); + + # Cleanup the test data + $node_publisher->safe_psql( + 'postgres', q{ + DELETE FROM test_tab WHERE (a > 2); + ALTER TABLE test_tab DROP COLUMN c, DROP COLUMN d, DROP COLUMN e, DROP COLUMN f; + }); + $node_publisher->wait_for_catchup($appname); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -37,6 +142,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -54,76 +163,25 @@ my $result = "SELECT count(*), count(c), count(d = 999) FROM test_tab"); is($result, qq(2|0|0), 'check initial data was copied to subscriber'); -# a small (non-streamed) transaction with DDL and DML -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab VALUES (3, md5(3::text)); -ALTER TABLE test_tab ADD COLUMN c INT; -SAVEPOINT s1; -INSERT INTO test_tab VALUES (4, md5(4::text), -4); -COMMIT; -}); - -# large (streamed) transaction with DDL and DML -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(5, 1000) s(i); -ALTER TABLE test_tab ADD COLUMN d INT; -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001, 2000) s(i); -COMMIT; -}); - -# a small (non-streamed) transaction with DDL and DML -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab VALUES (2001, md5(2001::text), -2001, 2*2001); -ALTER TABLE test_tab ADD COLUMN e INT; -SAVEPOINT s1; -INSERT INTO test_tab VALUES (2002, md5(2002::text), -2002, 2*2002, -3*2002); -COMMIT; -}); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d), count(e) FROM test_tab"); -is($result, qq(2002|1999|1002|1), - 'check data was copied to subscriber in streaming mode and extra columns contain local defaults' +################################### +# Test using streaming mode 'apply' +################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" ); -# A large (streamed) transaction with DDL and DML. One of the DDL is performed -# after DML to ensure that we invalidate the schema sent for test_tab so that -# the next transaction has to send the schema again. -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(2003,5000) s(i); -ALTER TABLE test_tab ADD COLUMN f INT; -COMMIT; -}); - -# A small transaction that won't get streamed. This is just to ensure that we -# send the schema again to reflect the last column added in the previous test. -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i, 4*i FROM generate_series(5001,5005) s(i); -COMMIT; -}); +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = apply)"); -$node_publisher->wait_for_catchup($appname); +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d), count(e), count(f) FROM test_tab"); -is($result, qq(5005|5002|4005|3004|5), - 'check data was copied to subscriber for both streaming and non-streaming transactions' -); +test_streaming($node_publisher, $node_subscriber, $appname, 1); $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/018_stream_subxact_abort.pl b/src/test/subscription/t/018_stream_subxact_abort.pl index 7155442e76..71f8c1dd05 100644 --- a/src/test/subscription/t/018_stream_subxact_abort.pl +++ b/src/test/subscription/t/018_stream_subxact_abort.pl @@ -8,6 +8,113 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" option +# so the same code can be run both for the streaming=on and streaming=apply +# cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_apply) = @_; + + # Check that a background worker starts if "streaming" option is + # specified as "apply". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_apply) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + # large (streamed) transaction with DDL, DML and ROLLBACKs + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i); + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501,1000) s(i); + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001,1500) s(i); + SAVEPOINT s3; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501,2000) s(i); + ROLLBACK TO s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001,2500) s(i); + ROLLBACK TO s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2501,3000) s(i); + SAVEPOINT s4; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3001,3500) s(i); + SAVEPOINT s5; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3501,4000) s(i); + COMMIT; + }); + + if ($is_apply) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c) FROM test_tab"); + is($result, qq(2000|0), + 'check rollback to savepoint was reflected on subscriber and extra columns contain local defaults' + ); + + # large (streamed) transaction with subscriber receiving out of order + # subtransaction ROLLBACKs + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(4001,4500) s(i); + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001,5500) s(i); + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(6001,6500) s(i); + SAVEPOINT s3; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(7001,7500) s(i); + RELEASE s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(8001,8500) s(i); + ROLLBACK TO s1; + COMMIT; + }); + + $node_publisher->wait_for_catchup($appname); + + $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c) FROM test_tab"); + is($result, qq(2500|0), + 'check rollback to savepoint was reflected on subscriber'); + + # large (streamed) transaction with subscriber receiving rollback + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(8501,9000) s(i); + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(9001,9500) s(i); + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(9501,10000) s(i); + ROLLBACK; + }); + + $node_publisher->wait_for_catchup($appname); + + $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c) FROM test_tab"); + is($result, qq(2500|0), 'check rollback was reflected on subscriber'); + + # Cleanup the test data + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE (a > 2)"); + $node_publisher->wait_for_catchup($appname); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -36,6 +143,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -53,81 +164,25 @@ my $result = "SELECT count(*), count(c) FROM test_tab"); is($result, qq(2|0), 'check initial data was copied to subscriber'); -# large (streamed) transaction with DDL, DML and ROLLBACKs -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i); -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501,1000) s(i); -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001,1500) s(i); -SAVEPOINT s3; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501,2000) s(i); -ROLLBACK TO s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001,2500) s(i); -ROLLBACK TO s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2501,3000) s(i); -SAVEPOINT s4; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3001,3500) s(i); -SAVEPOINT s5; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3501,4000) s(i); -COMMIT; -}); - -$node_publisher->wait_for_catchup($appname); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c) FROM test_tab"); -is($result, qq(2000|0), - 'check rollback to savepoint was reflected on subscriber and extra columns contain local defaults' +################################### +# Test using streaming mode 'apply' +################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" ); -# large (streamed) transaction with subscriber receiving out of order -# subtransaction ROLLBACKs -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(4001,4500) s(i); -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001,5500) s(i); -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(6001,6500) s(i); -SAVEPOINT s3; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(7001,7500) s(i); -RELEASE s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(8001,8500) s(i); -ROLLBACK TO s1; -COMMIT; -}); - -$node_publisher->wait_for_catchup($appname); +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = apply)"); -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c) FROM test_tab"); -is($result, qq(2500|0), - 'check rollback to savepoint was reflected on subscriber'); - -# large (streamed) transaction with subscriber receiving rollback -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(8501,9000) s(i); -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(9001,9500) s(i); -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(9501,10000) s(i); -ROLLBACK; -}); +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c) FROM test_tab"); -is($result, qq(2500|0), 'check rollback was reflected on subscriber'); +test_streaming($node_publisher, $node_subscriber, $appname, 1); $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/019_stream_subxact_ddl_abort.pl b/src/test/subscription/t/019_stream_subxact_ddl_abort.pl index dbd0fca4d1..d0cd869430 100644 --- a/src/test/subscription/t/019_stream_subxact_ddl_abort.pl +++ b/src/test/subscription/t/019_stream_subxact_ddl_abort.pl @@ -9,6 +9,69 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" option +# so the same code can be run both for the streaming=on and streaming=apply +# cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_apply) = @_; + + # Check that a background worker starts if "streaming" option is + # specified as "apply". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_apply) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + # large (streamed) transaction with DDL, DML and ROLLBACKs + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i); + ALTER TABLE test_tab ADD COLUMN c INT; + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(501,1000) s(i); + ALTER TABLE test_tab ADD COLUMN d INT; + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001,1500) s(i); + ALTER TABLE test_tab ADD COLUMN e INT; + SAVEPOINT s3; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(1501,2000) s(i); + ALTER TABLE test_tab DROP COLUMN c; + ROLLBACK TO s1; + INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(501,1000) s(i); + COMMIT; + }); + + if ($is_apply) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c) FROM test_tab"); + is($result, qq(1000|500), + 'check rollback to savepoint was reflected on subscriber and extra columns contain local defaults' + ); + + # Cleanup the test data + $node_publisher->safe_psql( + 'postgres', q{ + DELETE FROM test_tab WHERE (a > 2); + ALTER TABLE test_tab DROP COLUMN c; + }); + $node_publisher->wait_for_catchup($appname); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -37,6 +100,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -54,35 +121,26 @@ my $result = "SELECT count(*), count(c) FROM test_tab"); is($result, qq(2|0), 'check initial data was copied to subscriber'); -# large (streamed) transaction with DDL, DML and ROLLBACKs -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i); -ALTER TABLE test_tab ADD COLUMN c INT; -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(501,1000) s(i); -ALTER TABLE test_tab ADD COLUMN d INT; -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001,1500) s(i); -ALTER TABLE test_tab ADD COLUMN e INT; -SAVEPOINT s3; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(1501,2000) s(i); -ALTER TABLE test_tab DROP COLUMN c; -ROLLBACK TO s1; -INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(501,1000) s(i); -COMMIT; -}); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c) FROM test_tab"); -is($result, qq(1000|500), - 'check rollback to savepoint was reflected on subscriber and extra columns contain local defaults' +################################### +# Test using streaming mode 'apply' +################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" ); +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = apply)"); + +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; + +test_streaming($node_publisher, $node_subscriber, $appname, 1); + $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/022_twophase_cascade.pl b/src/test/subscription/t/022_twophase_cascade.pl index 7a797f37ba..b52af74e35 100644 --- a/src/test/subscription/t/022_twophase_cascade.pl +++ b/src/test/subscription/t/022_twophase_cascade.pl @@ -11,6 +11,208 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" option +# so the same code can be run both for the streaming=on and streaming=apply +# cases. +sub test_streaming +{ + my ($node_A, $node_B, $node_C, $appname_B, $appname_C, $streaming_mode) = + @_; + + my $oldpid_B = $node_A->safe_psql( + 'postgres', " + SELECT pid FROM pg_stat_replication + WHERE application_name = '$appname_B' AND state = 'streaming';"); + my $oldpid_C = $node_B->safe_psql( + 'postgres', " + SELECT pid FROM pg_stat_replication + WHERE application_name = '$appname_C' AND state = 'streaming';"); + + # Setup logical replication streaming mode + + $node_B->safe_psql( + 'postgres', " + ALTER SUBSCRIPTION tap_sub_B + SET (streaming = $streaming_mode);"); + $node_C->safe_psql( + 'postgres', " + ALTER SUBSCRIPTION tap_sub_C + SET (streaming = $streaming_mode)"); + + # Wait for subscribers to finish initialization + + $node_A->poll_query_until( + 'postgres', " + SELECT pid != $oldpid_B FROM pg_stat_replication + WHERE application_name = '$appname_B' AND state = 'streaming';" + ) or die "Timed out while waiting for apply to restart"; + $node_B->poll_query_until( + 'postgres', " + SELECT pid != $oldpid_C FROM pg_stat_replication + WHERE application_name = '$appname_C' AND state = 'streaming';" + ) or die "Timed out while waiting for apply to restart"; + + ############################### + # Test 2PC PREPARE / COMMIT PREPARED. + # 1. Data is streamed as a 2PC transaction. + # 2. Then do commit prepared. + # + # Expect all data is replicated on subscriber(s) after the commit. + ############################### + + # Check that a background worker starts if "streaming" option is + # specified as "apply". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($streaming_mode eq 'apply') + { + $node_B->append_conf('postgresql.conf', "log_min_messages = debug1"); + $node_B->reload; + + $node_C->append_conf('postgresql.conf', "log_min_messages = debug1"); + $node_C->reload; + } + + # Insert, update and delete enough rows to exceed the 64kB limit. + # Then 2PC PREPARE + $node_A->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + if ($streaming_mode eq 'apply') + { + $node_B->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_B->append_conf('postgresql.conf', "log_min_messages = warning"); + $node_B->reload; + + $node_C->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_C->append_conf('postgresql.conf', "log_min_messages = warning"); + $node_C->reload; + } + + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); + + # check the transaction state is prepared on subscriber(s) + my $result = + $node_B->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber B'); + $result = + $node_C->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber C'); + + # 2PC COMMIT + $node_A->safe_psql('postgres', "COMMIT PREPARED 'test_prepared_tab';"); + + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); + + # check that transaction was committed on subscriber(s) + $result = $node_B->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), + 'Rows inserted by 2PC have committed on subscriber B, and extra columns have local defaults' + ); + $result = $node_C->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), + 'Rows inserted by 2PC have committed on subscriber C, and extra columns have local defaults' + ); + + # check the transaction state is ended on subscriber(s) + $result = + $node_B->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is committed on subscriber B'); + $result = + $node_C->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is committed on subscriber C'); + + ############################### + # Test 2PC PREPARE with a nested ROLLBACK TO SAVEPOINT. + # 0. Cleanup from previous test leaving only 2 rows. + # 1. Insert one more row. + # 2. Record a SAVEPOINT. + # 3. Data is streamed using 2PC. + # 4. Do rollback to SAVEPOINT prior to the streamed inserts. + # 5. Then COMMIT PREPARED. + # + # Expect data after the SAVEPOINT is aborted leaving only 3 rows (= 2 original + 1 from step 1). + ############################### + + # First, delete the data except for 2 rows (delete will be replicated) + $node_A->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); + + # 2PC PREPARE with a nested ROLLBACK TO SAVEPOINT + $node_A->safe_psql( + 'postgres', " + BEGIN; + INSERT INTO test_tab VALUES (9999, 'foobar'); + SAVEPOINT sp_inner; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + ROLLBACK TO SAVEPOINT sp_inner; + PREPARE TRANSACTION 'outer'; + "); + + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); + + # check the transaction state prepared on subscriber(s) + $result = + $node_B->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber B'); + $result = + $node_C->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber C'); + + # 2PC COMMIT + $node_A->safe_psql('postgres', "COMMIT PREPARED 'outer';"); + + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); + + # check the transaction state is ended on subscriber + $result = + $node_B->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is ended on subscriber B'); + $result = + $node_C->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is ended on subscriber C'); + + # check inserts are visible at subscriber(s). + # All the streamed data (prior to the SAVEPOINT) should be rolled back. + # (9999, 'foobar') should be committed. + $result = $node_B->safe_psql('postgres', + "SELECT count(*) FROM test_tab where b = 'foobar';"); + is($result, qq(1), 'Rows committed are present on subscriber B'); + $result = + $node_B->safe_psql('postgres', "SELECT count(*) FROM test_tab;"); + is($result, qq(3), 'Rows committed are present on subscriber B'); + $result = $node_C->safe_psql('postgres', + "SELECT count(*) FROM test_tab where b = 'foobar';"); + is($result, qq(1), 'Rows committed are present on subscriber C'); + $result = + $node_C->safe_psql('postgres', "SELECT count(*) FROM test_tab;"); + is($result, qq(3), 'Rows committed are present on subscriber C'); + + # Cleanup the test data + $node_A->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); +} + ############################### # Setup a cascade of pub/sub nodes. # node_A -> node_B -> node_C @@ -260,160 +462,15 @@ is($result, qq(21), 'Rows committed are present on subscriber C'); # 2PC + STREAMING TESTS # --------------------- -my $oldpid_B = $node_A->safe_psql( - 'postgres', " - SELECT pid FROM pg_stat_replication - WHERE application_name = '$appname_B' AND state = 'streaming';"); -my $oldpid_C = $node_B->safe_psql( - 'postgres', " - SELECT pid FROM pg_stat_replication - WHERE application_name = '$appname_C' AND state = 'streaming';"); - -# Setup logical replication (streaming = on) - -$node_B->safe_psql( - 'postgres', " - ALTER SUBSCRIPTION tap_sub_B - SET (streaming = on);"); -$node_C->safe_psql( - 'postgres', " - ALTER SUBSCRIPTION tap_sub_C - SET (streaming = on)"); - -# Wait for subscribers to finish initialization - -$node_A->poll_query_until( - 'postgres', " - SELECT pid != $oldpid_B FROM pg_stat_replication - WHERE application_name = '$appname_B' AND state = 'streaming';" -) or die "Timed out while waiting for apply to restart"; -$node_B->poll_query_until( - 'postgres', " - SELECT pid != $oldpid_C FROM pg_stat_replication - WHERE application_name = '$appname_C' AND state = 'streaming';" -) or die "Timed out while waiting for apply to restart"; - -############################### -# Test 2PC PREPARE / COMMIT PREPARED. -# 1. Data is streamed as a 2PC transaction. -# 2. Then do commit prepared. -# -# Expect all data is replicated on subscriber(s) after the commit. -############################### - -# Insert, update and delete enough rows to exceed the 64kB limit. -# Then 2PC PREPARE -$node_A->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_A->wait_for_catchup($appname_B); -$node_B->wait_for_catchup($appname_C); +################################ +# Test using streaming mode 'on' +################################ +test_streaming($node_A, $node_B, $node_C, $appname_B, $appname_C, 'on'); -# check the transaction state is prepared on subscriber(s) -$result = - $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber B'); -$result = - $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber C'); - -# 2PC COMMIT -$node_A->safe_psql('postgres', "COMMIT PREPARED 'test_prepared_tab';"); - -$node_A->wait_for_catchup($appname_B); -$node_B->wait_for_catchup($appname_C); - -# check that transaction was committed on subscriber(s) -$result = $node_B->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), - 'Rows inserted by 2PC have committed on subscriber B, and extra columns have local defaults' -); -$result = $node_C->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), - 'Rows inserted by 2PC have committed on subscriber C, and extra columns have local defaults' -); - -# check the transaction state is ended on subscriber(s) -$result = - $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is committed on subscriber B'); -$result = - $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is committed on subscriber C'); - -############################### -# Test 2PC PREPARE with a nested ROLLBACK TO SAVEPOINT. -# 0. Cleanup from previous test leaving only 2 rows. -# 1. Insert one more row. -# 2. Record a SAVEPOINT. -# 3. Data is streamed using 2PC. -# 4. Do rollback to SAVEPOINT prior to the streamed inserts. -# 5. Then COMMIT PREPARED. -# -# Expect data after the SAVEPOINT is aborted leaving only 3 rows (= 2 original + 1 from step 1). -############################### - -# First, delete the data except for 2 rows (delete will be replicated) -$node_A->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); - -# 2PC PREPARE with a nested ROLLBACK TO SAVEPOINT -$node_A->safe_psql( - 'postgres', " - BEGIN; - INSERT INTO test_tab VALUES (9999, 'foobar'); - SAVEPOINT sp_inner; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - ROLLBACK TO SAVEPOINT sp_inner; - PREPARE TRANSACTION 'outer'; - "); - -$node_A->wait_for_catchup($appname_B); -$node_B->wait_for_catchup($appname_C); - -# check the transaction state prepared on subscriber(s) -$result = - $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber B'); -$result = - $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber C'); - -# 2PC COMMIT -$node_A->safe_psql('postgres', "COMMIT PREPARED 'outer';"); - -$node_A->wait_for_catchup($appname_B); -$node_B->wait_for_catchup($appname_C); - -# check the transaction state is ended on subscriber -$result = - $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is ended on subscriber B'); -$result = - $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is ended on subscriber C'); - -# check inserts are visible at subscriber(s). -# All the streamed data (prior to the SAVEPOINT) should be rolled back. -# (9999, 'foobar') should be committed. -$result = $node_B->safe_psql('postgres', - "SELECT count(*) FROM test_tab where b = 'foobar';"); -is($result, qq(1), 'Rows committed are present on subscriber B'); -$result = $node_B->safe_psql('postgres', "SELECT count(*) FROM test_tab;"); -is($result, qq(3), 'Rows committed are present on subscriber B'); -$result = $node_C->safe_psql('postgres', - "SELECT count(*) FROM test_tab where b = 'foobar';"); -is($result, qq(1), 'Rows committed are present on subscriber C'); -$result = $node_C->safe_psql('postgres', "SELECT count(*) FROM test_tab;"); -is($result, qq(3), 'Rows committed are present on subscriber C'); +################################### +# Test using streaming mode 'apply' +################################### +test_streaming($node_A, $node_B, $node_C, $appname_B, $appname_C, 'apply'); ############################### # check all the cleanup diff --git a/src/test/subscription/t/023_twophase_stream.pl b/src/test/subscription/t/023_twophase_stream.pl index d8475d25a4..f55c5c95e7 100644 --- a/src/test/subscription/t/023_twophase_stream.pl +++ b/src/test/subscription/t/023_twophase_stream.pl @@ -8,6 +8,266 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" option +# so the same code can be run both for the streaming=on and streaming=apply +# cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_apply) = @_; + + ############################### + # Test 2PC PREPARE / COMMIT PREPARED. + # 1. Data is streamed as a 2PC transaction. + # 2. Then do commit prepared. + # + # Expect all data is replicated on subscriber side after the commit. + ############################### + + # Check that a background worker starts if "streaming" option is + # specified as "apply". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_apply) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + # check that 2PC gets replicated to subscriber + # Insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + if ($is_apply) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is in prepared state on subscriber + my $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber'); + + # 2PC transaction gets committed + $node_publisher->safe_psql('postgres', + "COMMIT PREPARED 'test_prepared_tab';"); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is committed on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), + 'Rows inserted by 2PC have committed on subscriber, and extra columns contain local defaults' + ); + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is committed on subscriber'); + + ############################### + # Test 2PC PREPARE / ROLLBACK PREPARED. + # 1. Table is deleted back to 2 rows which are replicated on subscriber. + # 2. Data is streamed using 2PC. + # 3. Do rollback prepared. + # + # Expect data rolls back leaving only the original 2 rows. + ############################### + + # First, delete the data except for 2 rows (will be replicated) + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE a > 2;"); + + # Then insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is in prepared state on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber'); + + # 2PC transaction gets aborted + $node_publisher->safe_psql('postgres', + "ROLLBACK PREPARED 'test_prepared_tab';"); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is aborted on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(2|2|2), + 'Rows inserted by 2PC are rolled back, leaving only the original 2 rows' + ); + + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is aborted on subscriber'); + + ############################### + # Check that 2PC COMMIT PREPARED is decoded properly on crash restart. + # 1. insert, update and delete enough rows to exceed the 64kB limit. + # 2. Then server crashes before the 2PC transaction is committed. + # 3. After servers are restarted the pending transaction is committed. + # + # Expect all data is replicated on subscriber side after the commit. + # Note: both publisher and subscriber do crash/restart. + ############################### + + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + $node_subscriber->stop('immediate'); + $node_publisher->stop('immediate'); + + $node_publisher->start; + $node_subscriber->start; + + # commit post the restart + $node_publisher->safe_psql('postgres', + "COMMIT PREPARED 'test_prepared_tab';"); + $node_publisher->wait_for_catchup($appname); + + # check inserts are visible + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), + 'Rows inserted by 2PC have committed on subscriber, and extra columns contain local defaults' + ); + + ############################### + # Do INSERT after the PREPARE but before ROLLBACK PREPARED. + # 1. Table is deleted back to 2 rows which are replicated on subscriber. + # 2. Data is streamed using 2PC. + # 3. A single row INSERT is done which is after the PREPARE. + # 4. Then do a ROLLBACK PREPARED. + # + # Expect the 2PC data rolls back leaving only 3 rows on the subscriber + # (the original 2 + inserted 1). + ############################### + + # First, delete the data except for 2 rows (will be replicated) + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE a > 2;"); + + # Then insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is in prepared state on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber'); + + # Insert a different record (now we are outside of the 2PC transaction) + # Note: the 2PC transaction still holds row locks so make sure this insert is for a separate primary key + $node_publisher->safe_psql('postgres', + "INSERT INTO test_tab VALUES (99999, 'foobar')"); + + # 2PC transaction gets aborted + $node_publisher->safe_psql('postgres', + "ROLLBACK PREPARED 'test_prepared_tab';"); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is aborted on subscriber, + # but the extra INSERT outside of the 2PC still was replicated + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3|3|3), + 'check the outside insert was copied to subscriber'); + + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is aborted on subscriber'); + + ############################### + # Do INSERT after the PREPARE but before COMMIT PREPARED. + # 1. Table is deleted back to 2 rows which are replicated on subscriber. + # 2. Data is streamed using 2PC. + # 3. A single row INSERT is done which is after the PREPARE. + # 4. Then do a COMMIT PREPARED. + # + # Expect 2PC data + the extra row are on the subscriber + # (the 3334 + inserted 1 = 3335). + ############################### + + # First, delete the data except for 2 rows (will be replicated) + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE a > 2;"); + + # Then insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is in prepared state on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber'); + + # Insert a different record (now we are outside of the 2PC transaction) + # Note: the 2PC transaction still holds row locks so make sure this insert is for a separare primary key + $node_publisher->safe_psql('postgres', + "INSERT INTO test_tab VALUES (99999, 'foobar')"); + + # 2PC transaction gets committed + $node_publisher->safe_psql('postgres', + "COMMIT PREPARED 'test_prepared_tab';"); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is committed on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3335|3335|3335), + 'Rows inserted by 2PC (as well as outside insert) have committed on subscriber, and extra columns contain local defaults' + ); + + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is committed on subscriber'); + + # Cleanup the test data + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE a > 2;"); + $node_publisher->wait_for_catchup($appname); +} + ############################### # Setup ############################### @@ -48,6 +308,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql( 'postgres', " CREATE SUBSCRIPTION tap_sub @@ -70,236 +334,30 @@ my $twophase_query = $node_subscriber->poll_query_until('postgres', $twophase_query) or die "Timed out while waiting for subscriber to enable twophase"; -############################### # Check initial data was copied to subscriber -############################### my $result = $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab"); is($result, qq(2|2|2), 'check initial data was copied to subscriber'); -############################### -# Test 2PC PREPARE / COMMIT PREPARED. -# 1. Data is streamed as a 2PC transaction. -# 2. Then do commit prepared. -# -# Expect all data is replicated on subscriber side after the commit. -############################### - -# check that 2PC gets replicated to subscriber -# Insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is in prepared state on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber'); - -# 2PC transaction gets committed -$node_publisher->safe_psql('postgres', - "COMMIT PREPARED 'test_prepared_tab';"); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is committed on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), - 'Rows inserted by 2PC have committed on subscriber, and extra columns contain local defaults' -); -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is committed on subscriber'); - -############################### -# Test 2PC PREPARE / ROLLBACK PREPARED. -# 1. Table is deleted back to 2 rows which are replicated on subscriber. -# 2. Data is streamed using 2PC. -# 3. Do rollback prepared. -# -# Expect data rolls back leaving only the original 2 rows. -############################### - -# First, delete the data except for 2 rows (will be replicated) -$node_publisher->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); - -# Then insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is in prepared state on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber'); - -# 2PC transaction gets aborted -$node_publisher->safe_psql('postgres', - "ROLLBACK PREPARED 'test_prepared_tab';"); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is aborted on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(2|2|2), - 'Rows inserted by 2PC are rolled back, leaving only the original 2 rows'); - -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is aborted on subscriber'); - -############################### -# Check that 2PC COMMIT PREPARED is decoded properly on crash restart. -# 1. insert, update and delete enough rows to exceed the 64kB limit. -# 2. Then server crashes before the 2PC transaction is committed. -# 3. After servers are restarted the pending transaction is committed. -# -# Expect all data is replicated on subscriber side after the commit. -# Note: both publisher and subscriber do crash/restart. -############################### - -$node_publisher->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_subscriber->stop('immediate'); -$node_publisher->stop('immediate'); - -$node_publisher->start; -$node_subscriber->start; - -# commit post the restart -$node_publisher->safe_psql('postgres', - "COMMIT PREPARED 'test_prepared_tab';"); -$node_publisher->wait_for_catchup($appname); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -# check inserts are visible -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), - 'Rows inserted by 2PC have committed on subscriber, and extra columns contain local defaults' +################################### +# Test using streaming mode 'apply' +################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" ); -############################### -# Do INSERT after the PREPARE but before ROLLBACK PREPARED. -# 1. Table is deleted back to 2 rows which are replicated on subscriber. -# 2. Data is streamed using 2PC. -# 3. A single row INSERT is done which is after the PREPARE. -# 4. Then do a ROLLBACK PREPARED. -# -# Expect the 2PC data rolls back leaving only 3 rows on the subscriber -# (the original 2 + inserted 1). -############################### - -# First, delete the data except for 2 rows (will be replicated) -$node_publisher->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); - -# Then insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is in prepared state on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber'); - -# Insert a different record (now we are outside of the 2PC transaction) -# Note: the 2PC transaction still holds row locks so make sure this insert is for a separate primary key -$node_publisher->safe_psql('postgres', - "INSERT INTO test_tab VALUES (99999, 'foobar')"); - -# 2PC transaction gets aborted -$node_publisher->safe_psql('postgres', - "ROLLBACK PREPARED 'test_prepared_tab';"); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is aborted on subscriber, -# but the extra INSERT outside of the 2PC still was replicated -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3|3|3), 'check the outside insert was copied to subscriber'); - -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is aborted on subscriber'); - -############################### -# Do INSERT after the PREPARE but before COMMIT PREPARED. -# 1. Table is deleted back to 2 rows which are replicated on subscriber. -# 2. Data is streamed using 2PC. -# 3. A single row INSERT is done which is after the PREPARE. -# 4. Then do a COMMIT PREPARED. -# -# Expect 2PC data + the extra row are on the subscriber -# (the 3334 + inserted 1 = 3335). -############################### - -# First, delete the data except for 2 rows (will be replicated) -$node_publisher->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); - -# Then insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is in prepared state on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber'); - -# Insert a different record (now we are outside of the 2PC transaction) -# Note: the 2PC transaction still holds row locks so make sure this insert is for a separare primary key -$node_publisher->safe_psql('postgres', - "INSERT INTO test_tab VALUES (99999, 'foobar')"); - -# 2PC transaction gets committed -$node_publisher->safe_psql('postgres', - "COMMIT PREPARED 'test_prepared_tab';"); - -$node_publisher->wait_for_catchup($appname); +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = apply)"); -# check that transaction is committed on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3335|3335|3335), - 'Rows inserted by 2PC (as well as outside insert) have committed on subscriber, and extra columns contain local defaults' -); +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is committed on subscriber'); +test_streaming($node_publisher, $node_subscriber, $appname, 1); ############################### # check all the cleanup -- 2.23.0.windows.1 [application/octet-stream] v11-0003-A-temporary-patch-that-includes-patches-in-anoth.patch (11.5K, ../../OS3PR01MB6275C4679CFE3414340A259B9EAD9@OS3PR01MB6275.jpnprd01.prod.outlook.com/4-v11-0003-A-temporary-patch-that-includes-patches-in-anoth.patch) download | inline diff: From b9d55bb9321f2b886762da05130160628a8c8325 Mon Sep 17 00:00:00 2001 From: wangw <[email protected]> Date: Wed, 15 Jun 2022 16:15:50 +0800 Subject: [PATCH v11 3/5] A temporary patch that includes patches in another bugfix thread. The partition cache map on subscriber have several bugs (see thread [1]). Because patch 0004 is developed based on the patches in [1], so I merged the latest 0002-patch (see [2]) and the latest 0003-patch (see [3]) into a temporary patch 0003 here. After the patches in [1] are committed, I will delete this temporary patch. [1] - https://www.postgresql.org/message-id/OSZPR01MB6310F46CD425A967E4AEF736FDA49%40OSZPR01MB6310.jpnprd01.prod.outlook.com [2] - https://www.postgresql.org/message-id/OSZPR01MB6310757179D431D5480AF6CCFDAD9%40OSZPR01MB6310.jpnprd01.prod.outlook.com [3] - https://www.postgresql.org/message-id/OSZPR01MB63107255AE453AB182524523FDAA9%40OSZPR01MB6310.jpnprd01.prod.outlook.com --- src/backend/replication/logical/relation.c | 155 ++++++++++++++------- src/backend/replication/logical/worker.c | 23 ++- src/include/replication/logicalrelation.h | 1 + src/test/subscription/t/013_partition.pl | 29 ++++ 4 files changed, 151 insertions(+), 57 deletions(-) diff --git a/src/backend/replication/logical/relation.c b/src/backend/replication/logical/relation.c index 9c9ec144d8..18e657cdfe 100644 --- a/src/backend/replication/logical/relation.c +++ b/src/backend/replication/logical/relation.c @@ -249,6 +249,72 @@ logicalrep_report_missing_attrs(LogicalRepRelation *remoterel, } } +/* + * Check if replica identity matches and mark the updatable flag. + * + * We allow for stricter replica identity (fewer columns) on subscriber as + * that will not stop us from finding unique tuple. IE, if publisher has + * identity (id,timestamp) and subscriber just (id) this will not be a + * problem, but in the opposite scenario it will. + * + * Don't throw any error here just mark the relation entry as not updatable, + * as replica identity is only for updates and deletes but inserts can be + * replicated even without it. + */ +static void +logicalrep_rel_mark_updatable(LogicalRepRelMapEntry *entry) +{ + Bitmapset *idkey; + LogicalRepRelation *remoterel = &entry->remoterel; + int i; + + entry->updatable = true; + + /* + * If it is a partitioned table, we don't check it, we will check its + * partition later. + */ + if (entry->localrel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + return; + + idkey = RelationGetIndexAttrBitmap(entry->localrel, + INDEX_ATTR_BITMAP_IDENTITY_KEY); + /* fallback to PK if no replica identity */ + if (idkey == NULL) + { + idkey = RelationGetIndexAttrBitmap(entry->localrel, + INDEX_ATTR_BITMAP_PRIMARY_KEY); + /* + * If no replica identity index and no PK, the published table + * must have replica identity FULL. + */ + if (idkey == NULL && remoterel->replident != REPLICA_IDENTITY_FULL) + entry->updatable = false; + } + + i = -1; + while ((i = bms_next_member(idkey, i)) >= 0) + { + int attnum = i + FirstLowInvalidHeapAttributeNumber; + + if (!AttrNumberIsForUserDefinedAttr(attnum)) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("logical replication target relation \"%s.%s\" uses " + "system columns in REPLICA IDENTITY index", + remoterel->nspname, remoterel->relname))); + + attnum = AttrNumberGetAttrOffset(attnum); + + if (entry->attrmap->attnums[attnum] < 0 || + !bms_is_member(entry->attrmap->attnums[attnum], remoterel->attkeys)) + { + entry->updatable = false; + break; + } + } +} + /* * Open the local relation associated with the remote one. * @@ -307,7 +373,6 @@ logicalrep_rel_open(LogicalRepRelId remoteid, LOCKMODE lockmode) if (!entry->localrelvalid) { Oid relid; - Bitmapset *idkey; TupleDesc desc; MemoryContext oldctx; int i; @@ -365,55 +430,8 @@ logicalrep_rel_open(LogicalRepRelId remoteid, LOCKMODE lockmode) /* be tidy */ bms_free(missingatts); - /* - * Check that replica identity matches. We allow for stricter replica - * identity (fewer columns) on subscriber as that will not stop us - * from finding unique tuple. IE, if publisher has identity - * (id,timestamp) and subscriber just (id) this will not be a problem, - * but in the opposite scenario it will. - * - * Don't throw any error here just mark the relation entry as not - * updatable, as replica identity is only for updates and deletes but - * inserts can be replicated even without it. - */ - entry->updatable = true; - idkey = RelationGetIndexAttrBitmap(entry->localrel, - INDEX_ATTR_BITMAP_IDENTITY_KEY); - /* fallback to PK if no replica identity */ - if (idkey == NULL) - { - idkey = RelationGetIndexAttrBitmap(entry->localrel, - INDEX_ATTR_BITMAP_PRIMARY_KEY); - - /* - * If no replica identity index and no PK, the published table - * must have replica identity FULL. - */ - if (idkey == NULL && remoterel->replident != REPLICA_IDENTITY_FULL) - entry->updatable = false; - } - - i = -1; - while ((i = bms_next_member(idkey, i)) >= 0) - { - int attnum = i + FirstLowInvalidHeapAttributeNumber; - - if (!AttrNumberIsForUserDefinedAttr(attnum)) - ereport(ERROR, - (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), - errmsg("logical replication target relation \"%s.%s\" uses " - "system columns in REPLICA IDENTITY index", - remoterel->nspname, remoterel->relname))); - - attnum = AttrNumberGetAttrOffset(attnum); - - if (entry->attrmap->attnums[attnum] < 0 || - !bms_is_member(entry->attrmap->attnums[attnum], remoterel->attkeys)) - { - entry->updatable = false; - break; - } - } + /* Check that replica identity matches. */ + logicalrep_rel_mark_updatable(entry); entry->localrelvalid = true; } @@ -486,6 +504,40 @@ logicalrep_partmap_invalidate_cb(Datum arg, Oid reloid) } } +/* + * Reset the entries in the partition map that refer to remoterel. + * + * Called when new relation mapping is sent by the publisher to update our + * expected view of incoming data from said publisher. + * + * Note that we don't update the remoterel information in the entry here, + * we will update the information in logicalrep_partition_open to avoid + * unnecessary work. + */ +void +logicalrep_partmap_reset_relmap(LogicalRepRelation *remoterel) +{ + HASH_SEQ_STATUS status; + LogicalRepPartMapEntry *part_entry; + LogicalRepRelMapEntry *entry; + + if (LogicalRepPartMap == NULL) + return; + + hash_seq_init(&status, LogicalRepPartMap); + while ((part_entry = (LogicalRepPartMapEntry *) hash_seq_search(&status)) != NULL) + { + entry = &part_entry->relmapentry; + + if (entry->remoterel.remoteid != remoterel->remoteid) + continue; + + logicalrep_relmap_free_entry(entry); + + memset(entry, 0, sizeof(LogicalRepRelMapEntry)); + } +} + /* * Initialize the partition map cache. */ @@ -617,7 +669,8 @@ logicalrep_partition_open(LogicalRepRelMapEntry *root, attrmap->maplen * sizeof(AttrNumber)); } - entry->updatable = root->updatable; + /* Check that replica identity matches. */ + logicalrep_rel_mark_updatable(entry); entry->localrelvalid = true; diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c index 3f8dea0fc1..dac85a4101 100644 --- a/src/backend/replication/logical/worker.c +++ b/src/backend/replication/logical/worker.c @@ -1803,6 +1803,9 @@ apply_handle_relation(StringInfo s) rel = logicalrep_read_rel(s); logicalrep_relmap_update(rel); + + /* Also reset all entries in the partition map that refer to remoterel. */ + logicalrep_partmap_reset_relmap(rel); } /* @@ -2359,6 +2362,8 @@ apply_handle_tuple_routing(ApplyExecutionData *edata, TupleTableSlot *remoteslot_part; TupleConversionMap *map; MemoryContext oldctx; + LogicalRepRelMapEntry *part_entry = NULL; + AttrMap *attrmap = NULL; /* ModifyTableState is needed for ExecFindPartition(). */ edata->mtstate = mtstate = makeNode(ModifyTableState); @@ -2390,8 +2395,11 @@ apply_handle_tuple_routing(ApplyExecutionData *edata, remoteslot_part = table_slot_create(partrel, &estate->es_tupleTable); map = partrelinfo->ri_RootToPartitionMap; if (map != NULL) - remoteslot_part = execute_attr_map_slot(map->attrMap, remoteslot, + { + attrmap = map->attrMap; + remoteslot_part = execute_attr_map_slot(attrmap, remoteslot, remoteslot_part); + } else { remoteslot_part = ExecCopySlot(remoteslot_part, remoteslot); @@ -2399,6 +2407,14 @@ apply_handle_tuple_routing(ApplyExecutionData *edata, } MemoryContextSwitchTo(oldctx); + /* Check if we can do the update or delete on the leaf partition. */ + if(operation == CMD_UPDATE || operation == CMD_DELETE) + { + part_entry = logicalrep_partition_open(relmapentry, partrel, + attrmap); + check_relation_updatable(part_entry); + } + switch (operation) { case CMD_INSERT: @@ -2420,15 +2436,10 @@ apply_handle_tuple_routing(ApplyExecutionData *edata, * suitable partition. */ { - AttrMap *attrmap = map ? map->attrMap : NULL; - LogicalRepRelMapEntry *part_entry; TupleTableSlot *localslot; ResultRelInfo *partrelinfo_new; bool found; - part_entry = logicalrep_partition_open(relmapentry, partrel, - attrmap); - /* Get the matching local tuple from the partition. */ found = FindReplTupleInLocalRel(estate, partrel, &part_entry->remoterel, diff --git a/src/include/replication/logicalrelation.h b/src/include/replication/logicalrelation.h index 7bf8cd22bd..78cd7e77f5 100644 --- a/src/include/replication/logicalrelation.h +++ b/src/include/replication/logicalrelation.h @@ -38,6 +38,7 @@ typedef struct LogicalRepRelMapEntry } LogicalRepRelMapEntry; extern void logicalrep_relmap_update(LogicalRepRelation *remoterel); +extern void logicalrep_partmap_reset_relmap(LogicalRepRelation *remoterel); extern LogicalRepRelMapEntry *logicalrep_rel_open(LogicalRepRelId remoteid, LOCKMODE lockmode); diff --git a/src/test/subscription/t/013_partition.pl b/src/test/subscription/t/013_partition.pl index 06f9215018..e07a9217d7 100644 --- a/src/test/subscription/t/013_partition.pl +++ b/src/test/subscription/t/013_partition.pl @@ -853,4 +853,33 @@ $result = $node_subscriber2->safe_psql('postgres', "SELECT a, b, c FROM tab5 ORDER BY 1"); is($result, qq(3|1|), 'updates of tab5 replicated correctly after altering table on subscriber'); +# Test that replication into the partitioned target table continues to +# work correctly when the published table is altered. +$node_publisher->safe_psql( + 'postgres', q{ + ALTER TABLE tab5 DROP COLUMN b, ADD COLUMN c INT; + ALTER TABLE tab5 ADD COLUMN b INT;}); + +$node_publisher->safe_psql('postgres', "UPDATE tab5 SET c = 1 WHERE a = 3"); + +$node_publisher->wait_for_catchup('sub2'); + +$result = $node_subscriber2->safe_psql('postgres', + "SELECT a, b, c FROM tab5 ORDER BY 1"); +is($result, qq(3||1), 'updates of tab5 replicated correctly after altering table on publisher'); + +# Alter REPLICA IDENTITY on subscriber. +# No REPLICA IDENTITY in the partitioned table on subscriber, but what we check +# is the partition, so it works fine. +$node_subscriber2->safe_psql('postgres', + "ALTER TABLE tab5 REPLICA IDENTITY NOTHING"); + +$node_publisher->safe_psql('postgres', "UPDATE tab5 SET a = 4 WHERE a = 3"); + +$node_publisher->wait_for_catchup('sub2'); + +$result = $node_subscriber2->safe_psql('postgres', + "SELECT a, b, c FROM tab5_1 ORDER BY 1"); +is($result, qq(4||1), 'updates of tab5 replicated correctly'); + done_testing(); -- 2.23.0.windows.1 [application/octet-stream] v11-0004-Add-some-checks-before-using-apply-background-wo.patch (35.0K, ../../OS3PR01MB6275C4679CFE3414340A259B9EAD9@OS3PR01MB6275.jpnprd01.prod.outlook.com/5-v11-0004-Add-some-checks-before-using-apply-background-wo.patch) download | inline diff: From 93f0e0e5804404860560d2944c04d80cfa689f03 Mon Sep 17 00:00:00 2001 From: wangw <[email protected]> Date: Tue, 14 Jun 2022 11:23:52 +0800 Subject: [PATCH v11 4/5] Add some checks before using apply background worker to apply changes. If any of the following checks are violated, an error will be reported. 1. The unique columns between publisher and subscriber are difference. 2. There is any non-immutable function present in expression in subscriber's relation. Check from the following 3 items: a. The function in triggers; b. Column default value expressions and domain constraints; c. Constraint expressions. --- doc/src/sgml/ref/create_subscription.sgml | 4 + .../replication/logical/applybgwroker.c | 45 ++ src/backend/replication/logical/proto.c | 62 ++- src/backend/replication/logical/relation.c | 198 +++++++++ src/backend/replication/logical/tablesync.c | 1 + src/backend/replication/logical/worker.c | 15 +- src/backend/utils/cache/typcache.c | 17 + src/include/replication/logicalproto.h | 1 + src/include/replication/logicalrelation.h | 16 + src/include/replication/worker_internal.h | 1 + src/include/utils/typcache.h | 2 + .../subscription/t/022_twophase_cascade.pl | 7 + .../subscription/t/032_streaming_apply.pl | 391 ++++++++++++++++++ 13 files changed, 752 insertions(+), 8 deletions(-) create mode 100644 src/test/subscription/t/032_streaming_apply.pl diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml index d4caae0222..8de1a23ce4 100644 --- a/doc/src/sgml/ref/create_subscription.sgml +++ b/doc/src/sgml/ref/create_subscription.sgml @@ -240,6 +240,10 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl the transaction is committed. Note that if an error happens when applying changes in a background worker, it might not report the finish LSN of the remote transaction in the server log. + To run in this mode, there are following two requirements. The first + is that the unique column should be the same between publisher and + subscriber; the second is that there should not be any non-immutable + function in subscriber-side replicated table. </para> </listitem> </varlistentry> diff --git a/src/backend/replication/logical/applybgwroker.c b/src/backend/replication/logical/applybgwroker.c index 1f52b155d7..7d3c61c2e9 100644 --- a/src/backend/replication/logical/applybgwroker.c +++ b/src/backend/replication/logical/applybgwroker.c @@ -763,3 +763,48 @@ apply_bgworker_subxact_info_add(TransactionId current_xid) MemoryContextSwitchTo(oldctx); } } + +/* + * Check if changes on this logical replication relation can be applied by + * apply background worker. + * + * Although we maintains the commit order by allowing only one process to + * commit at a time, our access order to the relation has changed. + * This could cause unexpected problems if the unique column on the replicated + * table is inconsistent with the publisher-side or contains non-immutable + * functions when applying transactions in the apply background worker. + */ +void +apply_bgworker_relation_check(LogicalRepRelMapEntry *rel) +{ + /* Check only we are in apply bgworker. */ + if (!am_apply_bgworker()) + return; + + /* + * If it is a partitioned table, we do not check it, we will check its + * partition later. + */ + if (rel->localrel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + return; + + /* + * If any unique index exist, check that they are same as remoterel. + */ + if (!rel->sameunique) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot replicate relation with different unique index"), + errhint("Please change the streaming option to 'on' instead of 'apply'."))); + + /* + * Check if there is any non-immutable function present in expression in + * this relation. + */ + if (rel->volatility == FUNCTION_NONIMMUTABLE) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot replicate relation. There is at least one non-immutable function"), + errhint("Please change the streaming option to 'on' instead of 'apply'."))); + +} diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c index a888038eb4..6af3302270 100644 --- a/src/backend/replication/logical/proto.c +++ b/src/backend/replication/logical/proto.c @@ -23,7 +23,8 @@ /* * Protocol message flags. */ -#define LOGICALREP_IS_REPLICA_IDENTITY 1 +#define LOGICALREP_IS_REPLICA_IDENTITY 0x0001 +#define LOGICALREP_IS_UNIQUE 0x0002 #define MESSAGE_TRANSACTIONAL (1<<0) #define TRUNCATE_CASCADE (1<<0) @@ -933,11 +934,55 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns) TupleDesc desc; int i; uint16 nliveatts = 0; - Bitmapset *idattrs = NULL; + Bitmapset *idattrs = NULL, + *attunique = NULL; bool replidentfull; desc = RelationGetDescr(rel); + if (rel->rd_rel->relhasindex) + { + List *indexoidlist = RelationGetIndexList(rel); + ListCell *indexoidscan; + + foreach(indexoidscan, indexoidlist) + { + Oid indexoid = lfirst_oid(indexoidscan); + Relation indexRel; + + /* Look up the description for index */ + indexRel = RelationIdGetRelation(indexoid); + + if (!RelationIsValid(indexRel)) + elog(ERROR, "could not open relation with OID %u", indexoid); + + if (indexRel->rd_index->indisunique) + { + int i; + + /* Add referenced attributes to idindexattrs */ + for (i = 0; i < indexRel->rd_index->indnatts; i++) + { + int attrnum = indexRel->rd_index->indkey.values[i]; + + /* + * We don't include non-key columns into idindexattrs + * bitmaps. See RelationGetIndexAttrBitmap. + */ + if (attrnum != 0) + { + if (i < indexRel->rd_index->indnkeyatts && + !bms_is_member(attrnum - FirstLowInvalidHeapAttributeNumber, attunique)) + attunique = bms_add_member(attunique, + attrnum - FirstLowInvalidHeapAttributeNumber); + } + } + } + RelationClose(indexRel); + } + list_free(indexoidlist); + } + /* send number of live attributes */ for (i = 0; i < desc->natts; i++) { @@ -976,6 +1021,10 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns) idattrs)) flags |= LOGICALREP_IS_REPLICA_IDENTITY; + if (bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber, + attunique)) + flags |= LOGICALREP_IS_UNIQUE; + pq_sendbyte(out, flags); /* attribute name */ @@ -1001,7 +1050,8 @@ logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel) int natts; char **attnames; Oid *atttyps; - Bitmapset *attkeys = NULL; + Bitmapset *attkeys = NULL, + *attunique = NULL; natts = pq_getmsgint(in, 2); attnames = palloc(natts * sizeof(char *)); @@ -1012,11 +1062,14 @@ logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel) { uint8 flags; - /* Check for replica identity column */ + /* Check for replica identity and unique column */ flags = pq_getmsgbyte(in); if (flags & LOGICALREP_IS_REPLICA_IDENTITY) attkeys = bms_add_member(attkeys, i); + if (flags & LOGICALREP_IS_UNIQUE) + attunique = bms_add_member(attunique, i); + /* attribute name */ attnames[i] = pstrdup(pq_getmsgstring(in)); @@ -1030,6 +1083,7 @@ logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel) rel->attnames = attnames; rel->atttyps = atttyps; rel->attkeys = attkeys; + rel->attunique = attunique; rel->natts = natts; } diff --git a/src/backend/replication/logical/relation.c b/src/backend/replication/logical/relation.c index 18e657cdfe..8e17f95137 100644 --- a/src/backend/replication/logical/relation.c +++ b/src/backend/replication/logical/relation.c @@ -19,12 +19,19 @@ #include "access/table.h" #include "catalog/namespace.h" +#include "catalog/pg_proc.h" #include "catalog/pg_subscription_rel.h" +#include "commands/trigger.h" #include "executor/executor.h" #include "nodes/makefuncs.h" +#include "optimizer/optimizer.h" #include "replication/logicalrelation.h" #include "replication/worker_internal.h" +#include "rewrite/rewriteHandler.h" #include "utils/inval.h" +#include "utils/lsyscache.h" +#include "utils/syscache.h" +#include "utils/typcache.h" static MemoryContext LogicalRepRelMapContext = NULL; @@ -91,6 +98,23 @@ logicalrep_relmap_invalidate_cb(Datum arg, Oid reloid) } } +/* + * Reset the flag volatility of all existing entry in the relation map cache. + */ +static void +logicalrep_relmap_reset_volatility_cb(Datum arg, int cacheid, uint32 hashvalue) +{ + HASH_SEQ_STATUS hash_seq; + LogicalRepRelMapEntry *entry; + + if (LogicalRepRelMap == NULL) + return; + + hash_seq_init(&hash_seq, LogicalRepRelMap); + while ((entry = hash_seq_search(&hash_seq)) != NULL) + entry->volatility = FUNCTION_UNKNOWN; +} + /* * Initialize the relation map cache. */ @@ -116,6 +140,9 @@ logicalrep_relmap_init(void) /* Watch for invalidation events. */ CacheRegisterRelcacheCallback(logicalrep_relmap_invalidate_cb, (Datum) 0); + CacheRegisterSyscacheCallback(PROCOID, + logicalrep_relmap_reset_volatility_cb, + (Datum) 0); } /* @@ -142,6 +169,7 @@ logicalrep_relmap_free_entry(LogicalRepRelMapEntry *entry) pfree(remoterel->atttyps); } bms_free(remoterel->attkeys); + bms_free(remoterel->attunique); if (entry->attrmap) pfree(entry->attrmap); @@ -190,6 +218,7 @@ logicalrep_relmap_update(LogicalRepRelation *remoterel) } entry->remoterel.replident = remoterel->replident; entry->remoterel.attkeys = bms_copy(remoterel->attkeys); + entry->remoterel.attunique = bms_copy(remoterel->attunique); MemoryContextSwitchTo(oldctx); } @@ -315,6 +344,162 @@ logicalrep_rel_mark_updatable(LogicalRepRelMapEntry *entry) } } +/* + * Helper function to mark flags for apply background worker when opening the + * local table. + */ +static void +logicalrep_rel_mark_apply_bgworker(LogicalRepRelMapEntry *entry) +{ + Bitmapset *ukey; + int i; + TupleDesc tupdesc; + int attnum; + List *fkeys = NIL; + + /* + * Check that the unique column in the relation on the subscriber-side is + * also the unique column on the publisher-side. + */ + entry->sameunique = true; + ukey = RelationGetIndexAttrBitmap(entry->localrel, + INDEX_ATTR_BITMAP_KEY); + + if (ukey) + { + i = -1; + while ((i = bms_next_member(ukey, i)) >= 0) + { + int attnum = i + FirstLowInvalidHeapAttributeNumber; + + attnum = AttrNumberGetAttrOffset(attnum); + + if (entry->attrmap->attnums[attnum] < 0 || + !bms_is_member(entry->attrmap->attnums[attnum], entry->remoterel.attunique)) + { + entry->sameunique = false; + break; + } + } + } + + /* + * Check whether there is any non-immutable function in the local table. + * + * a. The function in triggers; + * b. Column default value expressions and domain constraints; + * c. Constraint expressions; + * d. Foreign keys. + */ + if (entry->volatility != FUNCTION_UNKNOWN) + return; + + /* Initialize the flag. */ + entry->volatility = FUNCTION_IMMUTABLE; + + /* Check the trigger functions. */ + if (entry->localrel->trigdesc != NULL) + { + for (i = 0; i < entry->localrel->trigdesc->numtriggers; i++) + { + Trigger *trig = entry->localrel->trigdesc->triggers + i; + + if (trig->tgenabled != TRIGGER_FIRES_ALWAYS && + trig->tgenabled != TRIGGER_FIRES_ON_REPLICA) + continue; + + if (func_volatile(trig->tgfoid) != PROVOLATILE_IMMUTABLE) + { + entry->volatility = FUNCTION_NONIMMUTABLE; + return; + } + } + } + + /* Check the columns. */ + tupdesc = RelationGetDescr(entry->localrel); + for (attnum = 0; attnum < tupdesc->natts; attnum++) + { + Form_pg_attribute att = TupleDescAttr(tupdesc, attnum); + + /* We don't need info for dropped or generated attributes */ + if (att->attisdropped || att->attgenerated) + continue; + + /* + * We don't need to check columns that only exist on the + * subscriber + */ + if (entry->attrmap->attnums[attnum] < 0) + continue; + + if (att->atthasdef) + { + Node *defaultexpr; + + defaultexpr = build_column_default(entry->localrel, attnum + 1); + if (contain_mutable_functions(defaultexpr)) + { + entry->volatility = FUNCTION_NONIMMUTABLE; + return; + } + } + + /* + * If the column is of a DOMAIN type, determine whether + * that domain has any CHECK expressions that are not + * immutable. + */ + if (get_typtype(att->atttypid) == TYPTYPE_DOMAIN) + { + List *domain_constraints; + ListCell *lc; + + domain_constraints = GetDomainConstraints(att->atttypid); + + foreach(lc, domain_constraints) + { + DomainConstraintState *con = (DomainConstraintState *) lfirst(lc); + + if (con->check_expr && contain_mutable_functions((Node *) con->check_expr)) + { + entry->volatility = FUNCTION_NONIMMUTABLE; + return; + } + } + } + } + + + /* Check the constraints. */ + tupdesc = RelationGetDescr(entry->localrel); + + if (tupdesc->constr) + { + ConstrCheck *check = tupdesc->constr->check; + + /* + * Determine if there are any CHECK constraints which + * contains non-immutable function. + */ + for (i = 0; i < tupdesc->constr->num_check; i++) + { + Expr *check_expr = stringToNode(check[i].ccbin); + + if (contain_mutable_functions((Node *) check_expr)) + { + entry->volatility = FUNCTION_NONIMMUTABLE; + return; + } + } + } + + /* Check the foreign keys. */ + fkeys = RelationGetFKeyList(entry->localrel); + if (fkeys) + entry->volatility = FUNCTION_NONIMMUTABLE; +} + /* * Open the local relation associated with the remote one. * @@ -433,6 +618,12 @@ logicalrep_rel_open(LogicalRepRelId remoteid, LOCKMODE lockmode) /* Check that replica identity matches. */ logicalrep_rel_mark_updatable(entry); + /* + * Set flags to check later whether changes could be applied in the + * apply background worker. + */ + logicalrep_rel_mark_apply_bgworker(entry); + entry->localrelvalid = true; } @@ -629,6 +820,7 @@ logicalrep_partition_open(LogicalRepRelMapEntry *root, } entry->remoterel.replident = remoterel->replident; entry->remoterel.attkeys = bms_copy(remoterel->attkeys); + entry->remoterel.attunique = bms_copy(remoterel->attunique); } entry->localrel = partrel; @@ -672,6 +864,12 @@ logicalrep_partition_open(LogicalRepRelMapEntry *root, /* Check that replica identity matches. */ logicalrep_rel_mark_updatable(entry); + /* + * Set flags to check later whether changes could be applied in the apply + * background worker. + */ + logicalrep_rel_mark_apply_bgworker(entry); + entry->localrelvalid = true; /* state and statelsn are left set to 0. */ diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c index 8ffba7e2e5..3cdbf8b457 100644 --- a/src/backend/replication/logical/tablesync.c +++ b/src/backend/replication/logical/tablesync.c @@ -884,6 +884,7 @@ fetch_remote_table_info(char *nspname, char *relname, lrel->attnames = palloc0(MaxTupleAttributeNumber * sizeof(char *)); lrel->atttyps = palloc0(MaxTupleAttributeNumber * sizeof(Oid)); lrel->attkeys = NULL; + lrel->attunique = NULL; /* * Store the columns as a list of names. Ignore those that are not diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c index dac85a4101..6770973e39 100644 --- a/src/backend/replication/logical/worker.c +++ b/src/backend/replication/logical/worker.c @@ -1918,6 +1918,8 @@ apply_handle_insert(StringInfo s) /* Set relation for error callback */ apply_error_callback_arg.rel = rel; + apply_bgworker_relation_check(rel); + /* Initialize the executor state. */ edata = create_edata_for_relation(rel); estate = edata->estate; @@ -2054,6 +2056,8 @@ apply_handle_update(StringInfo s) /* Check if we can do the update. */ check_relation_updatable(rel); + apply_bgworker_relation_check(rel); + /* Initialize the executor state. */ edata = create_edata_for_relation(rel); estate = edata->estate; @@ -2222,6 +2226,8 @@ apply_handle_delete(StringInfo s) /* Check if we can do the delete. */ check_relation_updatable(rel); + apply_bgworker_relation_check(rel); + /* Initialize the executor state. */ edata = create_edata_for_relation(rel); estate = edata->estate; @@ -2407,13 +2413,14 @@ apply_handle_tuple_routing(ApplyExecutionData *edata, } MemoryContextSwitchTo(oldctx); + part_entry = logicalrep_partition_open(relmapentry, partrel, + attrmap); + + apply_bgworker_relation_check(part_entry); + /* Check if we can do the update or delete on the leaf partition. */ if(operation == CMD_UPDATE || operation == CMD_DELETE) - { - part_entry = logicalrep_partition_open(relmapentry, partrel, - attrmap); check_relation_updatable(part_entry); - } switch (operation) { diff --git a/src/backend/utils/cache/typcache.c b/src/backend/utils/cache/typcache.c index 808f9ebd0d..b248899d82 100644 --- a/src/backend/utils/cache/typcache.c +++ b/src/backend/utils/cache/typcache.c @@ -2540,6 +2540,23 @@ compare_values_of_enum(TypeCacheEntry *tcache, Oid arg1, Oid arg2) return 0; } +/* + * GetDomainConstraints --- get DomainConstraintState list of specified domain type + */ +List * +GetDomainConstraints(Oid type_id) +{ + TypeCacheEntry *typentry; + List *constraints = NIL; + + typentry = lookup_type_cache(type_id, TYPECACHE_DOMAIN_CONSTR_INFO); + + if(typentry->domainData != NULL) + constraints = typentry->domainData->constraints; + + return constraints; +} + /* * Load (or re-load) the enumData member of the typcache entry. */ diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h index 7bba77c9e7..a503eb62c2 100644 --- a/src/include/replication/logicalproto.h +++ b/src/include/replication/logicalproto.h @@ -108,6 +108,7 @@ typedef struct LogicalRepRelation char replident; /* replica identity */ char relkind; /* remote relation kind */ Bitmapset *attkeys; /* Bitmap of key columns */ + Bitmapset *attunique; /* Bitmap of unique columns */ } LogicalRepRelation; /* Type mapping info */ diff --git a/src/include/replication/logicalrelation.h b/src/include/replication/logicalrelation.h index 78cd7e77f5..4476cf7cec 100644 --- a/src/include/replication/logicalrelation.h +++ b/src/include/replication/logicalrelation.h @@ -15,6 +15,17 @@ #include "access/attmap.h" #include "replication/logicalproto.h" +/* + * States to determine volatility of the function in expressions in one + * relation. + */ +typedef enum RelFuncVolatility +{ + FUNCTION_UNKNOWN = 0, /* initializing */ + FUNCTION_IMMUTABLE, /* all functions are immutable function */ + FUNCTION_NONIMMUTABLE /* at least one non-immutable function */ +} RelFuncVolatility; + typedef struct LogicalRepRelMapEntry { LogicalRepRelation remoterel; /* key is remoterel.remoteid */ @@ -31,6 +42,11 @@ typedef struct LogicalRepRelMapEntry Relation localrel; /* relcache entry (NULL when closed) */ AttrMap *attrmap; /* map of local attributes to remote ones */ bool updatable; /* Can apply updates/deletes? */ + bool sameunique; /* Are all unique columns of the local + relation contained by the unique columns in + remote? */ + RelFuncVolatility volatility; /* all functions in localrel are + immutable function? */ /* Sync state. */ char state; diff --git a/src/include/replication/worker_internal.h b/src/include/replication/worker_internal.h index 08e63de013..dd49d8e2ec 100644 --- a/src/include/replication/worker_internal.h +++ b/src/include/replication/worker_internal.h @@ -196,6 +196,7 @@ extern void apply_bgworker_free(ApplyBgworkerState *wstate); extern void apply_bgworker_check_status(void); extern void apply_bgworker_set_status(ApplyBgworkerStatus status); extern void apply_bgworker_subxact_info_add(TransactionId current_xid); +extern void apply_bgworker_relation_check(LogicalRepRelMapEntry *rel); static inline bool am_tablesync_worker(void) diff --git a/src/include/utils/typcache.h b/src/include/utils/typcache.h index 431ad7f1b3..ed7c2e7f48 100644 --- a/src/include/utils/typcache.h +++ b/src/include/utils/typcache.h @@ -199,6 +199,8 @@ extern uint64 assign_record_type_identifier(Oid type_id, int32 typmod); extern int compare_values_of_enum(TypeCacheEntry *tcache, Oid arg1, Oid arg2); +extern List *GetDomainConstraints(Oid type_id); + extern size_t SharedRecordTypmodRegistryEstimate(void); extern void SharedRecordTypmodRegistryInit(SharedRecordTypmodRegistry *, diff --git a/src/test/subscription/t/022_twophase_cascade.pl b/src/test/subscription/t/022_twophase_cascade.pl index b52af74e35..81791c8d3a 100644 --- a/src/test/subscription/t/022_twophase_cascade.pl +++ b/src/test/subscription/t/022_twophase_cascade.pl @@ -39,6 +39,13 @@ sub test_streaming ALTER SUBSCRIPTION tap_sub_C SET (streaming = $streaming_mode)"); + if ($streaming_mode eq 'apply') + { + $node_C->safe_psql( + 'postgres', " + ALTER TABLE test_tab ALTER c DROP DEFAULT"); + } + # Wait for subscribers to finish initialization $node_A->poll_query_until( diff --git a/src/test/subscription/t/032_streaming_apply.pl b/src/test/subscription/t/032_streaming_apply.pl new file mode 100644 index 0000000000..84a6900b33 --- /dev/null +++ b/src/test/subscription/t/032_streaming_apply.pl @@ -0,0 +1,391 @@ +# Copyright (c) 2022, PostgreSQL Global Development Group + +# Test the restrictions of streaming mode "apply" in logical replication + +use strict; +use warnings; +use PostgreSQL::Test::Cluster; +use PostgreSQL::Test::Utils; +use Test::More; + +my $offset = 0; + +# Create publisher node +my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); +$node_publisher->init(allows_streaming => 'logical'); +$node_publisher->append_conf('postgresql.conf', + 'logical_decoding_work_mem = 64kB'); +$node_publisher->start; + +# Create subscriber node +my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber'); +$node_subscriber->init; +$node_subscriber->start; + +# Setup structure on publisher +$node_publisher->safe_psql('postgres', + "CREATE TABLE test_tab (a int primary key, b varchar)"); +$node_publisher->safe_psql('postgres', + "CREATE TABLE test_tab_partitioned (a int primary key, b varchar)"); + +# Setup structure on subscriber +# We need to test normal table and partition table. +$node_subscriber->safe_psql('postgres', + "CREATE TABLE test_tab (a int primary key, b varchar)"); +$node_subscriber->safe_psql('postgres', + "CREATE TABLE test_tab_partitioned (a int primary key, b varchar) PARTITION BY RANGE(a)"); +$node_subscriber->safe_psql('postgres', + "CREATE TABLE test_tab_partition (LIKE test_tab_partitioned)"); +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partitioned ATTACH PARTITION test_tab_partition DEFAULT" +); + +# Setup logical replication +my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres'; +$node_publisher->safe_psql('postgres', + "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); +$node_publisher->safe_psql('postgres', + "CREATE PUBLICATION tap_pub_partitioned FOR TABLE test_tab_partitioned"); + +my $appname = 'tap_sub'; +$node_subscriber->safe_psql( + 'postgres', " + CREATE SUBSCRIPTION tap_sub + CONNECTION '$publisher_connstr application_name=$appname' + PUBLICATION tap_pub, tap_pub_partitioned + WITH (streaming = apply, copy_data = false)"); + +$node_publisher->wait_for_catchup($appname); + +# It is not allowed that the unique index on the publisher and the subscriber +# is different. Check the error reported by background worker in this case. +# First we check the unique index on normal table. +$node_subscriber->safe_psql('postgres', + "CREATE UNIQUE INDEX test_tab_b_idx ON test_tab (b)"); + +# Check that a background worker starts if "streaming" option is +# specified as "apply". We have to look for the DEBUG1 log messages +# about that, so temporarily bump up the log verbosity. +$node_subscriber->append_conf('postgresql.conf', "log_min_messages = debug1"); +$node_subscriber->reload; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); +$node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); +$node_subscriber->reload; + +$node_subscriber->wait_for_log( + qr/ERROR: cannot replicate relation with different unique index/, + $offset); + +# Drop the unique index on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', "DROP INDEX test_tab_b_idx"); + +$node_publisher->wait_for_catchup($appname); + +my $result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); +is($result, qq(5000), 'data replicated to subscriber after dropping index'); + +# Then we check the unique index on partition table. +$node_subscriber->safe_psql('postgres', + "CREATE UNIQUE INDEX test_tab_b_partition_idx ON test_tab_partition (b)"); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab_partitioned SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log( + qr/ERROR: cannot replicate relation with different unique index/, + $offset); + +# Drop the unique index on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', "DROP INDEX test_tab_b_partition_idx"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); +is($result, qq(5000), 'data replicated to subscriber after dropping index'); + +# Triggers which execute non-immutable function are not allowed on the +# subscriber side. Check the error reported by background worker in this case. +# First we check the trigger function on normal table. +$node_subscriber->safe_psql( + 'postgres', qq{ +CREATE FUNCTION trigger_func() RETURNS TRIGGER AS \$\$ + BEGIN + RETURN NULL; + END +\$\$ language plpgsql; +CREATE TRIGGER insert_trig +BEFORE INSERT ON test_tab +FOR EACH ROW EXECUTE PROCEDURE trigger_func(); +ALTER TABLE test_tab ENABLE REPLICA TRIGGER insert_trig; +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', "DELETE FROM test_tab"); + +$node_subscriber->wait_for_log( + qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, + $offset); + +# Drop the trigger on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "DROP TRIGGER insert_trig ON test_tab"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); +is($result, qq(0), 'data replicated to subscriber after dropping trigger'); + +# Then we check the unique index on partition table. +$node_subscriber->safe_psql( + 'postgres', qq{ +CREATE TRIGGER insert_trig +BEFORE INSERT ON test_tab_partition +FOR EACH ROW EXECUTE PROCEDURE trigger_func(); +ALTER TABLE test_tab_partition ENABLE REPLICA TRIGGER insert_trig; +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', "DELETE FROM test_tab_partitioned"); + +$node_subscriber->wait_for_log( + qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, + $offset); + +# Drop the trigger on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "DROP TRIGGER insert_trig ON test_tab_partition"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); +is($result, qq(0), 'data replicated to subscriber after dropping trigger'); + +# It is not allowed that column default value expression contains a +# non-immutable function on the subscriber side. Check the error reported by +# background worker in this case. +# First we check the column default value expression on normal table. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab ALTER COLUMN b SET DEFAULT random()"); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log( + qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, + $offset); + +# Drop default value on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab ALTER COLUMN b DROP DEFAULT"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); +is($result, qq(5000), + 'data replicated to subscriber after dropping default value expression'); + +# Then we check the column default value expression on partition table. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partition ALTER COLUMN b SET DEFAULT random()"); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab_partitioned SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log( + qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, + $offset); + +# Drop default value on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partition ALTER COLUMN b DROP DEFAULT"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); +is($result, qq(5000), + 'data replicated to subscriber after dropping default value expression'); + +# It is not allowed that domain constraint expression contains a non-immutable +# function on the subscriber side. Check the error reported by background +# worker in this case. +# Because the column type of the partition table must be the same as its parent +# table, only test normal table here. +$node_subscriber->safe_psql( + 'postgres', qq{ +CREATE DOMAIN test_domain AS int CHECK(VALUE > random()); +ALTER TABLE test_tab ALTER COLUMN a TYPE test_domain; +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', "DELETE FROM test_tab"); + +$node_subscriber->wait_for_log( + qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, + $offset); + +# Drop domain constraint expression on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab ALTER COLUMN a TYPE int"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); +is($result, qq(0), + 'data replicated to subscriber after dropping domain constraint expression' +); + +# It is not allowed that constraint expression contains a non-immutable function +# on the subscriber side. Check the error reported by background worker in this +# case. +# First we check the constraint expression on normal table. +$node_subscriber->safe_psql( + 'postgres', qq{ +ALTER TABLE test_tab ADD CONSTRAINT test_tab_con check (a > random()); +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log( + qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, + $offset); + +# Drop constraint on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab DROP CONSTRAINT test_tab_con"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); +is($result, qq(5000), + 'data replicated to subscriber after dropping constraint expression'); + +# Then we check the constraint expression on partition table. +$node_subscriber->safe_psql( + 'postgres', qq{ +ALTER TABLE test_tab_partition ADD CONSTRAINT test_tab_con check (a > random()); +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', "DELETE FROM test_tab_partitioned"); + +$node_subscriber->wait_for_log( + qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, + $offset); + +# Drop constraint on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partition DROP CONSTRAINT test_tab_con"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); +is($result, qq(0), + 'data replicated to subscriber after dropping constraint expression'); + +# It is not allowed that foreign key on the subscriber side. Check the error +# reported by background worker in this case. +# First we check the foreign key on normal table. +$node_publisher->safe_psql('postgres', "DELETE FROM test_tab"); +$node_publisher->wait_for_catchup($appname); +$node_subscriber->safe_psql( + 'postgres', qq{ +CREATE TABLE test_tab_f (a int primary key); +ALTER TABLE test_tab ADD CONSTRAINT test_tabfk FOREIGN KEY(a) REFERENCES test_tab_f(a); +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log( + qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, + $offset); + +# Drop the foreign key constraint on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab DROP CONSTRAINT test_tabfk"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); +is($result, qq(5000), + 'data replicated to subscriber after dropping the foreign key'); + +# Then we check the foreign key on partition table. +$node_publisher->wait_for_catchup($appname); +$node_subscriber->safe_psql( + 'postgres', qq{ +CREATE TABLE test_tab_partition_f (a int primary key); +ALTER TABLE test_tab_partition ADD CONSTRAINT test_tab_patition_fk FOREIGN KEY(a) REFERENCES test_tab_partition_f(a); +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab_partitioned SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log( + qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, + $offset); + +# Drop the foreign key constraint on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partition DROP CONSTRAINT test_tab_patition_fk"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); +is($result, qq(5000), + 'data replicated to subscriber after dropping the foreign key'); + +$node_subscriber->stop; +$node_publisher->stop; + +done_testing(); -- 2.23.0.windows.1 [application/octet-stream] v11-0005-Retry-to-apply-streaming-xact-only-in-apply-work.patch (25.2K, ../../OS3PR01MB6275C4679CFE3414340A259B9EAD9@OS3PR01MB6275.jpnprd01.prod.outlook.com/6-v11-0005-Retry-to-apply-streaming-xact-only-in-apply-work.patch) download | inline diff: From 6f5437dd58df8837a6a2104d6295cdee7d3e12fd Mon Sep 17 00:00:00 2001 From: wangw <[email protected]> Date: Wed, 15 Jun 2022 11:02:08 +0800 Subject: [PATCH v11 5/5] Retry to apply streaming xact only in apply worker. --- doc/src/sgml/catalogs.sgml | 11 ++ doc/src/sgml/ref/create_subscription.sgml | 4 + src/backend/catalog/pg_subscription.c | 1 + src/backend/catalog/system_views.sql | 4 +- src/backend/commands/subscriptioncmds.c | 1 + .../replication/logical/applybgwroker.c | 13 +- src/backend/replication/logical/worker.c | 89 +++++++++++ src/bin/pg_dump/pg_dump.c | 5 +- src/include/catalog/pg_subscription.h | 4 + .../subscription/t/032_streaming_apply.pl | 139 ++++++++++-------- 10 files changed, 203 insertions(+), 68 deletions(-) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index 5cb39b18bd..928ad02ace 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -7907,6 +7907,17 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>subretry</structfield> <type>bool</type> + </para> + <para> + If true, the subscription will not try to apply streaming transaction + in <literal>apply</literal> mode. See + <xref linkend="sql-createsubscription"/> for more information. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>subconninfo</structfield> <type>text</type> diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml index 8de1a23ce4..70fbf81668 100644 --- a/doc/src/sgml/ref/create_subscription.sgml +++ b/doc/src/sgml/ref/create_subscription.sgml @@ -244,6 +244,10 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl is that the unique column should be the same between publisher and subscriber; the second is that there should not be any non-immutable function in subscriber-side replicated table. + When applying a streaming transaction, if either requirement is not + met, the background worker will exit with an error. And when retrying + later, we will try to apply this transaction in <literal>on</literal> + mode. </para> </listitem> </varlistentry> diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c index add51caadf..1824390d7d 100644 --- a/src/backend/catalog/pg_subscription.c +++ b/src/backend/catalog/pg_subscription.c @@ -71,6 +71,7 @@ GetSubscription(Oid subid, bool missing_ok) sub->stream = subform->substream; sub->twophasestate = subform->subtwophasestate; sub->disableonerr = subform->subdisableonerr; + sub->retry = subform->subretry; /* Get conninfo */ datum = SysCacheGetAttr(SUBSCRIPTIONOID, diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql index fedaed533b..10f4dd6785 100644 --- a/src/backend/catalog/system_views.sql +++ b/src/backend/catalog/system_views.sql @@ -1298,8 +1298,8 @@ REVOKE ALL ON pg_replication_origin_status FROM public; -- All columns of pg_subscription except subconninfo are publicly readable. REVOKE ALL ON pg_subscription FROM public; GRANT SELECT (oid, subdbid, subskiplsn, subname, subowner, subenabled, - subbinary, substream, subtwophasestate, subdisableonerr, subslotname, - subsynccommit, subpublications) + subbinary, substream, subtwophasestate, subdisableonerr, + subretry, subslotname, subsynccommit, subpublications) ON pg_subscription TO public; CREATE VIEW pg_stat_subscription_stats AS diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c index 4080bba987..38697184d9 100644 --- a/src/backend/commands/subscriptioncmds.c +++ b/src/backend/commands/subscriptioncmds.c @@ -663,6 +663,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt, LOGICALREP_TWOPHASE_STATE_PENDING : LOGICALREP_TWOPHASE_STATE_DISABLED); values[Anum_pg_subscription_subdisableonerr - 1] = BoolGetDatum(opts.disableonerr); + values[Anum_pg_subscription_subretry - 1] = BoolGetDatum(false); values[Anum_pg_subscription_subconninfo - 1] = CStringGetTextDatum(conninfo); if (opts.slot_name) diff --git a/src/backend/replication/logical/applybgwroker.c b/src/backend/replication/logical/applybgwroker.c index 7d3c61c2e9..02b1ef5e62 100644 --- a/src/backend/replication/logical/applybgwroker.c +++ b/src/backend/replication/logical/applybgwroker.c @@ -103,6 +103,13 @@ apply_bgworker_find_or_start(TransactionId xid, bool start) if (start && !XLogRecPtrIsInvalid(MySubscription->skiplsn)) return NULL; + if (start && MySubscription->retry) + { + elog(DEBUG1, "retry to apply an streaming transaction in apply " + "background worker"); + return NULL; + } + /* * For streaming transactions that are being applied in apply background * worker, we cannot decide whether to apply the change for a relation @@ -794,8 +801,7 @@ apply_bgworker_relation_check(LogicalRepRelMapEntry *rel) if (!rel->sameunique) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("cannot replicate relation with different unique index"), - errhint("Please change the streaming option to 'on' instead of 'apply'."))); + errmsg("cannot replicate relation with different unique index"))); /* * Check if there is any non-immutable function present in expression in @@ -804,7 +810,6 @@ apply_bgworker_relation_check(LogicalRepRelMapEntry *rel) if (rel->volatility == FUNCTION_NONIMMUTABLE) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("cannot replicate relation. There is at least one non-immutable function"), - errhint("Please change the streaming option to 'on' instead of 'apply'."))); + errmsg("cannot replicate relation. There is at least one non-immutable function"))); } diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c index 6770973e39..064b97a84a 100644 --- a/src/backend/replication/logical/worker.c +++ b/src/backend/replication/logical/worker.c @@ -374,6 +374,8 @@ static void clear_subscription_skip_lsn(XLogRecPtr finish_lsn); static inline void set_apply_error_context_xact(TransactionId xid, XLogRecPtr lsn); static inline void reset_apply_error_context_info(void); +static void set_subscription_retry(bool retry); + /* * Should this worker apply changes for given relation. * @@ -890,6 +892,9 @@ apply_handle_commit(StringInfo s) apply_handle_commit_internal(&commit_data); + /* Set the flag that we will not retry later. */ + set_subscription_retry(false); + /* Check the status of apply background worker if any. */ apply_bgworker_check_status(); @@ -1001,6 +1006,9 @@ apply_handle_prepare(StringInfo s) in_remote_transaction = false; + /* Set the flag that we will not retry later. */ + set_subscription_retry(false); + /* Check the status of apply background worker if any. */ apply_bgworker_check_status(); @@ -1054,6 +1062,9 @@ apply_handle_commit_prepared(StringInfo s) store_flush_position(prepare_data.end_lsn); in_remote_transaction = false; + /* Set the flag that we will not retry later. */ + set_subscription_retry(false); + /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(prepare_data.end_lsn); @@ -1109,6 +1120,9 @@ apply_handle_rollback_prepared(StringInfo s) store_flush_position(rollback_data.rollback_end_lsn); in_remote_transaction = false; + /* Set the flag that we will not retry later. */ + set_subscription_retry(false); + /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(rollback_data.rollback_end_lsn); @@ -1195,6 +1209,9 @@ apply_handle_stream_prepare(StringInfo s) /* unlink the files with serialized changes and subxact info. */ stream_cleanup_files(MyLogicalRepWorker->subid, prepare_data.xid); } + + /* Set the flag that we will not retry later. */ + set_subscription_retry(false); } in_remote_transaction = false; @@ -1615,6 +1632,9 @@ apply_handle_stream_abort(StringInfo s) */ else serialize_stream_abort(xid, subxid); + + /* Set the flag that we will not retry later. */ + set_subscription_retry(false); } } @@ -2786,6 +2806,9 @@ apply_handle_stream_commit(StringInfo s) /* unlink the files with serialized changes and subxact info */ stream_cleanup_files(MyLogicalRepWorker->subid, xid); } + + /* Set the flag that we will not retry later. */ + set_subscription_retry(false); } /* Check the status of apply background worker if any. */ @@ -3854,6 +3877,9 @@ start_table_sync(XLogRecPtr *origin_startpos, char **myslotname) } PG_CATCH(); { + /* Set the flag that we will retry later. */ + set_subscription_retry(true); + if (MySubscription->disableonerr) DisableSubscriptionAndExit(); else @@ -3892,6 +3918,9 @@ start_apply(XLogRecPtr origin_startpos) } PG_CATCH(); { + /* Set the flag that we will retry later. */ + set_subscription_retry(true); + if (MySubscription->disableonerr) DisableSubscriptionAndExit(); else @@ -4393,3 +4422,63 @@ reset_apply_error_context_info(void) apply_error_callback_arg.remote_attnum = -1; set_apply_error_context_xact(InvalidTransactionId, InvalidXLogRecPtr); } + +/* + * Set subretry of pg_subscription catalog. + * + * If retry is true, subscriber is about to exit with an error. Otherwise, it + * means that the changes was applied successfully. + */ +static void +set_subscription_retry(bool retry) +{ + Relation rel; + HeapTuple tup; + bool started_tx = false; + bool nulls[Natts_pg_subscription]; + bool replaces[Natts_pg_subscription]; + Datum values[Natts_pg_subscription]; + + if (MySubscription->retry == retry || + am_apply_bgworker()) + return; + + if (!IsTransactionState()) + { + StartTransactionCommand(); + started_tx = true; + } + + /* Look up the subscription in the catalog */ + rel = table_open(SubscriptionRelationId, RowExclusiveLock); + tup = SearchSysCacheCopy1(SUBSCRIPTIONOID, + ObjectIdGetDatum(MySubscription->oid)); + + if (!HeapTupleIsValid(tup)) + elog(ERROR, "subscription \"%s\" does not exist", MySubscription->name); + + LockSharedObject(SubscriptionRelationId, MySubscription->oid, 0, + AccessShareLock); + + /* Form a new tuple. */ + memset(values, 0, sizeof(values)); + memset(nulls, false, sizeof(nulls)); + memset(replaces, false, sizeof(replaces)); + + /* reset subretry */ + values[Anum_pg_subscription_subretry - 1] = BoolGetDatum(retry); + replaces[Anum_pg_subscription_subretry - 1] = true; + + tup = heap_modify_tuple(tup, RelationGetDescr(rel), values, nulls, + replaces); + + /* Update the catalog. */ + CatalogTupleUpdate(rel, &tup->t_self, tup); + + /* Cleanup. */ + heap_freetuple(tup); + table_close(rel, NoLock); + + if (started_tx) + CommitTransactionCommand(); +} diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 6f8b30abb0..9b3ffe0888 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -4472,8 +4472,9 @@ getSubscriptions(Archive *fout) ntups = PQntuples(res); /* - * Get subscription fields. We don't include subskiplsn in the dump as - * after restoring the dump this value may no longer be relevant. + * Get subscription fields. We don't include subskiplsn and subretry in + * the dump as after restoring the dump this value may no longer be + * relevant. */ i_tableoid = PQfnumber(res, "tableoid"); i_oid = PQfnumber(res, "oid"); diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h index 9b394a45fe..dc7597bf80 100644 --- a/src/include/catalog/pg_subscription.h +++ b/src/include/catalog/pg_subscription.h @@ -76,6 +76,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW bool subdisableonerr; /* True if a worker error should cause the * subscription to be disabled */ + bool subretry; /* True if the previous apply change failed. */ + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* Connection string to the publisher */ text subconninfo BKI_FORCE_NOT_NULL; @@ -115,6 +117,8 @@ typedef struct Subscription bool disableonerr; /* Indicates if the subscription should be * automatically disabled if a worker error * occurs */ + bool retry; /* Indicates if the previous apply change + * failed. */ char *conninfo; /* Connection string to the publisher */ char *slotname; /* Name of the replication slot */ char *synccommit; /* Synchronous commit setting for worker */ diff --git a/src/test/subscription/t/032_streaming_apply.pl b/src/test/subscription/t/032_streaming_apply.pl index 84a6900b33..8f2e254b39 100644 --- a/src/test/subscription/t/032_streaming_apply.pl +++ b/src/test/subscription/t/032_streaming_apply.pl @@ -58,7 +58,9 @@ $node_subscriber->safe_psql( $node_publisher->wait_for_catchup($appname); # It is not allowed that the unique index on the publisher and the subscriber -# is different. Check the error reported by background worker in this case. +# is different. Check the error reported by background worker in this case. And +# after retrying in apply worker, we check if the data is replicated +# successfully. # First we check the unique index on normal table. $node_subscriber->safe_psql('postgres', "CREATE UNIQUE INDEX test_tab_b_idx ON test_tab (b)"); @@ -82,14 +84,15 @@ $node_subscriber->wait_for_log( qr/ERROR: cannot replicate relation with different unique index/, $offset); -# Drop the unique index on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', "DROP INDEX test_tab_b_idx"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); my $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); -is($result, qq(5000), 'data replicated to subscriber after dropping index'); +is($result, qq(5000), 'data replicated to subscribers after retrying because of unique index'); + +# Drop the unique index on the subscriber. +$node_subscriber->safe_psql('postgres', "DROP INDEX test_tab_b_idx"); # Then we check the unique index on partition table. $node_subscriber->safe_psql('postgres', @@ -106,17 +109,20 @@ $node_subscriber->wait_for_log( qr/ERROR: cannot replicate relation with different unique index/, $offset); -# Drop the unique index on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', "DROP INDEX test_tab_b_partition_idx"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); -is($result, qq(5000), 'data replicated to subscriber after dropping index'); +is($result, qq(5000), 'data replicated to subscribers after retrying because of unique index'); + +# Drop the unique index on the subscriber. +$node_subscriber->safe_psql('postgres', "DROP INDEX test_tab_b_partition_idx"); # Triggers which execute non-immutable function are not allowed on the # subscriber side. Check the error reported by background worker in this case. +# And after retrying in apply worker, we check if the data is replicated +# successfully. # First we check the trigger function on normal table. $node_subscriber->safe_psql( 'postgres', qq{ @@ -140,15 +146,16 @@ $node_subscriber->wait_for_log( qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, $offset); -# Drop the trigger on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "DROP TRIGGER insert_trig ON test_tab"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); -is($result, qq(0), 'data replicated to subscriber after dropping trigger'); +is($result, qq(0), 'data replicated to subscribers after retrying because of trigger'); + +# Drop the trigger on the subscriber. +$node_subscriber->safe_psql('postgres', + "DROP TRIGGER insert_trig ON test_tab"); # Then we check the unique index on partition table. $node_subscriber->safe_psql( @@ -168,19 +175,21 @@ $node_subscriber->wait_for_log( qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, $offset); -# Drop the trigger on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "DROP TRIGGER insert_trig ON test_tab_partition"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); -is($result, qq(0), 'data replicated to subscriber after dropping trigger'); +is($result, qq(0), 'data replicated to subscribers after retrying because of trigger'); + +# Drop the trigger on the subscriber. +$node_subscriber->safe_psql('postgres', + "DROP TRIGGER insert_trig ON test_tab_partition"); # It is not allowed that column default value expression contains a # non-immutable function on the subscriber side. Check the error reported by -# background worker in this case. +# background worker in this case. And after retrying in apply worker, we check +# if the data is replicated successfully. # First we check the column default value expression on normal table. $node_subscriber->safe_psql('postgres', "ALTER TABLE test_tab ALTER COLUMN b SET DEFAULT random()"); @@ -196,16 +205,17 @@ $node_subscriber->wait_for_log( qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, $offset); -# Drop default value on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "ALTER TABLE test_tab ALTER COLUMN b DROP DEFAULT"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); is($result, qq(5000), - 'data replicated to subscriber after dropping default value expression'); + 'data replicated to subscribers after retrying because of column default value'); + +# Drop default value on the subscriber. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab ALTER COLUMN b DROP DEFAULT"); # Then we check the column default value expression on partition table. $node_subscriber->safe_psql('postgres', @@ -222,20 +232,22 @@ $node_subscriber->wait_for_log( qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, $offset); -# Drop default value on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "ALTER TABLE test_tab_partition ALTER COLUMN b DROP DEFAULT"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); is($result, qq(5000), - 'data replicated to subscriber after dropping default value expression'); + 'data replicated to subscribers after retrying because of column default value'); + +# Drop default value on the subscriber. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partition ALTER COLUMN b DROP DEFAULT"); # It is not allowed that domain constraint expression contains a non-immutable # function on the subscriber side. Check the error reported by background -# worker in this case. +# worker in this case. And after retrying in apply worker, we check if the data +# is replicated successfully. # Because the column type of the partition table must be the same as its parent # table, only test normal table here. $node_subscriber->safe_psql( @@ -253,21 +265,23 @@ $node_subscriber->wait_for_log( qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, $offset); -# Drop domain constraint expression on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "ALTER TABLE test_tab ALTER COLUMN a TYPE int"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); is($result, qq(0), - 'data replicated to subscriber after dropping domain constraint expression' + 'data replicated to subscribers after retrying because of domain' ); -# It is not allowed that constraint expression contains a non-immutable function -# on the subscriber side. Check the error reported by background worker in this -# case. +# Drop domain constraint expression on the subscriber. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab ALTER COLUMN a TYPE int"); + +# It is not allowed that constraint expression contains a non-immutable +# function on the subscriber side. Check the error reported by background +# worker in this case. And after retrying in apply worker, we check if the data +# is replicated successfully. # First we check the constraint expression on normal table. $node_subscriber->safe_psql( 'postgres', qq{ @@ -285,16 +299,17 @@ $node_subscriber->wait_for_log( qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, $offset); -# Drop constraint on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "ALTER TABLE test_tab DROP CONSTRAINT test_tab_con"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); is($result, qq(5000), - 'data replicated to subscriber after dropping constraint expression'); + 'data replicated to subscribers after retrying because of constraint'); + +# Drop constraint on the subscriber. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab DROP CONSTRAINT test_tab_con"); # Then we check the constraint expression on partition table. $node_subscriber->safe_psql( @@ -311,19 +326,21 @@ $node_subscriber->wait_for_log( qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, $offset); -# Drop constraint on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "ALTER TABLE test_tab_partition DROP CONSTRAINT test_tab_con"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); is($result, qq(0), - 'data replicated to subscriber after dropping constraint expression'); + 'data replicated to subscribers after retrying because of constraint'); + +# Drop constraint on the subscriber. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partition DROP CONSTRAINT test_tab_con"); # It is not allowed that foreign key on the subscriber side. Check the error -# reported by background worker in this case. +# reported by background worker in this case. And after retrying in apply +# worker, we check if the data is replicated successfully. # First we check the foreign key on normal table. $node_publisher->safe_psql('postgres', "DELETE FROM test_tab"); $node_publisher->wait_for_catchup($appname); @@ -344,16 +361,17 @@ $node_subscriber->wait_for_log( qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, $offset); -# Drop the foreign key constraint on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "ALTER TABLE test_tab DROP CONSTRAINT test_tabfk"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); is($result, qq(5000), - 'data replicated to subscriber after dropping the foreign key'); + 'data replicated to subscribers after retrying because of foreign key'); + +# Drop the foreign key constraint on the subscriber. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab DROP CONSTRAINT test_tabfk"); # Then we check the foreign key on partition table. $node_publisher->wait_for_catchup($appname); @@ -374,16 +392,17 @@ $node_subscriber->wait_for_log( qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, $offset); -# Drop the foreign key constraint on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "ALTER TABLE test_tab_partition DROP CONSTRAINT test_tab_patition_fk"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); is($result, qq(5000), - 'data replicated to subscriber after dropping the foreign key'); + 'data replicated to subscribers after retrying because of foreign key'); + +# Drop the foreign key constraint on the subscriber. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partition DROP CONSTRAINT test_tab_patition_fk"); $node_subscriber->stop; $node_publisher->stop; -- 2.23.0.windows.1 ^ permalink raw reply [nested|flat] 66+ messages in thread
* Re: Perform streaming logical transactions by background workers and parallel apply @ 2022-06-15 12:12 Amit Kapila <[email protected]> parent: [email protected] <[email protected]> 1 sibling, 1 reply; 66+ messages in thread From: Amit Kapila @ 2022-06-15 12:12 UTC (permalink / raw) To: [email protected] <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Peter Smith <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; PostgreSQL Hackers <[email protected]> On Tue, Jun 14, 2022 at 9:07 AM [email protected] <[email protected]> wrote: > > > Attach the new patches. > Only changed patches 0001, 0004 and added new separate patch 0005. > Few questions/comments on 0001 =========================== 1. In the commit message, I see: "We also need to allow stream_stop to complete by the apply background worker to avoid deadlocks because T-1's current stream of changes can update rows in conflicting order with T-2's next stream of changes." Thinking about this, won't the T-1 and T-2 deadlock on the publisher node as well if the above statement is true? 2. + <para> + The apply background workers are taken from the pool defined by + <varname>max_logical_replication_workers</varname>. + </para> + <para> + The default value is 3. This parameter can only be set in the + <filename>postgresql.conf</filename> file or on the server command + line. + </para> Is there a reason to choose this number as 3? Why not 2 similar to max_sync_workers_per_subscription? 3. + + <para> + Setting streaming mode to <literal>apply</literal> could export invalid LSN + as finish LSN of failed transaction. Changing the streaming mode and making + the same conflict writes the finish LSN of the failed transaction in the + server log if required. + </para> How will the user identify that this is an invalid LSN value and she shouldn't use it to SKIP the transaction? Can we change the second sentence to: "User should change the streaming mode to 'on' if they would instead wish to see the finish LSN on error. Users can use finish LSN to SKIP applying the transaction." I think we can give reference to docs where the SKIP feature is explained. 4. + * This file contains routines that are intended to support setting up, using, + * and tearing down a ApplyBgworkerState. + * Refer to the comments in file header of logical/worker.c to see more + * informations about apply background worker. Typo. /informations/information. Consider having an empty line between the above two lines. 5. +ApplyBgworkerState * +apply_bgworker_find_or_start(TransactionId xid, bool start) { ... ... + if (!TransactionIdIsValid(xid)) + return NULL; + + /* + * We don't start new background worker if we are not in streaming apply + * mode. + */ + if (MySubscription->stream != SUBSTREAM_APPLY) + return NULL; + + /* + * We don't start new background worker if user has set skiplsn as it's + * possible that user want to skip the streaming transaction. For + * streaming transaction, we need to spill the transaction to disk so that + * we can get the last LSN of the transaction to judge whether to skip + * before starting to apply the change. + */ + if (start && !XLogRecPtrIsInvalid(MySubscription->skiplsn)) + return NULL; + + /* + * For streaming transactions that are being applied in apply background + * worker, we cannot decide whether to apply the change for a relation + * that is not in the READY state (see should_apply_changes_for_rel) as we + * won't know remote_final_lsn by that time. So, we don't start new apply + * background worker in this case. + */ + if (start && !AllTablesyncsReady()) + return NULL; ... ... } Can we move some of these starting checks to a separate function like canstartapplybgworker()? -- With Regards, Amit Kapila. ^ permalink raw reply [nested|flat] 66+ messages in thread
* RE: Perform streaming logical transactions by background workers and parallel apply @ 2022-06-17 07:17 [email protected] <[email protected]> parent: Amit Kapila <[email protected]> 0 siblings, 4 replies; 66+ messages in thread From: [email protected] @ 2022-06-17 07:17 UTC (permalink / raw) To: Amit Kapila <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Peter Smith <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; PostgreSQL Hackers <[email protected]> On Wed, Jun 15, 2022 at 8:13 PM Amit Kapila <[email protected]> wrote: > Few questions/comments on 0001 > =========================== Thanks for your comments. > 1. > In the commit message, I see: "We also need to allow stream_stop to > complete by the apply background worker to avoid deadlocks because > T-1's current stream of changes can update rows in conflicting order > with T-2's next stream of changes." > > Thinking about this, won't the T-1 and T-2 deadlock on the publisher > node as well if the above statement is true? Yes, I think so. I think if table's unique index/constraint of the publisher and the subscriber are consistent, the deadlock will occur on the publisher-side. If it is inconsistent, deadlock may only occur in the subscriber. But since we added the check for these (see patch 0004), so it seems okay to not handle this at STREAM_STOP. BTW, I made the following improvements to the code (#a, #c are improved in 0004 patch, #b, #d and #e are improved in 0001 patch.) : a. I added some comments in the function apply_handle_stream_stop to explain why we do not need to allow stream_stop to complete by the apply background worker. b. I deleted related commit message in 0001 patch and the related comments in file header (worker.c). c. Renamed the function logicalrep_rel_mark_apply_bgworker to logicalrep_rel_mark_safe_in_apply_bgworker. Also did some slight improvements in this function. d. When apply worker sends stream xact messages to apply background worker, only wait for apply background worker to complete when commit, prepare and abort of toplevel xact. e. The state setting of apply background worker was not very accurate before, so improved this (see the invocations to function pgstat_report_activity in function LogicalApplyBgwLoop, apply_handle_stream_start and apply_handle_stream_abort). > 2. > + <para> > + The apply background workers are taken from the pool defined by > + <varname>max_logical_replication_workers</varname>. > + </para> > + <para> > + The default value is 3. This parameter can only be set in the > + <filename>postgresql.conf</filename> file or on the server command > + line. > + </para> > > Is there a reason to choose this number as 3? Why not 2 similar to > max_sync_workers_per_subscription? Improved the default as suggested. > 3. > + > + <para> > + Setting streaming mode to <literal>apply</literal> could export invalid LSN > + as finish LSN of failed transaction. Changing the streaming mode and making > + the same conflict writes the finish LSN of the failed transaction in the > + server log if required. > + </para> > > How will the user identify that this is an invalid LSN value and she > shouldn't use it to SKIP the transaction? Can we change the second > sentence to: "User should change the streaming mode to 'on' if they > would instead wish to see the finish LSN on error. Users can use > finish LSN to SKIP applying the transaction." I think we can give > reference to docs where the SKIP feature is explained. Improved the sentence as suggested. And I added the reference after the statement in your suggestion. It looks like: ``` ... Users can use finish LSN to SKIP applying the transaction by running <link linkend="sql-altersubscription"><command>ALTER SUBSCRIPTION ... SKIP</command></link>. ``` > 4. > + * This file contains routines that are intended to support setting up, using, > + * and tearing down a ApplyBgworkerState. > + * Refer to the comments in file header of logical/worker.c to see more > + * informations about apply background worker. > > Typo. /informations/information. > > Consider having an empty line between the above two lines. Improved the message as suggested. > 5. > +ApplyBgworkerState * > +apply_bgworker_find_or_start(TransactionId xid, bool start) > { > ... > ... > + if (!TransactionIdIsValid(xid)) > + return NULL; > + > + /* > + * We don't start new background worker if we are not in streaming apply > + * mode. > + */ > + if (MySubscription->stream != SUBSTREAM_APPLY) > + return NULL; > + > + /* > + * We don't start new background worker if user has set skiplsn as it's > + * possible that user want to skip the streaming transaction. For > + * streaming transaction, we need to spill the transaction to disk so that > + * we can get the last LSN of the transaction to judge whether to skip > + * before starting to apply the change. > + */ > + if (start && !XLogRecPtrIsInvalid(MySubscription->skiplsn)) > + return NULL; > + > + /* > + * For streaming transactions that are being applied in apply background > + * worker, we cannot decide whether to apply the change for a relation > + * that is not in the READY state (see should_apply_changes_for_rel) as we > + * won't know remote_final_lsn by that time. So, we don't start new apply > + * background worker in this case. > + */ > + if (start && !AllTablesyncsReady()) > + return NULL; > ... > ... > } > > Can we move some of these starting checks to a separate function like > canstartapplybgworker()? Improved as suggested. BTW, I rebased the temporary patch 0003 because one patch in thread [1] is committed (see commit b7658c24c7 in HEAD). Attach the new patches. Only changed patches 0001, 0004. Regards, Wang wei Attachments: [application/octet-stream] v12-0001-Perform-streaming-logical-transactions-by-backgr.patch (98.1K, ../../OS3PR01MB62753CDFD0DF1FCBC978AFB39EAF9@OS3PR01MB6275.jpnprd01.prod.outlook.com/2-v12-0001-Perform-streaming-logical-transactions-by-backgr.patch) download | inline diff: From 93b9390a56a651a82b832d19eba7c53dc44a9dd7 Mon Sep 17 00:00:00 2001 From: "houzj.fnst" <[email protected]> Date: Wed, 20 Apr 2022 16:45:07 +0800 Subject: [PATCH v12 1/5] Perform streaming logical transactions by background workers Currently, for large transactions, the publisher sends the data in multiple streams (changes divided into chunks depending upon logical_decoding_work_mem), and then on the subscriber-side, the apply worker writes the changes into temporary files and once it receives the commit, it reads from the file and applies the entire transaction. To improve the performance of such transactions, we can instead allow them to be applied via background workers. In this approach, we assign a new apply background worker (if available) as soon as the xact's first stream is received and the main apply worker will send changes to this new worker via shared memory. The apply background worker will directly apply the change instead of writing it to temporary files. We keep this worker assigned till the transaction commit is received and also wait for the worker to finish at commit. This preserves commit ordering and avoids writing to and reading from file in most cases. We still need to spill if there is no worker available. This patch also extends the SUBSCRIPTION 'streaming' option so that the user can control whether to apply the streaming transaction in an apply background worker or spill the change to disk. User can set the streaming option to 'on/off', 'apply'. For now, 'apply' means the streaming will be applied via a apply background worker if available. 'on' means the streaming transaction will be spilled to disk. --- doc/src/sgml/catalogs.sgml | 10 +- doc/src/sgml/config.sgml | 25 + doc/src/sgml/logical-replication.sgml | 9 + doc/src/sgml/protocol.sgml | 19 + doc/src/sgml/ref/create_subscription.sgml | 24 +- src/backend/access/transam/xact.c | 13 + src/backend/commands/subscriptioncmds.c | 66 +- src/backend/postmaster/bgworker.c | 3 + src/backend/replication/logical/Makefile | 3 +- .../replication/logical/applybgwroker.c | 775 ++++++++++++++++++ src/backend/replication/logical/decode.c | 10 +- src/backend/replication/logical/launcher.c | 112 ++- src/backend/replication/logical/origin.c | 26 +- src/backend/replication/logical/proto.c | 35 +- .../replication/logical/reorderbuffer.c | 10 +- src/backend/replication/logical/tablesync.c | 10 +- src/backend/replication/logical/worker.c | 645 +++++++++++---- src/backend/replication/pgoutput/pgoutput.c | 2 +- src/backend/utils/activity/wait_event.c | 3 + src/backend/utils/misc/guc.c | 12 + src/backend/utils/misc/postgresql.conf.sample | 1 + src/bin/pg_dump/pg_dump.c | 6 +- src/include/catalog/pg_subscription.h | 17 +- src/include/replication/logicallauncher.h | 1 + src/include/replication/logicalproto.h | 19 +- src/include/replication/logicalworker.h | 1 + src/include/replication/origin.h | 2 +- src/include/replication/reorderbuffer.h | 7 +- src/include/replication/worker_internal.h | 103 ++- src/include/utils/wait_event.h | 1 + src/test/regress/expected/subscription.out | 2 +- src/tools/pgindent/typedefs.list | 5 + 32 files changed, 1760 insertions(+), 217 deletions(-) create mode 100644 src/backend/replication/logical/applybgwroker.c diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index c00c93dd7b..5cb39b18bd 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -7873,11 +7873,15 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l <row> <entry role="catalog_table_entry"><para role="column_definition"> - <structfield>substream</structfield> <type>bool</type> + <structfield>substream</structfield> <type>char</type> </para> <para> - If true, the subscription will allow streaming of in-progress - transactions + Controls how to handle the streaming of in-progress transactions: + <literal>f</literal> = disallow streaming of in-progress transactions, + <literal>t</literal> = spill the changes of in-progress transactions to + disk and apply at once after the transaction is committed on the + publisher, + <literal>a</literal> = apply changes directly using a background worker </para></entry> </row> diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml index f273fc95b9..5f6f66aa87 100644 --- a/doc/src/sgml/config.sgml +++ b/doc/src/sgml/config.sgml @@ -4970,6 +4970,31 @@ ANY <replaceable class="parameter">num_sync</replaceable> ( <replaceable class=" </listitem> </varlistentry> + <varlistentry id="guc-max-apply-bgworkers-per-subscription" xreflabel="max_apply_bgworkers_per_subscription"> + <term><varname>max_apply_bgworkers_per_subscription</varname> (<type>integer</type>) + <indexterm> + <primary><varname>max_apply_bgworkers_per_subscription</varname> configuration parameter</primary> + </indexterm> + </term> + <listitem> + <para> + Maximum number of apply background workers per subscription. This + parameter controls the amount of parallelism of the streaming of + in-progress transactions if we set subscription option + <literal>streaming</literal> to <literal>apply</literal>. + </para> + <para> + The apply background workers are taken from the pool defined by + <varname>max_logical_replication_workers</varname>. + </para> + <para> + The default value is 2. This parameter can only be set in the + <filename>postgresql.conf</filename> file or on the server command + line. + </para> + </listitem> + </varlistentry> + </variablelist> </sect2> diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml index 145ea71d61..abf9c65486 100644 --- a/doc/src/sgml/logical-replication.sgml +++ b/doc/src/sgml/logical-replication.sgml @@ -948,6 +948,15 @@ CONTEXT: processing remote data for replication origin "pg_16395" during "INSER might not violate any constraint. This can easily make the subscriber inconsistent. </para> + + <para> + Setting streaming mode to <literal>apply</literal> could export invalid LSN + as finish LSN of failed transaction. User should change the streaming mode + to 'on' if they would instead wish to see the finish LSN on error. Users + can use finish LSN to SKIP applying the transaction by running <link + linkend="sql-altersubscription"><command>ALTER SUBSCRIPTION ... + SKIP</command></link>. + </para> </sect1> <sect1 id="logical-replication-restrictions"> diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml index bb30340892..34be1b2698 100644 --- a/doc/src/sgml/protocol.sgml +++ b/doc/src/sgml/protocol.sgml @@ -6809,6 +6809,25 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;" </listitem> </varlistentry> + <varlistentry> + <term>Int64 (XLogRecPtr)</term> + <listitem> + <para> + The LSN of the abort. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Int64 (TimestampTz)</term> + <listitem> + <para> + Abort timestamp of the transaction. The value is in number + of microseconds since PostgreSQL epoch (2000-01-01). + </para> + </listitem> + </varlistentry> + <varlistentry> <term>Int32 (TransactionId)</term> <listitem> diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml index 35b39c28da..d4caae0222 100644 --- a/doc/src/sgml/ref/create_subscription.sgml +++ b/doc/src/sgml/ref/create_subscription.sgml @@ -217,13 +217,29 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl </varlistentry> <varlistentry> - <term><literal>streaming</literal> (<type>boolean</type>)</term> + <term><literal>streaming</literal> (<type>enum</type>)</term> <listitem> <para> Specifies whether to enable streaming of in-progress transactions - for this subscription. By default, all transactions - are fully decoded on the publisher and only then sent to the - subscriber as a whole. + for this subscription. The default value is <literal>off</literal>, + all transactions are fully decoded on the publisher and only then + sent to the subscriber as a whole. + </para> + + <para> + If set to <literal>on</literal>, the changes of transaction are + written to temporary files and then applied at once after the + transaction is committed on the publisher. + </para> + + <para> + If set to <literal>apply</literal> incoming + changes are directly applied via one of the background workers, if + available. If no background worker is free to handle streaming + transaction then the changes are written to a file and applied after + the transaction is committed. Note that if an error happens when + applying changes in a background worker, it might not report the + finish LSN of the remote transaction in the server log. </para> </listitem> </varlistentry> diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c index 47d80b0d25..678540ba25 100644 --- a/src/backend/access/transam/xact.c +++ b/src/backend/access/transam/xact.c @@ -1711,6 +1711,7 @@ RecordTransactionAbort(bool isSubXact) int nchildren; TransactionId *children; TimestampTz xact_time; + bool replorigin; /* * If we haven't been assigned an XID, nobody will care whether we aborted @@ -1741,6 +1742,13 @@ RecordTransactionAbort(bool isSubXact) elog(PANIC, "cannot abort transaction %u, it was already committed", xid); + /* + * Are we using the replication origins feature? Or, in other words, + * are we replaying remote actions? + */ + replorigin = (replorigin_session_origin != InvalidRepOriginId && + replorigin_session_origin != DoNotReplicateId); + /* Fetch the data we need for the abort record */ nrels = smgrGetPendingDeletes(false, &rels); nchildren = xactGetCommittedChildren(&children); @@ -1765,6 +1773,11 @@ RecordTransactionAbort(bool isSubXact) MyXactFlags, InvalidTransactionId, NULL); + if (replorigin) + /* Move LSNs forward for this replication origin */ + replorigin_session_advance(replorigin_session_origin_lsn, + XactLastRecEnd); + /* * Report the latest async abort LSN, so that the WAL writer knows to * flush this abort. There's nothing to be gained by delaying this, since diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c index 83e6eae855..4080bba987 100644 --- a/src/backend/commands/subscriptioncmds.c +++ b/src/backend/commands/subscriptioncmds.c @@ -83,7 +83,7 @@ typedef struct SubOpts bool copy_data; bool refresh; bool binary; - bool streaming; + char streaming; bool twophase; bool disableonerr; XLogRecPtr lsn; @@ -95,6 +95,62 @@ static List *merge_publications(List *oldpublist, List *newpublist, bool addpub, static void ReportSlotConnectionError(List *rstates, Oid subid, char *slotname, char *err); +/* + * Extract the streaming mode value from a DefElem. This is like + * defGetBoolean() but also accepts the special value of "apply". + */ +static char +defGetStreamingMode(DefElem *def) +{ + /* + * If no parameter given, assume "true" is meant. + */ + if (def->arg == NULL) + return SUBSTREAM_ON; + + /* + * Allow 0, 1, "true", "false", "on", "off" or "apply". + */ + switch (nodeTag(def->arg)) + { + case T_Integer: + switch (intVal(def->arg)) + { + case 0: + return SUBSTREAM_OFF; + case 1: + return SUBSTREAM_ON; + default: + /* otherwise, error out below */ + break; + } + break; + default: + { + char *sval = defGetString(def); + + /* + * The set of strings accepted here should match up with the + * grammar's opt_boolean_or_string production. + */ + if (pg_strcasecmp(sval, "false") == 0 || + pg_strcasecmp(sval, "off") == 0) + return SUBSTREAM_OFF; + if (pg_strcasecmp(sval, "true") == 0 || + pg_strcasecmp(sval, "on") == 0) + return SUBSTREAM_ON; + if (pg_strcasecmp(sval, "apply") == 0) + return SUBSTREAM_APPLY; + } + break; + } + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s requires a Boolean value or \"apply\"", + def->defname))); + return SUBSTREAM_OFF; /* keep compiler quiet */ +} + /* * Common option parsing function for CREATE and ALTER SUBSCRIPTION commands. * @@ -132,7 +188,7 @@ parse_subscription_options(ParseState *pstate, List *stmt_options, if (IsSet(supported_opts, SUBOPT_BINARY)) opts->binary = false; if (IsSet(supported_opts, SUBOPT_STREAMING)) - opts->streaming = false; + opts->streaming = SUBSTREAM_OFF; if (IsSet(supported_opts, SUBOPT_TWOPHASE_COMMIT)) opts->twophase = false; if (IsSet(supported_opts, SUBOPT_DISABLE_ON_ERR)) @@ -233,7 +289,7 @@ parse_subscription_options(ParseState *pstate, List *stmt_options, errorConflictingDefElem(defel, pstate); opts->specified_opts |= SUBOPT_STREAMING; - opts->streaming = defGetBoolean(defel); + opts->streaming = defGetStreamingMode(defel); } else if (strcmp(defel->defname, "two_phase") == 0) { @@ -601,7 +657,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt, values[Anum_pg_subscription_subowner - 1] = ObjectIdGetDatum(owner); values[Anum_pg_subscription_subenabled - 1] = BoolGetDatum(opts.enabled); values[Anum_pg_subscription_subbinary - 1] = BoolGetDatum(opts.binary); - values[Anum_pg_subscription_substream - 1] = BoolGetDatum(opts.streaming); + values[Anum_pg_subscription_substream - 1] = CharGetDatum(opts.streaming); values[Anum_pg_subscription_subtwophasestate - 1] = CharGetDatum(opts.twophase ? LOGICALREP_TWOPHASE_STATE_PENDING : @@ -1060,7 +1116,7 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt, if (IsSet(opts.specified_opts, SUBOPT_STREAMING)) { values[Anum_pg_subscription_substream - 1] = - BoolGetDatum(opts.streaming); + CharGetDatum(opts.streaming); replaces[Anum_pg_subscription_substream - 1] = true; } diff --git a/src/backend/postmaster/bgworker.c b/src/backend/postmaster/bgworker.c index 40601aefd9..40ccb8993c 100644 --- a/src/backend/postmaster/bgworker.c +++ b/src/backend/postmaster/bgworker.c @@ -128,6 +128,9 @@ static const struct }, { "ApplyWorkerMain", ApplyWorkerMain + }, + { + "ApplyBgworkerMain", ApplyBgworkerMain } }; diff --git a/src/backend/replication/logical/Makefile b/src/backend/replication/logical/Makefile index c4e2fdeb71..a24a41969e 100644 --- a/src/backend/replication/logical/Makefile +++ b/src/backend/replication/logical/Makefile @@ -26,6 +26,7 @@ OBJS = \ reorderbuffer.o \ snapbuild.o \ tablesync.o \ - worker.o + worker.o \ + applybgwroker.o include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/replication/logical/applybgwroker.c b/src/backend/replication/logical/applybgwroker.c new file mode 100644 index 0000000000..7e8681590f --- /dev/null +++ b/src/backend/replication/logical/applybgwroker.c @@ -0,0 +1,775 @@ +/*------------------------------------------------------------------------- + * applybgwroker.c + * Support routines for applying xact by apply background worker + * + * Copyright (c) 2016-2022, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/replication/logical/applybgwroker.c + * + * This file contains routines that are intended to support setting up, using, + * and tearing down a ApplyBgworkerState. + * + * Refer to the comments in file header of logical/worker.c to see more + * information about apply background worker. + * + *------------------------------------------------------------------------- + */ + +#include "postgres.h" + +#include "libpq/pqformat.h" +#include "mb/pg_wchar.h" +#include "pgstat.h" +#include "postmaster/interrupt.h" +#include "replication/logicallauncher.h" +#include "replication/logicalworker.h" +#include "replication/origin.h" +#include "replication/walreceiver.h" +#include "replication/worker_internal.h" +#include "storage/ipc.h" +#include "storage/procarray.h" +#include "tcop/tcopprot.h" +#include "utils/inval.h" +#include "utils/memutils.h" +#include "utils/resowner.h" +#include "utils/syscache.h" + +#define PG_LOGICAL_APPLY_SHM_MAGIC 0x79fb2447 // TODO Consider change + +/* + * DSM keys for apply background worker. Unlike other parallel execution code, + * since we don't need to worry about DSM keys conflicting with plan_node_id we + * can use small integers. + */ +#define APPLY_BGWORKER_KEY_SHARED 1 +#define APPLY_BGWORKER_KEY_MQ 2 + +/* + * entry for a hash table we use to map from xid to our apply background worker + * state. + */ +typedef struct ApplyBgworkerEntry +{ + TransactionId xid; + ApplyBgworkerState *wstate; +} ApplyBgworkerEntry; + +/* Apply background workers hash table (initialized on first use) */ +static HTAB *ApplyWorkersHash = NULL; +static List *ApplyWorkersFreeList = NIL; +static List *ApplyWorkersList = NIL; + +/* + * Fields to record the share informations between main apply worker and apply + * background worker. + */ +volatile ApplyBgworkerShared *MyParallelState = NULL; + +List *subxactlist = NIL; + +/* apply background worker setup */ +static bool canstartapplybgworker(TransactionId xid, bool start); +static ApplyBgworkerState *apply_bgworker_setup(void); +static void apply_bgworker_setup_dsm(ApplyBgworkerState *wstate); + +/* + * Confirm if we can try to start a new apply background worker. + */ +static bool +canstartapplybgworker(TransactionId xid, bool start) +{ + if (!TransactionIdIsValid(xid)) + return false; + + /* + * We don't start new background worker if we are not in streaming apply + * mode. + */ + if (MySubscription->stream != SUBSTREAM_APPLY) + return false; + + /* + * We don't start new background worker if user has set skiplsn as it's + * possible that user want to skip the streaming transaction. For + * streaming transaction, we need to spill the transaction to disk so that + * we can get the last LSN of the transaction to judge whether to skip + * before starting to apply the change. + */ + if (start && !XLogRecPtrIsInvalid(MySubscription->skiplsn)) + return false; + + /* + * For streaming transactions that are being applied in apply background + * worker, we cannot decide whether to apply the change for a relation + * that is not in the READY state (see should_apply_changes_for_rel) as we + * won't know remote_final_lsn by that time. So, we don't start new apply + * background worker in this case. + */ + if (start && !AllTablesyncsReady()) + return false; + + return true; +} + +/* + * Look up worker inside ApplyWorkersHash for requested xid. + * + * If start flag is true, try to start a new worker if not found in hash table. + */ +ApplyBgworkerState * +apply_bgworker_find_or_start(TransactionId xid, bool start) +{ + bool found; + ApplyBgworkerState *wstate; + ApplyBgworkerEntry *entry = NULL; + + if (!canstartapplybgworker(xid, start)) + return NULL; + + /* First time through, initialize apply workers hashtable */ + if (ApplyWorkersHash == NULL) + { + HASHCTL ctl; + + MemSet(&ctl, 0, sizeof(ctl)); + ctl.keysize = sizeof(TransactionId); + ctl.entrysize = sizeof(ApplyBgworkerEntry); + ctl.hcxt = ApplyContext; + + ApplyWorkersHash = hash_create("logical apply workers hash", 8, &ctl, + HASH_ELEM | HASH_BLOBS | HASH_CONTEXT); + } + + /* + * Find entry for requested transaction. + */ + entry = hash_search(ApplyWorkersHash, &xid, start ? HASH_ENTER : HASH_FIND, + &found); + if (found) + { + entry->wstate->pstate->status = APPLY_BGWORKER_BUSY; + return entry->wstate; + } + else if (!start) + return NULL; + + /* + * Now, we try to get a apply background worker. If there is at least one + * worker in the idle list, then take one. Otherwise, we try to start a + * new apply background worker. + */ + if (list_length(ApplyWorkersFreeList) > 0) + { + wstate = (ApplyBgworkerState *) llast(ApplyWorkersFreeList); + ApplyWorkersFreeList = list_delete_last(ApplyWorkersFreeList); + } + else + { + wstate = apply_bgworker_setup(); + + if (wstate == NULL) + { + /* + * If the apply background worker cannot be launched, remove entry + * in hash table. + */ + hash_search(ApplyWorkersHash, &xid, HASH_REMOVE, &found); + return NULL; + } + } + + /* Fill up the hash entry */ + wstate->pstate->status = APPLY_BGWORKER_BUSY; + wstate->pstate->server_version = walrcv_server_version(LogRepWorkerWalRcvConn); + wstate->pstate->stream_xid = xid; + entry->wstate = wstate; + entry->xid = xid; + + return wstate; +} + +/* + * Add the worker to the free list and remove the entry from hash table. + */ +void +apply_bgworker_free(ApplyBgworkerState *wstate) +{ + MemoryContext oldctx; + TransactionId xid = wstate->pstate->stream_xid; + + Assert(wstate->pstate->status == APPLY_BGWORKER_FINISHED); + + oldctx = MemoryContextSwitchTo(ApplyContext); + + hash_search(ApplyWorkersHash, &xid, HASH_REMOVE, NULL); + + elog(DEBUG1, "adding finished apply worker #%u for xid %u to the idle list", + wstate->pstate->n, wstate->pstate->stream_xid); + + ApplyWorkersFreeList = lappend(ApplyWorkersFreeList, wstate); + + MemoryContextSwitchTo(oldctx); +} + +/* Apply Background Worker main loop */ +static void +LogicalApplyBgwLoop(shm_mq_handle *mqh, volatile ApplyBgworkerShared *pst) +{ + shm_mq_result shmq_res; + PGPROC *registrant; + ErrorContextCallback errcallback; + XLogRecPtr last_received = InvalidXLogRecPtr; + + registrant = BackendPidGetProc(MyBgworkerEntry->bgw_notify_pid); + SetLatch(®istrant->procLatch); + + /* + * Push apply error context callback. Fields will be filled during + * applying a change. + */ + errcallback.callback = apply_error_callback; + errcallback.previous = error_context_stack; + error_context_stack = &errcallback; + + for (;;) + { + void *data; + Size len; + int c; + StringInfoData s; + MemoryContext oldctx; + XLogRecPtr start_lsn; + XLogRecPtr end_lsn; + TimestampTz send_time; + + CHECK_FOR_INTERRUPTS(); + + /* Ensure we are reading the data into our memory context. */ + oldctx = MemoryContextSwitchTo(ApplyMessageContext); + + shmq_res = shm_mq_receive(mqh, &len, &data, false); + + if (shmq_res != SHM_MQ_SUCCESS) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("lost connection to the main apply worker"))); + + if (len == 0) + break; + + s.cursor = 0; + s.maxlen = -1; + s.data = (char *) data; + s.len = len; + + /* + * We use first byte of message for additional communication between + * main Logical replication worker and apply bgworkers, so if it + * differs from 'w', then process it first. + */ + c = pq_getmsgbyte(&s); + switch (c) + { + /* End message of streaming chunk */ + case LOGICAL_REP_MSG_STREAM_STOP: + elog(DEBUG1, "[Apply BGW #%u] ended processing streaming chunk," + "waiting on shm_mq_receive", pst->n); + + in_streamed_transaction = false; + pgstat_report_activity(STATE_IDLEINTRANSACTION, NULL); + continue; + case 'w': + break; + default: + elog(ERROR, "[Apply BGW #%u] unexpected message \"%c\"", + pst->n, c); + break; + } + + start_lsn = pq_getmsgint64(&s); + end_lsn = pq_getmsgint64(&s); + send_time = pq_getmsgint64(&s); + + if (last_received < start_lsn) + last_received = start_lsn; + + if (last_received < end_lsn) + last_received = end_lsn; + + /* + * TO IMPROVE: Do we need to display the apply background worker's + * information in pg_stat_replication ? + */ + UpdateWorkerStats(last_received, send_time, false); + + apply_dispatch(&s); + + if (ConfigReloadPending) + { + ConfigReloadPending = false; + ProcessConfigFile(PGC_SIGHUP); + } + + MemoryContextSwitchTo(oldctx); + MemoryContextReset(ApplyMessageContext); + } + + MemoryContextSwitchTo(TopMemoryContext); + MemoryContextReset(ApplyContext); + + /* Pop the error context stack */ + error_context_stack = errcallback.previous; + + elog(DEBUG1, "[Apply BGW #%u] exiting", pst->n); + + /* Signal main process that we are done. */ + SetLatch(®istrant->procLatch); +} + +/* + * Set the exit status so that the main apply worker can realize we have + * shutdown. + */ +static void +ApplyBgwShutdown(int code, Datum arg) +{ + SpinLockAcquire(&MyParallelState->mutex); + MyParallelState->status = APPLY_BGWORKER_EXIT; + SpinLockRelease(&MyParallelState->mutex); + + dsm_detach((dsm_segment *) DatumGetPointer(arg)); +} + +/* + * Apply Background Worker entry point + */ +void +ApplyBgworkerMain(Datum main_arg) +{ + volatile ApplyBgworkerShared *pst; + + dsm_handle handle; + dsm_segment *seg; + shm_toc *toc; + shm_mq *mq; + shm_mq_handle *mqh; + MemoryContext oldcontext; + RepOriginId originid; + int worker_slot = DatumGetInt32(main_arg); + char originname[NAMEDATALEN]; + + /* Attach to slot */ + logicalrep_worker_attach(worker_slot); + + MemoryContextSwitchTo(TopMemoryContext); + + /* Load the subscription into persistent memory context. */ + ApplyContext = AllocSetContextCreate(TopMemoryContext, + "ApplyContext", + ALLOCSET_DEFAULT_SIZES); + + /* + * Init the ApplyMessageContext which we clean up after each replication + * protocol message. + */ + ApplyMessageContext = AllocSetContextCreate(ApplyContext, + "ApplyMessageContext", + ALLOCSET_DEFAULT_SIZES); + + /* Setup signal handling */ + pqsignal(SIGHUP, SignalHandlerForConfigReload); + pqsignal(SIGTERM, die); + BackgroundWorkerUnblockSignals(); + + /* + * Connect to the dynamic shared memory segment. + * + * The backend that registered this worker passed us the ID of a shared + * memory segment to which we must attach for further instructions. In + * order to attach to dynamic shared memory, we need a resource owner. + * Once we've mapped the segment in our address space, attach to the table + * of contents so we can locate the various data structures we'll need to + * find within the segment. + */ + CurrentResourceOwner = ResourceOwnerCreate(NULL, "Logical apply worker"); + memcpy(&handle, MyBgworkerEntry->bgw_extra, sizeof(dsm_handle)); + seg = dsm_attach(handle); + if (seg == NULL) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("unable to map dynamic shared memory segment"))); + toc = shm_toc_attach(PG_LOGICAL_APPLY_SHM_MAGIC, dsm_segment_address(seg)); + if (toc == NULL) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("bad magic number in dynamic shared memory segment"))); + + before_shmem_exit(ApplyBgwShutdown, PointerGetDatum(seg)); + + /* Look up the parallel state. */ + pst = shm_toc_lookup(toc, APPLY_BGWORKER_KEY_SHARED, false); + MyParallelState = pst; + + /* + * Attach to the message queue. + */ + mq = shm_toc_lookup(toc, APPLY_BGWORKER_KEY_MQ, false); + shm_mq_set_receiver(mq, MyProc); + mqh = shm_mq_attach(mq, seg, NULL); + + /* Run as replica session replication role. */ + SetConfigOption("session_replication_role", "replica", + PGC_SUSET, PGC_S_OVERRIDE); + + /* Connect to our database. */ + BackgroundWorkerInitializeConnectionByOid(MyLogicalRepWorker->dbid, + MyLogicalRepWorker->userid, + 0); + + /* + * Set always-secure search path, so malicious users can't redirect user + * code (e.g. pg_index.indexprs). + */ + SetConfigOption("search_path", "", PGC_SUSET, PGC_S_OVERRIDE); + + /* + * Set the client encoding to the database encoding, since that is what + * the leader will expect. + */ + SetClientEncoding(GetDatabaseEncoding()); + + stream_xid = pst->stream_xid; + + StartTransactionCommand(); + oldcontext = MemoryContextSwitchTo(ApplyContext); + + MySubscription = GetSubscription(MyLogicalRepWorker->subid, true); + if (!MySubscription) + { + ereport(LOG, + (errmsg("logical replication apply worker for subscription %u will not " + "start because the subscription was removed during startup", + MyLogicalRepWorker->subid))); + proc_exit(0); + } + + MySubscriptionValid = true; + MemoryContextSwitchTo(oldcontext); + + /* Setup synchronous commit according to the user's wishes */ + SetConfigOption("synchronous_commit", MySubscription->synccommit, + PGC_BACKEND, PGC_S_OVERRIDE); + + /* Keep us informed about subscription changes. */ + CacheRegisterSyscacheCallback(SUBSCRIPTIONOID, + subscription_change_cb, + (Datum) 0); + + CommitTransactionCommand(); + + /* Setup replication origin tracking. */ + StartTransactionCommand(); + snprintf(originname, sizeof(originname), "pg_%u", MySubscription->oid); + originid = replorigin_by_name(originname, true); + if (!OidIsValid(originid)) + originid = replorigin_create(originname); + + /* + * The apply background worker doesn't need to monopolize this replication + * origin which was already acquired by its leader process. + */ + replorigin_session_setup(originid, false); + replorigin_session_origin = originid; + CommitTransactionCommand(); + + /* + * Allocate the origin name in long-lived context for error context + * message. + */ + apply_error_callback_arg.origin_name = MemoryContextStrdup(ApplyContext, + originname); + + /* + * Indicate that we're fully initialized and ready to begin the main part + * of the apply operation. + */ + apply_bgworker_set_status(APPLY_BGWORKER_ATTACHED); + + elog(DEBUG1, "[Apply BGW #%u] started", pst->n); + + LogicalApplyBgwLoop(mqh, pst); + + /* + * We're done. Explicitly detach the shared memory segment so that we + * don't get a resource leak warning at commit time. This will fire any + * on_dsm_detach callbacks we've registered, as well. Once that's done, + * we can go ahead and exit. + */ + dsm_detach(seg); + proc_exit(0); +} + +/* + * Set up a dynamic shared memory segment. + * + * We set up a control region that contains a ApplyBgworkerShared, + * plus one region per message queue. There are as many message queues as + * the number of workers. + */ +static void +apply_bgworker_setup_dsm(ApplyBgworkerState *wstate) +{ + shm_toc_estimator e; + Size segsize; + dsm_segment *seg; + shm_toc *toc; + ApplyBgworkerShared *pst; + shm_mq *mq; + int64 queue_size = 160000000; /* 16 MB for now */ + + /* + * Estimate how much shared memory we need. + * + * Because the TOC machinery may choose to insert padding of oddly-sized + * requests, we must estimate each chunk separately. + * + * We need one key to register the location of the header, and we need + * another key to track the location of the message queue. + */ + shm_toc_initialize_estimator(&e); + shm_toc_estimate_chunk(&e, sizeof(ApplyBgworkerShared)); + shm_toc_estimate_chunk(&e, (Size) queue_size); + + shm_toc_estimate_keys(&e, 1 + 1); + segsize = shm_toc_estimate(&e); + + /* Create the shared memory segment and establish a table of contents. */ + seg = dsm_create(shm_toc_estimate(&e), 0); + toc = shm_toc_create(PG_LOGICAL_APPLY_SHM_MAGIC, dsm_segment_address(seg), + segsize); + + /* Set up the header region. */ + pst = shm_toc_allocate(toc, sizeof(ApplyBgworkerShared)); + SpinLockInit(&pst->mutex); + pst->status = APPLY_BGWORKER_BUSY; + pst->server_version = walrcv_server_version(LogRepWorkerWalRcvConn); + pst->stream_xid = stream_xid; + pst->n = list_length(ApplyWorkersList) + 1; + + shm_toc_insert(toc, APPLY_BGWORKER_KEY_SHARED, pst); + + /* Set up one message queue per worker, plus one. */ + mq = shm_mq_create(shm_toc_allocate(toc, (Size) queue_size), + (Size) queue_size); + shm_toc_insert(toc, APPLY_BGWORKER_KEY_MQ, mq); + shm_mq_set_sender(mq, MyProc); + + /* Attach the queues. */ + wstate->mq_handle = shm_mq_attach(mq, seg, NULL); + + /* Return results to caller. */ + wstate->dsm_seg = seg; + wstate->pstate = pst; +} + +/* + * Start apply background worker process and allocate shared memory for it. + */ +static ApplyBgworkerState * +apply_bgworker_setup(void) +{ + MemoryContext oldcontext; + bool launched; + ApplyBgworkerState *wstate; + int napplyworkers; + + elog(DEBUG1, "setting up apply worker #%u", list_length(ApplyWorkersList) + 1); + + /* Check If there are free worker slot(s) */ + LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); + napplyworkers = logicalrep_apply_background_worker_count(MyLogicalRepWorker->subid); + LWLockRelease(LogicalRepWorkerLock); + if (napplyworkers >= max_apply_bgworkers_per_subscription) + return NULL; + + oldcontext = MemoryContextSwitchTo(ApplyContext); + + wstate = (ApplyBgworkerState *) palloc0(sizeof(ApplyBgworkerState)); + + /* Setup shared memory */ + apply_bgworker_setup_dsm(wstate); + + launched = logicalrep_worker_launch(MyLogicalRepWorker->dbid, + MySubscription->oid, + MySubscription->name, + MyLogicalRepWorker->userid, + InvalidOid, + dsm_segment_handle(wstate->dsm_seg)); + + if (launched) + { + /* Wait for worker to attach. */ + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_ATTACHED); + + ApplyWorkersList = lappend(ApplyWorkersList, wstate); + } + else + { + pfree(wstate); + wstate = NULL; + } + + MemoryContextSwitchTo(oldcontext); + + return wstate; +} + +/* + * Send the data to the specified apply background worker via shared-memory queue. + */ +void +apply_bgworker_send_data(ApplyBgworkerState *wstate, Size nbytes, const void *data) +{ + shm_mq_result result; + + result = shm_mq_send(wstate->mq_handle, nbytes, data, false, true); + + if (result != SHM_MQ_SUCCESS) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("could not send tuples to shared-memory queue"))); +} + +/* + * Wait until the status of apply background worker reaches the + * 'wait_for_status' + */ +void +apply_bgworker_wait_for(ApplyBgworkerState *wstate, + ApplyBgworkerStatus wait_for_status) +{ + for (;;) + { + char status; + + SpinLockAcquire(&wstate->pstate->mutex); + status = wstate->pstate->status; + SpinLockRelease(&wstate->pstate->mutex); + + /* Done if already in correct status. */ + if (status == wait_for_status) + break; + + /* If any workers (or the postmaster) have died, we have failed. */ + if (status == APPLY_BGWORKER_EXIT) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("background worker %u failed to apply transaction %u", + wstate->pstate->n, wstate->pstate->stream_xid))); + + /* Wait to be signalled. */ + WaitLatch(MyLatch, WL_LATCH_SET | WL_EXIT_ON_PM_DEATH, 0, + WAIT_EVENT_LOGICAL_APPLY_WORKER_STATE_CHANGE); + + /* Reset the latch so we don't spin. */ + ResetLatch(MyLatch); + + /* An interrupt may have occurred while we were waiting. */ + CHECK_FOR_INTERRUPTS(); + } +} + +/* + * Check the status of workers and report an error if any apply background + * worker has exited unexpectedly. + * + * Exit if any relation is not in the READY state and if any worker is handling + * the streaming transaction at the same time. Because for streaming + * transactions that is being applied in apply background worker, we cannot + * decide whether to apply the change for a relation that is not in the READY + * state (see should_apply_changes_for_rel) as we won't know remote_final_lsn + * by that time. + */ +void +apply_bgworker_check_status(void) +{ + ListCell *lc; + + if (am_apply_bgworker() || MySubscription->stream != SUBSTREAM_APPLY) + return; + + foreach(lc, ApplyWorkersList) + { + ApplyBgworkerState *wstate = (ApplyBgworkerState *) lfirst(lc); + + /* + * We don't lock here as in the worst case we will just detect the + * failure of worker a bit later. + */ + if (wstate->pstate->status == APPLY_BGWORKER_EXIT) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("background worker %u exited unexpectedly", + wstate->pstate->n))); + } + + if (list_length(ApplyWorkersFreeList) != list_length(ApplyWorkersList) && + !AllTablesyncsReady()) + { + ereport(LOG, + (errmsg("logical replication apply workers for subscription \"%s\" will restart", + MySubscription->name), + errdetail("Cannot handle streamed replication transaction by apply " + "bgworkers until all tables are synchronized"))); + + proc_exit(0); + } +} + +/* Set the apply background worker status */ +void +apply_bgworker_set_status(ApplyBgworkerStatus status) +{ + if (!am_apply_bgworker()) + return; + + elog(DEBUG1, "[Apply BGW #%u] set status to %d", MyParallelState->n, status); + + SpinLockAcquire(&MyParallelState->mutex); + MyParallelState->status = status; + SpinLockRelease(&MyParallelState->mutex); +} + +/* + * Define a savepoint for a subxact in apply background worker if needed. + * + * Inside apply background worker we can figure out that new subtransaction was + * started if new change arrived with different xid. In that case we can define + * named savepoint, so that we were able to commit/rollback it separately + * later. + * Special case is if the first change comes from subtransaction, then + * we check that current_xid differs from stream_xid. + */ +void +apply_bgworker_subxact_info_add(TransactionId current_xid) +{ + if (current_xid != stream_xid && + !list_member_int(subxactlist, (int) current_xid)) + { + MemoryContext oldctx; + char spname[MAXPGPATH]; + + snprintf(spname, MAXPGPATH, "savepoint_for_xid_%u", current_xid); + + elog(DEBUG1, "[Apply BGW #%u] defining savepoint %s", + MyParallelState->n, spname); + + DefineSavepoint(spname); + CommitTransactionCommand(); + + oldctx = MemoryContextSwitchTo(ApplyContext); + subxactlist = lappend_int(subxactlist, (int) current_xid); + MemoryContextSwitchTo(oldctx); + } +} diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c index aa2427ba73..00fdde0830 100644 --- a/src/backend/replication/logical/decode.c +++ b/src/backend/replication/logical/decode.c @@ -651,9 +651,10 @@ DecodeCommit(LogicalDecodingContext *ctx, XLogRecordBuffer *buf, { for (i = 0; i < parsed->nsubxacts; i++) { - ReorderBufferForget(ctx->reorder, parsed->subxacts[i], buf->origptr); + ReorderBufferForget(ctx->reorder, parsed->subxacts[i], buf->origptr, + commit_time); } - ReorderBufferForget(ctx->reorder, xid, buf->origptr); + ReorderBufferForget(ctx->reorder, xid, buf->origptr, commit_time); return; } @@ -821,10 +822,11 @@ DecodeAbort(LogicalDecodingContext *ctx, XLogRecordBuffer *buf, for (i = 0; i < parsed->nsubxacts; i++) { ReorderBufferAbort(ctx->reorder, parsed->subxacts[i], - buf->record->EndRecPtr); + buf->record->EndRecPtr, abort_time); } - ReorderBufferAbort(ctx->reorder, xid, buf->record->EndRecPtr); + ReorderBufferAbort(ctx->reorder, xid, buf->record->EndRecPtr, + abort_time); } /* update the decoding stats */ diff --git a/src/backend/replication/logical/launcher.c b/src/backend/replication/logical/launcher.c index 2bdab53e19..e46d9129b3 100644 --- a/src/backend/replication/logical/launcher.c +++ b/src/backend/replication/logical/launcher.c @@ -54,6 +54,7 @@ int max_logical_replication_workers = 4; int max_sync_workers_per_subscription = 2; +int max_apply_bgworkers_per_subscription = 2; LogicalRepWorker *MyLogicalRepWorker = NULL; @@ -73,6 +74,7 @@ static void logicalrep_launcher_onexit(int code, Datum arg); static void logicalrep_worker_onexit(int code, Datum arg); static void logicalrep_worker_detach(void); static void logicalrep_worker_cleanup(LogicalRepWorker *worker); +static void logicalrep_worker_stop_internal(LogicalRepWorker *worker); static bool on_commit_launcher_wakeup = false; @@ -223,6 +225,10 @@ logicalrep_worker_find(Oid subid, Oid relid, bool only_running) { LogicalRepWorker *w = &LogicalRepCtx->workers[i]; + /* We only need main apply worker or table sync worker here */ + if (w->subworker) + continue; + if (w->in_use && w->subid == subid && w->relid == relid && (!only_running || w->proc)) { @@ -259,11 +265,11 @@ logicalrep_workers_find(Oid subid, bool only_running) } /* - * Start new apply background worker, if possible. + * Start new background worker, if possible. */ -void +bool logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, Oid userid, - Oid relid) + Oid relid, dsm_handle subworker_dsm) { BackgroundWorker bgw; BackgroundWorkerHandle *bgw_handle; @@ -273,6 +279,10 @@ logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, Oid userid, LogicalRepWorker *worker = NULL; int nsyncworkers; TimestampTz now; + bool is_subworker = (subworker_dsm != DSM_HANDLE_INVALID); + + /* We don't support table sync in subworker */ + Assert(!(is_subworker && OidIsValid(relid))); ereport(DEBUG1, (errmsg_internal("starting logical replication worker for subscription \"%s\"", @@ -350,7 +360,7 @@ retry: if (OidIsValid(relid) && nsyncworkers >= max_sync_workers_per_subscription) { LWLockRelease(LogicalRepWorkerLock); - return; + return false; } /* @@ -364,7 +374,7 @@ retry: (errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED), errmsg("out of logical replication worker slots"), errhint("You might need to increase max_logical_replication_workers."))); - return; + return false; } /* Prepare the worker slot. */ @@ -379,6 +389,7 @@ retry: worker->relstate = SUBREL_STATE_UNKNOWN; worker->relstate_lsn = InvalidXLogRecPtr; worker->stream_fileset = NULL; + worker->subworker = is_subworker; worker->last_lsn = InvalidXLogRecPtr; TIMESTAMP_NOBEGIN(worker->last_send_time); TIMESTAMP_NOBEGIN(worker->last_recv_time); @@ -396,19 +407,30 @@ retry: BGWORKER_BACKEND_DATABASE_CONNECTION; bgw.bgw_start_time = BgWorkerStart_RecoveryFinished; snprintf(bgw.bgw_library_name, BGW_MAXLEN, "postgres"); - snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyWorkerMain"); + + if (!is_subworker) + snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyWorkerMain"); + else + snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyBgworkerMain"); + if (OidIsValid(relid)) snprintf(bgw.bgw_name, BGW_MAXLEN, "logical replication worker for subscription %u sync %u", subid, relid); - else + else if (!is_subworker) snprintf(bgw.bgw_name, BGW_MAXLEN, "logical replication worker for subscription %u", subid); + else + snprintf(bgw.bgw_name, BGW_MAXLEN, + "logical replication background apply worker for subscription %u ", subid); snprintf(bgw.bgw_type, BGW_MAXLEN, "logical replication worker"); bgw.bgw_restart_time = BGW_NEVER_RESTART; bgw.bgw_notify_pid = MyProcPid; bgw.bgw_main_arg = Int32GetDatum(slot); + if (is_subworker) + memcpy(bgw.bgw_extra, &subworker_dsm, sizeof(dsm_handle)); + if (!RegisterDynamicBackgroundWorker(&bgw, &bgw_handle)) { /* Failed to start worker, so clean up the worker slot. */ @@ -421,11 +443,13 @@ retry: (errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED), errmsg("out of background worker slots"), errhint("You might need to increase max_worker_processes."))); - return; + return false; } /* Now wait until it attaches. */ WaitForReplicationWorkerAttach(worker, generation, bgw_handle); + + return true; } /* @@ -436,7 +460,6 @@ void logicalrep_worker_stop(Oid subid, Oid relid) { LogicalRepWorker *worker; - uint16 generation; LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); @@ -449,6 +472,22 @@ logicalrep_worker_stop(Oid subid, Oid relid) return; } + logicalrep_worker_stop_internal(worker); + + LWLockRelease(LogicalRepWorkerLock); +} + +/* + * Workhorse for logicalrep_worker_stop() and logicalrep_worker_detach(). Stop + * the worker and wait for it to die. + */ +static void +logicalrep_worker_stop_internal(LogicalRepWorker *worker) +{ + uint16 generation; + + Assert(LWLockHeldByMe(LogicalRepWorkerLock)); + /* * Remember which generation was our worker so we can check if what we see * is still the same one. @@ -485,10 +524,7 @@ logicalrep_worker_stop(Oid subid, Oid relid) * different, meaning that a different worker has taken the slot. */ if (!worker->in_use || worker->generation != generation) - { - LWLockRelease(LogicalRepWorkerLock); return; - } /* Worker has assigned proc, so it has started. */ if (worker->proc) @@ -522,8 +558,6 @@ logicalrep_worker_stop(Oid subid, Oid relid) LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); } - - LWLockRelease(LogicalRepWorkerLock); } /* @@ -599,6 +633,29 @@ logicalrep_worker_attach(int slot) static void logicalrep_worker_detach(void) { + /* + * If we are the main apply worker, stop all the apply background workers + * we started before. + */ + if (!MyLogicalRepWorker->subworker) + { + List *workers; + ListCell *lc; + + LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); + + workers = logicalrep_workers_find(MyLogicalRepWorker->subid, true); + foreach(lc, workers) + { + LogicalRepWorker *w = (LogicalRepWorker *) lfirst(lc); + + if (w->subworker) + logicalrep_worker_stop_internal(w); + } + + LWLockRelease(LogicalRepWorkerLock); + } + /* Block concurrent access. */ LWLockAcquire(LogicalRepWorkerLock, LW_EXCLUSIVE); @@ -621,6 +678,7 @@ logicalrep_worker_cleanup(LogicalRepWorker *worker) worker->userid = InvalidOid; worker->subid = InvalidOid; worker->relid = InvalidOid; + worker->subworker = false; } /* @@ -679,6 +737,30 @@ logicalrep_sync_worker_count(Oid subid) return res; } +/* + * Count the number of registered (not necessarily running) apply background + * worker for a subscription. + */ +int +logicalrep_apply_background_worker_count(Oid subid) +{ + int i; + int res = 0; + + Assert(LWLockHeldByMe(LogicalRepWorkerLock)); + + /* Search for attached worker for a given subscription id. */ + for (i = 0; i < max_logical_replication_workers; i++) + { + LogicalRepWorker *w = &LogicalRepCtx->workers[i]; + + if (w->subid == subid && w->subworker) + res++; + } + + return res; +} + /* * ApplyLauncherShmemSize * Compute space needed for replication launcher shared memory @@ -868,7 +950,7 @@ ApplyLauncherMain(Datum main_arg) wait_time = wal_retrieve_retry_interval; logicalrep_worker_launch(sub->dbid, sub->oid, sub->name, - sub->owner, InvalidOid); + sub->owner, InvalidOid, DSM_HANDLE_INVALID); } } diff --git a/src/backend/replication/logical/origin.c b/src/backend/replication/logical/origin.c index 21937ab2d3..9273011b0a 100644 --- a/src/backend/replication/logical/origin.c +++ b/src/backend/replication/logical/origin.c @@ -1063,12 +1063,21 @@ ReplicationOriginExitCleanup(int code, Datum arg) * array doesn't have to be searched when calling * replorigin_session_advance(). * - * Obviously only one such cached origin can exist per process and the current + * Normally only one such cached origin can exist per process and the current * cached value can only be set again after the previous value is torn down * with replorigin_session_reset(). + * + * However, If must_acquire is false, we allow process to get the slot which is + * already acquired by other process. It's safe because 1) The only caller + * (apply background workers) will maintain the commit order by allowing only + * one process to commit at a time, so no two workers will be operating on the + * same origin at the same time (see comments in logical/worker.c). 2) Even + * though we try to advance the session's origin concurrently, it's safe to do + * so as we change/advance the session_origin LSNs under replicate_state + * LWLock. */ void -replorigin_session_setup(RepOriginId node) +replorigin_session_setup(RepOriginId node, bool must_acquire) { static bool registered_cleanup; int i; @@ -1110,7 +1119,7 @@ replorigin_session_setup(RepOriginId node) if (curstate->roident != node) continue; - else if (curstate->acquired_by != 0) + else if (curstate->acquired_by != 0 && must_acquire) { ereport(ERROR, (errcode(ERRCODE_OBJECT_IN_USE), @@ -1141,7 +1150,14 @@ replorigin_session_setup(RepOriginId node) Assert(session_replication_state->roident != InvalidRepOriginId); - session_replication_state->acquired_by = MyProcPid; + if (must_acquire) + session_replication_state->acquired_by = MyProcPid; + else if (session_replication_state->acquired_by == 0) + ereport(ERROR, + (errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED), + errmsg("could not find correct replication state slot for replication origin with OID %u for apply background worker", + node), + errhint("There is no replication state slot set by its main apply worker."))); LWLockRelease(ReplicationOriginLock); @@ -1321,7 +1337,7 @@ pg_replication_origin_session_setup(PG_FUNCTION_ARGS) name = text_to_cstring((text *) DatumGetPointer(PG_GETARG_DATUM(0))); origin = replorigin_by_name(name, false); - replorigin_session_setup(origin); + replorigin_session_setup(origin, true); replorigin_session_origin = origin; diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c index ff8513e2d2..a888038eb4 100644 --- a/src/backend/replication/logical/proto.c +++ b/src/backend/replication/logical/proto.c @@ -1166,28 +1166,47 @@ logicalrep_read_stream_commit(StringInfo in, LogicalRepCommitData *commit_data) */ void logicalrep_write_stream_abort(StringInfo out, TransactionId xid, - TransactionId subxid) + ReorderBufferTXN *txn, XLogRecPtr abort_lsn) { pq_sendbyte(out, LOGICAL_REP_MSG_STREAM_ABORT); - Assert(TransactionIdIsValid(xid) && TransactionIdIsValid(subxid)); + Assert(TransactionIdIsValid(xid) && TransactionIdIsValid(txn->xid)); /* transaction ID */ pq_sendint32(out, xid); - pq_sendint32(out, subxid); + pq_sendint32(out, txn->xid); + pq_sendint64(out, abort_lsn); + pq_sendint64(out, txn->xact_time.abort_time); } /* * Read STREAM ABORT from the output stream. */ void -logicalrep_read_stream_abort(StringInfo in, TransactionId *xid, - TransactionId *subxid) +logicalrep_read_stream_abort(StringInfo in, + LogicalRepStreamAbortData *abort_data, + bool include_abort_lsn) { - Assert(xid && subxid); + Assert(abort_data); + + abort_data->xid = pq_getmsgint(in, 4); + abort_data->subxid = pq_getmsgint(in, 4); - *xid = pq_getmsgint(in, 4); - *subxid = pq_getmsgint(in, 4); + /* + * If the version of the publisher is lower than the version of the + * subscriber, it may not support sending these two fields, so only take + * these fields when include_abort_lsn is true. + */ + if (include_abort_lsn) + { + abort_data->abort_lsn = pq_getmsgint64(in); + abort_data->abort_time = pq_getmsgint64(in); + } + else + { + abort_data->abort_lsn = InvalidXLogRecPtr; + abort_data->abort_time = 0; + } } /* diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c index 8da5f9089c..3d2bbdb38d 100644 --- a/src/backend/replication/logical/reorderbuffer.c +++ b/src/backend/replication/logical/reorderbuffer.c @@ -2826,7 +2826,8 @@ ReorderBufferFinishPrepared(ReorderBuffer *rb, TransactionId xid, * disk. */ void -ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn) +ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn, + TimestampTz abort_time) { ReorderBufferTXN *txn; @@ -2837,6 +2838,8 @@ ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn) if (txn == NULL) return; + txn->xact_time.abort_time = abort_time; + /* For streamed transactions notify the remote node about the abort. */ if (rbtxn_is_streamed(txn)) { @@ -2911,7 +2914,8 @@ ReorderBufferAbortOld(ReorderBuffer *rb, TransactionId oldestRunningXid) * to this xid might re-create the transaction incompletely. */ void -ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn) +ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn, + TimestampTz abort_time) { ReorderBufferTXN *txn; @@ -2922,6 +2926,8 @@ ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn) if (txn == NULL) return; + txn->xact_time.abort_time = abort_time; + /* For streamed transactions notify the remote node about the abort. */ if (rbtxn_is_streamed(txn)) rb->stream_abort(rb, txn, lsn); diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c index 670c6fcada..8ffba7e2e5 100644 --- a/src/backend/replication/logical/tablesync.c +++ b/src/backend/replication/logical/tablesync.c @@ -568,7 +568,8 @@ process_syncing_tables_for_apply(XLogRecPtr current_lsn) MySubscription->oid, MySubscription->name, MyLogicalRepWorker->userid, - rstate->relid); + rstate->relid, + DSM_HANDLE_INVALID); hentry->last_start_time = now; } } @@ -589,6 +590,9 @@ process_syncing_tables_for_apply(XLogRecPtr current_lsn) void process_syncing_tables(XLogRecPtr current_lsn) { + if (MyLogicalRepWorker->subworker) + return; + if (am_tablesync_worker()) process_syncing_tables_for_sync(current_lsn); else @@ -1273,7 +1277,7 @@ LogicalRepSyncTableStart(XLogRecPtr *origin_startpos) * time this tablesync was launched. */ originid = replorigin_by_name(originname, false); - replorigin_session_setup(originid); + replorigin_session_setup(originid, true); replorigin_session_origin = originid; *origin_startpos = replorigin_session_get_progress(false); @@ -1384,7 +1388,7 @@ LogicalRepSyncTableStart(XLogRecPtr *origin_startpos) true /* go backward */ , true /* WAL log */ ); UnlockRelationOid(ReplicationOriginRelationId, RowExclusiveLock); - replorigin_session_setup(originid); + replorigin_session_setup(originid, true); replorigin_session_origin = originid; } else diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c index 607f719fd6..18d6f33479 100644 --- a/src/backend/replication/logical/worker.c +++ b/src/backend/replication/logical/worker.c @@ -22,8 +22,23 @@ * STREAMED TRANSACTIONS * --------------------- * Streamed transactions (large transactions exceeding a memory limit on the - * upstream) are not applied immediately, but instead, the data is written - * to temporary files and then applied at once when the final commit arrives. + * upstream) are applied via one of two approaches. + * + * 1) Separate background workers + * + * Assign a new apply background worker (if available) as soon as the xact's + * first stream is received and the main apply worker will send changes to this + * new worker via shared memory. We keep this worker assigned till the + * transaction commit is received and also wait for the worker to finish at + * commit. This preserves commit ordering and avoids writing to and reading + * from file in most cases. We still need to spill if there is no worker + * available. + * + * 2) Write to temporary files and apply when the final commit arrives + * + * If no worker is available to handle streamed transaction, the data is + * written to temporary files and then applied at once when the final commit + * arrives. * * Unlike the regular (non-streamed) case, handling streamed transactions has * to handle aborts of both the toplevel transaction and subtransactions. This @@ -219,20 +234,7 @@ typedef struct ApplyExecutionData PartitionTupleRouting *proute; /* partition routing info */ } ApplyExecutionData; -/* Struct for saving and restoring apply errcontext information */ -typedef struct ApplyErrorCallbackArg -{ - LogicalRepMsgType command; /* 0 if invalid */ - LogicalRepRelMapEntry *rel; - - /* Remote node information */ - int remote_attnum; /* -1 if invalid */ - TransactionId remote_xid; - XLogRecPtr finish_lsn; - char *origin_name; -} ApplyErrorCallbackArg; - -static ApplyErrorCallbackArg apply_error_callback_arg = +ApplyErrorCallbackArg apply_error_callback_arg = { .command = 0, .rel = NULL, @@ -242,7 +244,7 @@ static ApplyErrorCallbackArg apply_error_callback_arg = .origin_name = NULL, }; -static MemoryContext ApplyMessageContext = NULL; +MemoryContext ApplyMessageContext = NULL; MemoryContext ApplyContext = NULL; /* per stream context for streaming transactions */ @@ -251,27 +253,38 @@ static MemoryContext LogicalStreamingContext = NULL; WalReceiverConn *LogRepWorkerWalRcvConn = NULL; Subscription *MySubscription = NULL; -static bool MySubscriptionValid = false; +bool MySubscriptionValid = false; bool in_remote_transaction = false; static XLogRecPtr remote_final_lsn = InvalidXLogRecPtr; /* fields valid only when processing streamed transaction */ -static bool in_streamed_transaction = false; +bool in_streamed_transaction = false; + +TransactionId stream_xid = InvalidTransactionId; +static ApplyBgworkerState *stream_apply_worker = NULL; + +/* check if we are applying the transaction in apply background worker */ +#define apply_bgworker_active() (in_streamed_transaction && stream_apply_worker != NULL) -static TransactionId stream_xid = InvalidTransactionId; +/* + * The number of changes during one streaming block (only for apply background + * workers) + */ +static uint32 nchanges = 0; /* * We enable skipping all data modification changes (INSERT, UPDATE, etc.) for * the subscription if the remote transaction's finish LSN matches the subskiplsn. * Once we start skipping changes, we don't stop it until we skip all changes of * the transaction even if pg_subscription is updated and MySubscription->skiplsn - * gets changed or reset during that. Also, in streaming transaction cases, we - * don't skip receiving and spooling the changes since we decide whether or not + * gets changed or reset during that. Also, in streaming transaction cases (streaming = on), + * we don't skip receiving and spooling the changes since we decide whether or not * to skip applying the changes when starting to apply changes. The subskiplsn is * cleared after successfully skipping the transaction or applying non-empty * transaction. The latter prevents the mistakenly specified subskiplsn from - * being left. + * being left. Note that we cannot skip the streaming transaction in apply mode, + * because we cannot get the finish LSN before applying the changes. */ static XLogRecPtr skip_xact_finish_lsn = InvalidXLogRecPtr; #define is_skipping_changes() (unlikely(!XLogRecPtrIsInvalid(skip_xact_finish_lsn))) @@ -324,9 +337,6 @@ static void maybe_reread_subscription(void); static void DisableSubscriptionAndExit(void); -/* prototype needed because of stream_commit */ -static void apply_dispatch(StringInfo s); - static void apply_handle_commit_internal(LogicalRepCommitData *commit_data); static void apply_handle_insert_internal(ApplyExecutionData *edata, ResultRelInfo *relinfo, @@ -359,7 +369,6 @@ static void stop_skipping_changes(void); static void clear_subscription_skip_lsn(XLogRecPtr finish_lsn); /* Functions for apply error callback */ -static void apply_error_callback(void *arg); static inline void set_apply_error_context_xact(TransactionId xid, XLogRecPtr lsn); static inline void reset_apply_error_context_info(void); @@ -426,41 +435,76 @@ end_replication_step(void) } /* - * Handle streamed transactions. + * Handle streamed transactions for both main apply worker and apply background + * worker. * - * If in streaming mode (receiving a block of streamed transaction), we - * simply redirect it to a file for the proper toplevel transaction. + * In streaming case (receiving a block of streamed transaction), for + * SUBSTREAM_ON mode, we simply redirect it to a file for the proper toplevel + * transaction, and for SUBSTREAM_APPLY mode, we send the changes to background + * apply worker (LOGICAL_REP_MSG_RELATION or LOGICAL_REP_MSG_TYPE changes will + * also be applied in main apply worker). * - * Returns true for streamed transactions, false otherwise (regular mode). + * For non-streamed transactions, returns false; + * For streamed transactions, returns true if in main apply worker (except we + * apply streamed transaction in "apply" mode and address + * LOGICAL_REP_MSG_RELATION or LOGICAL_REP_MSG_TYPE changes), false otherwise. */ static bool handle_streamed_transaction(LogicalRepMsgType action, StringInfo s) { - TransactionId xid; + TransactionId current_xid = InvalidTransactionId; /* not in streaming mode */ - if (!in_streamed_transaction) + if (!(in_streamed_transaction || am_apply_bgworker())) return false; - Assert(stream_fd != NULL); Assert(TransactionIdIsValid(stream_xid)); /* * We should have received XID of the subxact as the first part of the * message, so extract it. */ - xid = pq_getmsgint(s, 4); + current_xid = pq_getmsgint(s, 4); - if (!TransactionIdIsValid(xid)) + if (!TransactionIdIsValid(current_xid)) ereport(ERROR, (errcode(ERRCODE_PROTOCOL_VIOLATION), errmsg_internal("invalid transaction ID in streamed replication transaction"))); - /* Add the new subxact to the array (unless already there). */ - subxact_info_add(xid); + if (am_apply_bgworker()) + { + /* define a savepoint for a subxact if needed. */ + apply_bgworker_subxact_info_add(current_xid); - /* write the change to the current file */ - stream_write_change(action, s); + return false; + } + else if (apply_bgworker_active()) + { + /* + * If we decided to apply the changes of this transaction in an apply + * background worker, pass the data to the worker. + */ + apply_bgworker_send_data(stream_apply_worker, s->len, s->data); + nchanges += 1; + + /* + * XXX The publisher side doesn't always send relation/type update + * messages after the streaming transaction, so also update the + * relation/type in main apply worker here. See function + * cleanup_rel_sync_cache. + */ + if (action == LOGICAL_REP_MSG_RELATION || + action == LOGICAL_REP_MSG_TYPE) + return false; + } + else + { + /* Add the new subxact to the array (unless already there). */ + subxact_info_add(current_xid); + + /* write the change to the current file */ + stream_write_change(action, s); + } return true; } @@ -844,6 +888,9 @@ apply_handle_commit(StringInfo s) apply_handle_commit_internal(&commit_data); + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); + /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(commit_data.end_lsn); @@ -898,7 +945,9 @@ apply_handle_prepare_internal(LogicalRepPreparedTxnData *prepare_data) * BeginTransactionBlock is necessary to balance the EndTransactionBlock * called within the PrepareTransactionBlock below. */ - BeginTransactionBlock(); + if (!IsTransactionBlock()) + BeginTransactionBlock(); + CommitTransactionCommand(); /* Completes the preceding Begin command. */ /* @@ -950,6 +999,9 @@ apply_handle_prepare(StringInfo s) in_remote_transaction = false; + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); + /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(prepare_data.end_lsn); @@ -1064,10 +1116,6 @@ apply_handle_rollback_prepared(StringInfo s) /* * Handle STREAM PREPARE. - * - * Logic is in two parts: - * 1. Replay all the spooled operations - * 2. Mark the transaction as prepared */ static void apply_handle_stream_prepare(StringInfo s) @@ -1088,24 +1136,70 @@ apply_handle_stream_prepare(StringInfo s) logicalrep_read_stream_prepare(s, &prepare_data); set_apply_error_context_xact(prepare_data.xid, prepare_data.prepare_lsn); - elog(DEBUG1, "received prepare for streamed transaction %u", prepare_data.xid); + if (am_apply_bgworker()) + { + /* Mark the transaction as prepared. */ + apply_handle_prepare_internal(&prepare_data); - /* Replay all the spooled operations. */ - apply_spooled_messages(prepare_data.xid, prepare_data.prepare_lsn); + CommitTransactionCommand(); - /* Mark the transaction as prepared. */ - apply_handle_prepare_internal(&prepare_data); + pgstat_report_stat(false); - CommitTransactionCommand(); + list_free(subxactlist); + subxactlist = NIL; - pgstat_report_stat(false); + apply_bgworker_set_status(APPLY_BGWORKER_FINISHED); + } + else + { + ApplyBgworkerState *wstate = apply_bgworker_find_or_start(prepare_data.xid, false); - store_flush_position(prepare_data.end_lsn); + elog(DEBUG1, "received prepare for streamed transaction %u", + prepare_data.xid); + + /* + * This is the main apply worker. Check if we are processing this + * transaction in a apply background worker. + */ + if (wstate) + { + apply_bgworker_send_data(wstate, s->len, s->data); + + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_FINISHED); + apply_bgworker_free(wstate); + + pgstat_report_stat(false); + store_flush_position(prepare_data.end_lsn); + } + else + { + /* + * This is the main apply worker and the transaction has been + * serialized to file, replay all the spooled operations. + */ + apply_spooled_messages(prepare_data.xid, prepare_data.prepare_lsn); + + /* Mark the transaction as prepared. */ + apply_handle_prepare_internal(&prepare_data); + + CommitTransactionCommand(); + + pgstat_report_stat(false); + + store_flush_position(prepare_data.end_lsn); + + in_remote_transaction = false; + + /* unlink the files with serialized changes and subxact info. */ + stream_cleanup_files(MyLogicalRepWorker->subid, prepare_data.xid); + } + } in_remote_transaction = false; + stream_apply_worker = NULL; - /* unlink the files with serialized changes and subxact info. */ - stream_cleanup_files(MyLogicalRepWorker->subid, prepare_data.xid); + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(prepare_data.end_lsn); @@ -1155,15 +1249,6 @@ apply_handle_stream_start(StringInfo s) (errcode(ERRCODE_PROTOCOL_VIOLATION), errmsg_internal("duplicate STREAM START message"))); - /* - * Start a transaction on stream start, this transaction will be committed - * on the stream stop unless it is a tablesync worker in which case it - * will be committed after processing all the messages. We need the - * transaction for handling the buffile, used for serializing the - * streaming data and subxact info. - */ - begin_replication_step(); - /* notify handle methods we're processing a remote transaction */ in_streamed_transaction = true; @@ -1177,36 +1262,92 @@ apply_handle_stream_start(StringInfo s) set_apply_error_context_xact(stream_xid, InvalidXLogRecPtr); - /* - * Initialize the worker's stream_fileset if we haven't yet. This will be - * used for the entire duration of the worker so create it in a permanent - * context. We create this on the very first streaming message from any - * transaction and then use it for this and other streaming transactions. - * Now, we could create a fileset at the start of the worker as well but - * then we won't be sure that it will ever be used. - */ - if (MyLogicalRepWorker->stream_fileset == NULL) + if (am_apply_bgworker()) { - MemoryContext oldctx; - - oldctx = MemoryContextSwitchTo(ApplyContext); + /* + * Make sure the handle apply_dispatch methods are aware we're in a + * remote transaction. + */ + in_remote_transaction = true; - MyLogicalRepWorker->stream_fileset = palloc(sizeof(FileSet)); - FileSetInit(MyLogicalRepWorker->stream_fileset); + /* If we are in a apply background worker, begin the transaction */ + AcceptInvalidationMessages(); + maybe_reread_subscription(); - MemoryContextSwitchTo(oldctx); + StartTransactionCommand(); + BeginTransactionBlock(); + CommitTransactionCommand(); } + else + { + /* + * This is the main apply worker. Check if there is any free apply + * background worker we can use to process this transaction. + */ + stream_apply_worker = apply_bgworker_find_or_start(stream_xid, first_segment); - /* open the spool file for this transaction */ - stream_open_file(MyLogicalRepWorker->subid, stream_xid, first_segment); + if (stream_apply_worker) + { + /* + * If we have found a free worker or if we are already applying this + * transaction in an apply background worker, then we pass the data to + * that worker. + */ + if (first_segment) + apply_bgworker_send_data(stream_apply_worker, s->len, s->data); - /* if this is not the first segment, open existing subxact file */ - if (!first_segment) - subxact_info_read(MyLogicalRepWorker->subid, stream_xid); + nchanges = 0; + elog(DEBUG1, "starting streaming of xid %u", stream_xid); + } - pgstat_report_activity(STATE_RUNNING, NULL); + /* + * If no worker is available for the first stream start, we start to + * serialize all the changes of the transaction. + */ + else + { + /* + * Start a transaction on stream start, this transaction will be + * committed on the stream stop unless it is a tablesync worker in + * which case it will be committed after processing all the messages. + * We need the transaction for handling the buffile, used for + * serializing the streaming data and subxact info. + */ + begin_replication_step(); - end_replication_step(); + /* + * Initialize the worker's stream_fileset if we haven't yet. This will + * be used for the entire duration of the worker so create it in a + * permanent context. We create this on the very first streaming + * message from any transaction and then use it for this and other + * streaming transactions. Now, we could create a fileset at the start + * of the worker as well but then we won't be sure that it will ever + * be used. + */ + if (MyLogicalRepWorker->stream_fileset == NULL) + { + MemoryContext oldctx; + + oldctx = MemoryContextSwitchTo(ApplyContext); + + MyLogicalRepWorker->stream_fileset = palloc(sizeof(FileSet)); + FileSetInit(MyLogicalRepWorker->stream_fileset); + + MemoryContextSwitchTo(oldctx); + } + + /* open the spool file for this transaction */ + stream_open_file(MyLogicalRepWorker->subid, stream_xid, first_segment); + + /* if this is not the first segment, open existing subxact file */ + if (!first_segment) + subxact_info_read(MyLogicalRepWorker->subid, stream_xid); + + end_replication_step(); + } + } + + pgstat_report_activity(STATE_RUNNING, NULL); } /* @@ -1220,44 +1361,46 @@ apply_handle_stream_stop(StringInfo s) (errcode(ERRCODE_PROTOCOL_VIOLATION), errmsg_internal("STREAM STOP message without STREAM START"))); - /* - * Close the file with serialized changes, and serialize information about - * subxacts for the toplevel transaction. - */ - subxact_info_write(MyLogicalRepWorker->subid, stream_xid); - stream_close_file(); + if (apply_bgworker_active()) + { + char action = LOGICAL_REP_MSG_STREAM_STOP; - /* We must be in a valid transaction state */ - Assert(IsTransactionState()); + apply_bgworker_send_data(stream_apply_worker, 1, &action); - /* Commit the per-stream transaction */ - CommitTransactionCommand(); + elog(DEBUG1, "stopped streaming of xid %u, %u changes streamed", stream_xid, nchanges); + } + else + { + /* + * Close the file with serialized changes, and serialize information + * about subxacts for the toplevel transaction. + */ + subxact_info_write(MyLogicalRepWorker->subid, stream_xid); + stream_close_file(); - in_streamed_transaction = false; + /* We must be in a valid transaction state */ + Assert(IsTransactionState()); - /* Reset per-stream context */ - MemoryContextReset(LogicalStreamingContext); + /* Commit the per-stream transaction */ + CommitTransactionCommand(); + + /* Reset per-stream context */ + MemoryContextReset(LogicalStreamingContext); + } + + in_streamed_transaction = false; + stream_apply_worker = NULL; pgstat_report_activity(STATE_IDLE, NULL); reset_apply_error_context_info(); } /* - * Handle STREAM abort message. + * Handle STREAM ABORT message when the transaction was spilled to disk. */ static void -apply_handle_stream_abort(StringInfo s) +serialize_stream_abort(TransactionId xid, TransactionId subxid) { - TransactionId xid; - TransactionId subxid; - - if (in_streamed_transaction) - ereport(ERROR, - (errcode(ERRCODE_PROTOCOL_VIOLATION), - errmsg_internal("STREAM ABORT message without STREAM STOP"))); - - logicalrep_read_stream_abort(s, &xid, &subxid); - /* * If the two XIDs are the same, it's in fact abort of toplevel xact, so * just delete the files with serialized info. @@ -1339,8 +1482,136 @@ apply_handle_stream_abort(StringInfo s) end_replication_step(); CommitTransactionCommand(); } +} - reset_apply_error_context_info(); +/* + * Handle STREAM ABORT message. + */ +static void +apply_handle_stream_abort(StringInfo s) +{ + TransactionId xid; + TransactionId subxid; + LogicalRepStreamAbortData abort_data; + bool include_abort_lsn = false; + + if (in_streamed_transaction) + ereport(ERROR, + (errcode(ERRCODE_PROTOCOL_VIOLATION), + errmsg_internal("STREAM ABORT message without STREAM STOP"))); + + /* Check whether the publisher sends abort_lsn and abort_time. */ + if (am_apply_bgworker()) + include_abort_lsn = MyParallelState->server_version >= 150000; + + logicalrep_read_stream_abort(s, &abort_data, include_abort_lsn); + + xid = abort_data.xid; + subxid = abort_data.subxid; + + if (am_apply_bgworker()) + { + elog(DEBUG1, "[Apply BGW #%u] aborting current transaction xid=%u, subxid=%u", + MyParallelState->n, GetCurrentTransactionIdIfAny(), + GetCurrentSubTransactionId()); + + /* + * Update origin state so we can restart streaming from correct + * position in case of crash. + */ + if (include_abort_lsn) + { + replorigin_session_origin_lsn = abort_data.abort_lsn; + replorigin_session_origin_timestamp = abort_data.abort_time; + } + + /* + * If the two XIDs are the same, it's in fact abort of toplevel xact, + * so just free the subxactlist. + */ + if (subxid == xid) + { + set_apply_error_context_xact(subxid, InvalidXLogRecPtr); + + AbortCurrentTransaction(); + + EndTransactionBlock(false); + CommitTransactionCommand(); + + in_remote_transaction = false; + pgstat_report_activity(STATE_IDLE, NULL); + + list_free(subxactlist); + subxactlist = NIL; + + apply_bgworker_set_status(APPLY_BGWORKER_FINISHED); + } + else + { + /* + * OK, so it's a subxact. Rollback to the savepoint. + * + * We also need to read the subxactlist, determine the offset + * tracked for the subxact, and truncate the list. + */ + int i; + bool found = false; + char spname[MAXPGPATH]; + + set_apply_error_context_xact(subxid, InvalidXLogRecPtr); + snprintf(spname, MAXPGPATH, "savepoint_for_xid_%u", subxid); + + elog(DEBUG1, "[Apply BGW #%u] rolling back to savepoint %s", + MyParallelState->n, spname); + + for (i = list_length(subxactlist) - 1; i >= 0; i--) + { + xid = (TransactionId) list_nth_int(subxactlist, i); + if (xid == subxid) + { + found = true; + break; + } + } + + if (found) + { + RollbackToSavepoint(spname); + CommitTransactionCommand(); + subxactlist = list_truncate(subxactlist, i + 1); + } + + pgstat_report_activity(STATE_IDLEINTRANSACTION, NULL); + } + + reset_apply_error_context_info(); + } + else + { + ApplyBgworkerState *wstate = apply_bgworker_find_or_start(xid, false); + + /* + * This is the main apply worker. Check if we are processing this + * transaction in a apply background worker. + */ + if (wstate) + { + apply_bgworker_send_data(wstate, s->len, s->data); + + if (subxid == xid) + { + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_FINISHED); + apply_bgworker_free(wstate); + } + } + + /* + * We are in main apply worker and the transaction has been serialized + * to file. + */ + else + serialize_stream_abort(xid, subxid); + } } /* @@ -1462,40 +1733,6 @@ apply_spooled_messages(TransactionId xid, XLogRecPtr lsn) return; } -/* - * Handle STREAM COMMIT message. - */ -static void -apply_handle_stream_commit(StringInfo s) -{ - TransactionId xid; - LogicalRepCommitData commit_data; - - if (in_streamed_transaction) - ereport(ERROR, - (errcode(ERRCODE_PROTOCOL_VIOLATION), - errmsg_internal("STREAM COMMIT message without STREAM STOP"))); - - xid = logicalrep_read_stream_commit(s, &commit_data); - set_apply_error_context_xact(xid, commit_data.commit_lsn); - - elog(DEBUG1, "received commit for streamed transaction %u", xid); - - apply_spooled_messages(xid, commit_data.commit_lsn); - - apply_handle_commit_internal(&commit_data); - - /* unlink the files with serialized changes and subxact info */ - stream_cleanup_files(MyLogicalRepWorker->subid, xid); - - /* Process any tables that are being synchronized in parallel. */ - process_syncing_tables(commit_data.end_lsn); - - pgstat_report_activity(STATE_IDLE, NULL); - - reset_apply_error_context_info(); -} - /* * Helper function for apply_handle_commit and apply_handle_stream_commit. */ @@ -2448,11 +2685,105 @@ apply_handle_truncate(StringInfo s) end_replication_step(); } +/* + * Handle STREAM COMMIT message. + */ +static void +apply_handle_stream_commit(StringInfo s) +{ + LogicalRepCommitData commit_data; + TransactionId xid; + + if (in_streamed_transaction) + ereport(ERROR, + (errcode(ERRCODE_PROTOCOL_VIOLATION), + errmsg_internal("STREAM COMMIT message without STREAM STOP"))); + + xid = logicalrep_read_stream_commit(s, &commit_data); + set_apply_error_context_xact(xid, commit_data.commit_lsn); + + if (am_apply_bgworker()) + { + /* + * Update origin state so we can restart streaming from correct + * position in case of crash. + */ + replorigin_session_origin_lsn = commit_data.end_lsn; + replorigin_session_origin_timestamp = commit_data.committime; + + CommitTransactionCommand(); + EndTransactionBlock(false); + CommitTransactionCommand(); + + in_remote_transaction = false; + + pgstat_report_stat(false); + + list_free(subxactlist); + subxactlist = NIL; + + apply_bgworker_set_status(APPLY_BGWORKER_FINISHED); + } + else + { + /* + * This is the main apply worker. Check if we are processing this + * transaction in an apply background worker. + */ + ApplyBgworkerState *wstate = apply_bgworker_find_or_start(xid, false); + + elog(DEBUG1, "received commit for streamed transaction %u", xid); + + if (wstate) + { + /* Send commit message */ + apply_bgworker_send_data(wstate, s->len, s->data); + + /* Wait for apply background worker to finish */ + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_FINISHED); + + pgstat_report_stat(false); + store_flush_position(commit_data.end_lsn); + stop_skipping_changes(); + + apply_bgworker_free(wstate); + + /* + * The transaction is either non-empty or skipped, so we clear the + * subskiplsn. + */ + clear_subscription_skip_lsn(commit_data.commit_lsn); + } + else + { + /* + * This is the main apply worker and the transaction has been + * serialized to file, replay all the spooled operations. + */ + apply_spooled_messages(xid, commit_data.commit_lsn); + + apply_handle_commit_internal(&commit_data); + + /* unlink the files with serialized changes and subxact info */ + stream_cleanup_files(MyLogicalRepWorker->subid, xid); + } + } + + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); + + /* Process any tables that are being synchronized in parallel. */ + process_syncing_tables(commit_data.end_lsn); + + pgstat_report_activity(STATE_IDLE, NULL); + + reset_apply_error_context_info(); +} /* * Logical replication protocol message dispatcher. */ -static void +void apply_dispatch(StringInfo s) { LogicalRepMsgType action = pq_getmsgbyte(s); @@ -2621,6 +2952,10 @@ store_flush_position(XLogRecPtr remote_lsn) { FlushPosition *flushpos; + /* We only need to collect the LSN in main apply worker */ + if (am_apply_bgworker()) + return; + /* Need to do this in permanent context */ MemoryContextSwitchTo(ApplyContext); @@ -2635,7 +2970,7 @@ store_flush_position(XLogRecPtr remote_lsn) /* Update statistics of the worker. */ -static void +void UpdateWorkerStats(XLogRecPtr last_lsn, TimestampTz send_time, bool reply) { MyLogicalRepWorker->last_lsn = last_lsn; @@ -2797,6 +3132,9 @@ LogicalRepApplyLoop(XLogRecPtr last_received) AcceptInvalidationMessages(); maybe_reread_subscription(); + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); + /* Process any table synchronization changes. */ process_syncing_tables(last_received); } @@ -3098,7 +3436,7 @@ maybe_reread_subscription(void) /* * Callback from subscription syscache invalidation. */ -static void +void subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue) { MySubscriptionValid = false; @@ -3694,7 +4032,7 @@ ApplyWorkerMain(Datum main_arg) originid = replorigin_by_name(originname, true); if (!OidIsValid(originid)) originid = replorigin_create(originname); - replorigin_session_setup(originid); + replorigin_session_setup(originid, true); replorigin_session_origin = originid; origin_startpos = replorigin_session_get_progress(false); CommitTransactionCommand(); @@ -3741,7 +4079,7 @@ ApplyWorkerMain(Datum main_arg) options.proto.logical.publication_names = MySubscription->publications; options.proto.logical.binary = MySubscription->binary; - options.proto.logical.streaming = MySubscription->stream; + options.proto.logical.streaming = (MySubscription->stream != SUBSTREAM_OFF); options.proto.logical.twophase = false; if (!am_tablesync_worker()) @@ -3899,7 +4237,8 @@ clear_subscription_skip_lsn(XLogRecPtr finish_lsn) XLogRecPtr myskiplsn = MySubscription->skiplsn; bool started_tx = false; - if (likely(XLogRecPtrIsInvalid(myskiplsn))) + if (likely(XLogRecPtrIsInvalid(myskiplsn)) || + am_apply_bgworker()) return; if (!IsTransactionState()) @@ -3971,7 +4310,7 @@ clear_subscription_skip_lsn(XLogRecPtr finish_lsn) } /* Error callback to give more context info about the change being applied */ -static void +void apply_error_callback(void *arg) { ApplyErrorCallbackArg *errarg = &apply_error_callback_arg; diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c index 8deae57143..6ebfa24baf 100644 --- a/src/backend/replication/pgoutput/pgoutput.c +++ b/src/backend/replication/pgoutput/pgoutput.c @@ -1833,7 +1833,7 @@ pgoutput_stream_abort(struct LogicalDecodingContext *ctx, Assert(rbtxn_is_streamed(toptxn)); OutputPluginPrepareWrite(ctx, true); - logicalrep_write_stream_abort(ctx->out, toptxn->xid, txn->xid); + logicalrep_write_stream_abort(ctx->out, toptxn->xid, txn, abort_lsn); OutputPluginWrite(ctx, true); cleanup_rel_sync_cache(toptxn->xid, false); diff --git a/src/backend/utils/activity/wait_event.c b/src/backend/utils/activity/wait_event.c index 87c15b9c6f..ba781e6f08 100644 --- a/src/backend/utils/activity/wait_event.c +++ b/src/backend/utils/activity/wait_event.c @@ -388,6 +388,9 @@ pgstat_get_wait_ipc(WaitEventIPC w) case WAIT_EVENT_HASH_GROW_BUCKETS_REINSERT: event_name = "HashGrowBucketsReinsert"; break; + case WAIT_EVENT_LOGICAL_APPLY_WORKER_STATE_CHANGE: + event_name = "LogicalApplyWorkerStateChange"; + break; case WAIT_EVENT_LOGICAL_SYNC_DATA: event_name = "LogicalSyncData"; break; diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c index a7cc49898b..d05e1d2de0 100644 --- a/src/backend/utils/misc/guc.c +++ b/src/backend/utils/misc/guc.c @@ -3220,6 +3220,18 @@ static struct config_int ConfigureNamesInt[] = NULL, NULL, NULL }, + { + {"max_apply_bgworkers_per_subscription", + PGC_SIGHUP, + REPLICATION_SUBSCRIBERS, + gettext_noop("Maximum number of apply backgrand workers per subscription."), + NULL, + }, + &max_apply_bgworkers_per_subscription, + 2, 0, MAX_BACKENDS, + NULL, NULL, NULL + }, + { {"log_rotation_age", PGC_SIGHUP, LOGGING_WHERE, gettext_noop("Sets the amount of time to wait before forcing " diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample index 48ad80cf2e..292808dd83 100644 --- a/src/backend/utils/misc/postgresql.conf.sample +++ b/src/backend/utils/misc/postgresql.conf.sample @@ -360,6 +360,7 @@ #max_logical_replication_workers = 4 # taken from max_worker_processes # (change requires restart) #max_sync_workers_per_subscription = 2 # taken from max_logical_replication_workers +#max_apply_bgworkers_per_subscription = 2 # taken from max_logical_replication_workers #------------------------------------------------------------------------------ diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 7cc9c72e49..6f8b30abb0 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -4450,7 +4450,7 @@ getSubscriptions(Archive *fout) if (fout->remoteVersion >= 140000) appendPQExpBufferStr(query, " s.substream,\n"); else - appendPQExpBufferStr(query, " false AS substream,\n"); + appendPQExpBufferStr(query, " 'f' AS substream,\n"); if (fout->remoteVersion >= 150000) appendPQExpBufferStr(query, @@ -4580,8 +4580,10 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo) if (strcmp(subinfo->subbinary, "t") == 0) appendPQExpBufferStr(query, ", binary = true"); - if (strcmp(subinfo->substream, "f") != 0) + if (strcmp(subinfo->substream, "t") == 0) appendPQExpBufferStr(query, ", streaming = on"); + else if (strcmp(subinfo->substream, "a") == 0) + appendPQExpBufferStr(query, ", streaming = apply"); if (strcmp(subinfo->subtwophasestate, two_phase_disabled) != 0) appendPQExpBufferStr(query, ", two_phase = on"); diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h index d1260f590c..9b394a45fe 100644 --- a/src/include/catalog/pg_subscription.h +++ b/src/include/catalog/pg_subscription.h @@ -68,7 +68,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW bool subbinary; /* True if the subscription wants the * publisher to send data in binary */ - bool substream; /* Stream in-progress transactions. */ + char substream; /* Stream in-progress transactions. + * See SUBSTREAM_xxx constants. */ char subtwophasestate; /* Stream two-phase transactions */ @@ -109,7 +110,7 @@ typedef struct Subscription bool enabled; /* Indicates if the subscription is enabled */ bool binary; /* Indicates if the subscription wants data in * binary format */ - bool stream; /* Allow streaming in-progress transactions. */ + char stream; /* Allow streaming in-progress transactions. */ char twophasestate; /* Allow streaming two-phase transactions */ bool disableonerr; /* Indicates if the subscription should be * automatically disabled if a worker error @@ -120,6 +121,18 @@ typedef struct Subscription List *publications; /* List of publication names to subscribe to */ } Subscription; +/* Disallow streaming in-progress transactions */ +#define SUBSTREAM_OFF 'f' + +/* + * Streaming transactions are written to a temporary file and applied only + * after the transaction is committed on upstream. + */ +#define SUBSTREAM_ON 't' + +/* Streaming transactions are applied immediately via a background worker */ +#define SUBSTREAM_APPLY 'a' + extern Subscription *GetSubscription(Oid subid, bool missing_ok); extern void FreeSubscription(Subscription *sub); extern void DisableSubscription(Oid subid); diff --git a/src/include/replication/logicallauncher.h b/src/include/replication/logicallauncher.h index f1e2821e25..ac8ef94381 100644 --- a/src/include/replication/logicallauncher.h +++ b/src/include/replication/logicallauncher.h @@ -14,6 +14,7 @@ extern PGDLLIMPORT int max_logical_replication_workers; extern PGDLLIMPORT int max_sync_workers_per_subscription; +extern PGDLLIMPORT int max_apply_bgworkers_per_subscription; extern void ApplyLauncherRegister(void); extern void ApplyLauncherMain(Datum main_arg); diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h index a771ab8ff3..7bba77c9e7 100644 --- a/src/include/replication/logicalproto.h +++ b/src/include/replication/logicalproto.h @@ -175,6 +175,17 @@ typedef struct LogicalRepRollbackPreparedTxnData char gid[GIDSIZE]; } LogicalRepRollbackPreparedTxnData; +/* + * Transaction protocol information for stream abort. + */ +typedef struct LogicalRepStreamAbortData +{ + TransactionId xid; + TransactionId subxid; + XLogRecPtr abort_lsn; + TimestampTz abort_time; +} LogicalRepStreamAbortData; + extern void logicalrep_write_begin(StringInfo out, ReorderBufferTXN *txn); extern void logicalrep_read_begin(StringInfo in, LogicalRepBeginData *begin_data); @@ -246,9 +257,11 @@ extern void logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn extern TransactionId logicalrep_read_stream_commit(StringInfo out, LogicalRepCommitData *commit_data); extern void logicalrep_write_stream_abort(StringInfo out, TransactionId xid, - TransactionId subxid); -extern void logicalrep_read_stream_abort(StringInfo in, TransactionId *xid, - TransactionId *subxid); + ReorderBufferTXN *txn, + XLogRecPtr abort_lsn); +extern void logicalrep_read_stream_abort(StringInfo in, + LogicalRepStreamAbortData *abort_data, + bool include_abort_lsn); extern char *logicalrep_message_type(LogicalRepMsgType action); #endif /* LOGICAL_PROTO_H */ diff --git a/src/include/replication/logicalworker.h b/src/include/replication/logicalworker.h index cd1b6e8afc..6a1af7f13c 100644 --- a/src/include/replication/logicalworker.h +++ b/src/include/replication/logicalworker.h @@ -13,6 +13,7 @@ #define LOGICALWORKER_H extern void ApplyWorkerMain(Datum main_arg); +extern void ApplyBgworkerMain(Datum main_arg); extern bool IsLogicalWorker(void); diff --git a/src/include/replication/origin.h b/src/include/replication/origin.h index 2d1b5e5c28..c7389b40a7 100644 --- a/src/include/replication/origin.h +++ b/src/include/replication/origin.h @@ -53,7 +53,7 @@ extern XLogRecPtr replorigin_get_progress(RepOriginId node, bool flush); extern void replorigin_session_advance(XLogRecPtr remote_commit, XLogRecPtr local_commit); -extern void replorigin_session_setup(RepOriginId node); +extern void replorigin_session_setup(RepOriginId node, bool must_acquire); extern void replorigin_session_reset(void); extern XLogRecPtr replorigin_session_get_progress(bool flush); diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h index 4a01f877e5..aba1640f4d 100644 --- a/src/include/replication/reorderbuffer.h +++ b/src/include/replication/reorderbuffer.h @@ -301,6 +301,7 @@ typedef struct ReorderBufferTXN { TimestampTz commit_time; TimestampTz prepare_time; + TimestampTz abort_time; } xact_time; /* @@ -647,9 +648,11 @@ extern void ReorderBufferFinishPrepared(ReorderBuffer *rb, TransactionId xid, extern void ReorderBufferAssignChild(ReorderBuffer *, TransactionId, TransactionId, XLogRecPtr commit_lsn); extern void ReorderBufferCommitChild(ReorderBuffer *, TransactionId, TransactionId, XLogRecPtr commit_lsn, XLogRecPtr end_lsn); -extern void ReorderBufferAbort(ReorderBuffer *, TransactionId, XLogRecPtr lsn); +extern void ReorderBufferAbort(ReorderBuffer *, TransactionId, XLogRecPtr lsn, + TimestampTz abort_time); extern void ReorderBufferAbortOld(ReorderBuffer *, TransactionId xid); -extern void ReorderBufferForget(ReorderBuffer *, TransactionId, XLogRecPtr lsn); +extern void ReorderBufferForget(ReorderBuffer *, TransactionId, XLogRecPtr lsn, + TimestampTz abort_time); extern void ReorderBufferInvalidate(ReorderBuffer *, TransactionId, XLogRecPtr lsn); extern void ReorderBufferSetBaseSnapshot(ReorderBuffer *, TransactionId, XLogRecPtr lsn, struct SnapshotData *snap); diff --git a/src/include/replication/worker_internal.h b/src/include/replication/worker_internal.h index 901845abc2..0302836dee 100644 --- a/src/include/replication/worker_internal.h +++ b/src/include/replication/worker_internal.h @@ -17,8 +17,11 @@ #include "access/xlogdefs.h" #include "catalog/pg_subscription.h" #include "datatype/timestamp.h" +#include "replication/logicalrelation.h" #include "storage/fileset.h" #include "storage/lock.h" +#include "storage/shm_mq.h" +#include "storage/shm_toc.h" #include "storage/spin.h" @@ -60,6 +63,9 @@ typedef struct LogicalRepWorker */ FileSet *stream_fileset; + /* Indicates if this slot is used for an apply background worker. */ + bool subworker; + /* Stats. */ XLogRecPtr last_lsn; TimestampTz last_send_time; @@ -68,9 +74,69 @@ typedef struct LogicalRepWorker TimestampTz reply_time; } LogicalRepWorker; +/* Struct for saving and restoring apply errcontext information */ +typedef struct ApplyErrorCallbackArg +{ + LogicalRepMsgType command; /* 0 if invalid */ + LogicalRepRelMapEntry *rel; + + /* Remote node information */ + int remote_attnum; /* -1 if invalid */ + TransactionId remote_xid; + XLogRecPtr finish_lsn; + char *origin_name; +} ApplyErrorCallbackArg; + +/* + * Status for apply background worker. + */ +typedef enum ApplyBgworkerStatus +{ + APPLY_BGWORKER_ATTACHED = 0, + APPLY_BGWORKER_BUSY, + APPLY_BGWORKER_FINISHED, + APPLY_BGWORKER_EXIT +} ApplyBgworkerStatus; + +/* + * Shared information among apply workers. + */ +typedef struct ApplyBgworkerShared +{ + slock_t mutex; + + /* Status for apply background worker. */ + ApplyBgworkerStatus status; + + /* server version of publisher. */ + int server_version; + + TransactionId stream_xid; + uint32 n; /* id of apply background worker */ +} ApplyBgworkerShared; + +/* + * Struct for maintaining an apply background worker. + */ +typedef struct ApplyBgworkerState +{ + shm_mq_handle *mq_handle; + dsm_segment *dsm_seg; + ApplyBgworkerShared volatile *pstate; +} ApplyBgworkerState; + /* Main memory context for apply worker. Permanent during worker lifetime. */ extern PGDLLIMPORT MemoryContext ApplyContext; +extern PGDLLIMPORT MemoryContext ApplyMessageContext; + +extern PGDLLIMPORT ApplyErrorCallbackArg apply_error_callback_arg; + +extern PGDLLIMPORT bool MySubscriptionValid; + +extern PGDLLIMPORT volatile ApplyBgworkerShared *MyParallelState; +extern PGDLLIMPORT List *subxactlist; + /* libpqreceiver connection */ extern PGDLLIMPORT struct WalReceiverConn *LogRepWorkerWalRcvConn; @@ -79,18 +145,22 @@ extern PGDLLIMPORT Subscription *MySubscription; extern PGDLLIMPORT LogicalRepWorker *MyLogicalRepWorker; extern PGDLLIMPORT bool in_remote_transaction; +extern PGDLLIMPORT bool in_streamed_transaction; +extern PGDLLIMPORT TransactionId stream_xid; extern void logicalrep_worker_attach(int slot); extern LogicalRepWorker *logicalrep_worker_find(Oid subid, Oid relid, bool only_running); extern List *logicalrep_workers_find(Oid subid, bool only_running); -extern void logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, - Oid userid, Oid relid); +extern bool logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, + Oid userid, Oid relid, + dsm_handle subworker_dsm); extern void logicalrep_worker_stop(Oid subid, Oid relid); extern void logicalrep_worker_wakeup(Oid subid, Oid relid); extern void logicalrep_worker_wakeup_ptr(LogicalRepWorker *worker); extern int logicalrep_sync_worker_count(Oid subid); +extern int logicalrep_apply_background_worker_count(Oid subid); extern void ReplicationOriginNameForTablesync(Oid suboid, Oid relid, char *originname, int szorgname); @@ -103,10 +173,39 @@ extern void process_syncing_tables(XLogRecPtr current_lsn); extern void invalidate_syncing_table_states(Datum arg, int cacheid, uint32 hashvalue); +extern void UpdateWorkerStats(XLogRecPtr last_lsn, TimestampTz send_time, + bool reply); + +/* prototype needed because of stream_commit */ +extern void apply_dispatch(StringInfo s); + +/* Function for apply error callback */ +extern void apply_error_callback(void *arg); + +extern void subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue); + +/* apply background worker setup and interactions */ +extern ApplyBgworkerState *apply_bgworker_find_or_start(TransactionId xid, + bool start); +extern void apply_bgworker_wait_for(ApplyBgworkerState *wstate, + ApplyBgworkerStatus wait_for_status); +extern void apply_bgworker_send_data(ApplyBgworkerState *wstate, Size nbytes, + const void *data); +extern void apply_bgworker_free(ApplyBgworkerState *wstate); +extern void apply_bgworker_check_status(void); +extern void apply_bgworker_set_status(ApplyBgworkerStatus status); +extern void apply_bgworker_subxact_info_add(TransactionId current_xid); + static inline bool am_tablesync_worker(void) { return OidIsValid(MyLogicalRepWorker->relid); } +static inline bool +am_apply_bgworker(void) +{ + return MyLogicalRepWorker->subworker; +} + #endif /* WORKER_INTERNAL_H */ diff --git a/src/include/utils/wait_event.h b/src/include/utils/wait_event.h index b578e2ec75..c2d2a114d7 100644 --- a/src/include/utils/wait_event.h +++ b/src/include/utils/wait_event.h @@ -105,6 +105,7 @@ typedef enum WAIT_EVENT_HASH_GROW_BUCKETS_ALLOCATE, WAIT_EVENT_HASH_GROW_BUCKETS_ELECT, WAIT_EVENT_HASH_GROW_BUCKETS_REINSERT, + WAIT_EVENT_LOGICAL_APPLY_WORKER_STATE_CHANGE, WAIT_EVENT_LOGICAL_SYNC_DATA, WAIT_EVENT_LOGICAL_SYNC_STATE_CHANGE, WAIT_EVENT_MQ_INTERNAL, diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out index 7fcfad1591..f769835af0 100644 --- a/src/test/regress/expected/subscription.out +++ b/src/test/regress/expected/subscription.out @@ -197,7 +197,7 @@ ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); DROP SUBSCRIPTION regress_testsub; -- fail - streaming must be boolean CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = foo); -ERROR: streaming requires a Boolean value +ERROR: streaming requires a Boolean value or "apply" -- now it works CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = true); WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 4fb746930a..664d4b8b1a 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -116,6 +116,10 @@ Append AppendPath AppendRelInfo AppendState +ApplyBgworkerEntry +ApplyBgworkerShared +ApplyBgworkerState +ApplyBgworkerStatus ApplyErrorCallbackArg ApplyExecutionData ApplySubXactData @@ -1483,6 +1487,7 @@ LogicalRepRelId LogicalRepRelMapEntry LogicalRepRelation LogicalRepRollbackPreparedTxnData +LogicalRepStreamAbortData LogicalRepTupleData LogicalRepTyp LogicalRepWorker -- 2.23.0.windows.1 [application/octet-stream] v12-0002-Test-streaming-apply-option-in-tap-test.patch (68.9K, ../../OS3PR01MB62753CDFD0DF1FCBC978AFB39EAF9@OS3PR01MB6275.jpnprd01.prod.outlook.com/3-v12-0002-Test-streaming-apply-option-in-tap-test.patch) download | inline diff: From 87dded75fbcb751e0aaf20aa00bce4e217b921cb Mon Sep 17 00:00:00 2001 From: "shiy.fnst" <[email protected]> Date: Fri, 13 May 2022 14:50:30 +0800 Subject: [PATCH v12 2/5] Test streaming apply option in tap test Change all TAP tests using the SUBSCRIPTION "streaming" option, so they now test both 'on' and 'apply' values. --- src/test/subscription/t/015_stream.pl | 199 ++++--- src/test/subscription/t/016_stream_subxact.pl | 119 +++-- src/test/subscription/t/017_stream_ddl.pl | 188 ++++--- .../t/018_stream_subxact_abort.pl | 195 ++++--- .../t/019_stream_subxact_ddl_abort.pl | 110 +++- .../subscription/t/022_twophase_cascade.pl | 363 +++++++------ .../subscription/t/023_twophase_stream.pl | 498 ++++++++++-------- 7 files changed, 1035 insertions(+), 637 deletions(-) diff --git a/src/test/subscription/t/015_stream.pl b/src/test/subscription/t/015_stream.pl index 6561b189de..b3d84b1c7c 100644 --- a/src/test/subscription/t/015_stream.pl +++ b/src/test/subscription/t/015_stream.pl @@ -8,6 +8,116 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" option +# so the same code can be run both for the streaming=on and streaming=apply +# cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_apply) = @_; + + # Interleave a pair of transactions, each exceeding the 64kB limit. + my $in = ''; + my $out = ''; + + my $timer = IPC::Run::timeout($PostgreSQL::Test::Utils::timeout_default); + + my $h = $node_publisher->background_psql('postgres', \$in, \$out, $timer, + on_error_stop => 0); + + # Check that a background worker starts if "streaming" option is + # specified as "apply". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_apply) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + $in .= q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + }; + $h->pump_nb; + + if ($is_apply) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001, 9999) s(i); + DELETE FROM test_tab WHERE a > 5000; + COMMIT; + }); + + $in .= q{ + COMMIT; + \q + }; + $h->finish; # errors make the next test fail, so ignore them here + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), + 'check extra columns contain local defaults'); + + # Test the streaming in binary mode + $node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET (binary = on)"); + + # Insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001, 10000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + COMMIT; + }); + + $node_publisher->wait_for_catchup($appname); + + $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(6667|6667|6667), + 'check extra columns contain local defaults'); + + # Change the local values of the extra columns on the subscriber, + # update publisher, and check that subscriber retains the expected + # values. This is to ensure that non-streaming transactions behave + # properly after a streaming transaction. + $node_subscriber->safe_psql('postgres', + "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'" + ); + $node_publisher->safe_psql('postgres', + "UPDATE test_tab SET b = md5(a::text)"); + + $node_publisher->wait_for_catchup($appname); + + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab" + ); + is($result, qq(6667|6667|6667), + 'check extra columns contain locally changed data'); + + # Cleanup the test data + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE (a > 2)"); + $node_publisher->wait_for_catchup($appname); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -37,6 +147,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -54,82 +168,25 @@ my $result = "SELECT count(*), count(c), count(d = 999) FROM test_tab"); is($result, qq(2|2|2), 'check initial data was copied to subscriber'); -# Interleave a pair of transactions, each exceeding the 64kB limit. -my $in = ''; -my $out = ''; - -my $timer = IPC::Run::timeout($PostgreSQL::Test::Utils::timeout_default); - -my $h = $node_publisher->background_psql('postgres', \$in, \$out, $timer, - on_error_stop => 0); - -$in .= q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -}; -$h->pump_nb; - -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001, 9999) s(i); -DELETE FROM test_tab WHERE a > 5000; -COMMIT; -}); - -$in .= q{ -COMMIT; -\q -}; -$h->finish; # errors make the next test fail, so ignore them here - -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), 'check extra columns contain local defaults'); - -# Test the streaming in binary mode -$node_subscriber->safe_psql('postgres', - "ALTER SUBSCRIPTION tap_sub SET (binary = on)"); - -# Insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001, 10000) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -COMMIT; -}); - -$node_publisher->wait_for_catchup($appname); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(6667|6667|6667), 'check extra columns contain local defaults'); +################################### +# Test using streaming mode 'apply' +################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" +); -# Change the local values of the extra columns on the subscriber, -# update publisher, and check that subscriber retains the expected -# values. This is to ensure that non-streaming transactions behave -# properly after a streaming transaction. $node_subscriber->safe_psql('postgres', - "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'" -); -$node_publisher->safe_psql('postgres', - "UPDATE test_tab SET b = md5(a::text)"); + "ALTER SUBSCRIPTION tap_sub SET(streaming = apply, binary = off)"); -$node_publisher->wait_for_catchup($appname); +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab" -); -is($result, qq(6667|6667|6667), - 'check extra columns contain locally changed data'); +test_streaming($node_publisher, $node_subscriber, $appname, 1); $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/016_stream_subxact.pl b/src/test/subscription/t/016_stream_subxact.pl index f27f1694f2..b5b6dc379f 100644 --- a/src/test/subscription/t/016_stream_subxact.pl +++ b/src/test/subscription/t/016_stream_subxact.pl @@ -8,6 +8,72 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" option +# so the same code can be run both for the streaming=on and streaming=apply +# cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_apply) = @_; + + # Check that a background worker starts if "streaming" option is + # specified as "apply". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_apply) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + # Insert, update and delete enough rows to exceed 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series( 3, 500) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501, 1000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001, 1500) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + SAVEPOINT s3; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501, 2000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + SAVEPOINT s4; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001, 2500) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + COMMIT; + }); + + if ($is_apply) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(1667|1667|1667), + 'check data was copied to subscriber in streaming mode and extra columns contain local defaults' + ); + + # Cleanup the test data + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE (a > 2)"); + $node_publisher->wait_for_catchup($appname); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -37,6 +103,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -54,41 +124,26 @@ my $result = "SELECT count(*), count(c), count(d = 999) FROM test_tab"); is($result, qq(2|2|2), 'check initial data was copied to subscriber'); -# Insert, update and delete enough rows to exceed 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series( 3, 500) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501, 1000) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001, 1500) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -SAVEPOINT s3; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501, 2000) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -SAVEPOINT s4; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001, 2500) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -COMMIT; -}); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(1667|1667|1667), - 'check data was copied to subscriber in streaming mode and extra columns contain local defaults' +################################### +# Test using streaming mode 'apply' +################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" ); +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = apply)"); + +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; + +test_streaming($node_publisher, $node_subscriber, $appname, 1); + $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/017_stream_ddl.pl b/src/test/subscription/t/017_stream_ddl.pl index 0bce63b716..7e5dc18cd2 100644 --- a/src/test/subscription/t/017_stream_ddl.pl +++ b/src/test/subscription/t/017_stream_ddl.pl @@ -8,6 +8,111 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" option +# so the same code can be run both for the streaming=on and streaming=apply +# cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_apply) = @_; + + # a small (non-streamed) transaction with DDL and DML + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab VALUES (3, md5(3::text)); + ALTER TABLE test_tab ADD COLUMN c INT; + SAVEPOINT s1; + INSERT INTO test_tab VALUES (4, md5(4::text), -4); + COMMIT; + }); + + # Check that a background worker starts if "streaming" option is + # specified as "apply". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_apply) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + # large (streamed) transaction with DDL and DML + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(5, 1000) s(i); + ALTER TABLE test_tab ADD COLUMN d INT; + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001, 2000) s(i); + COMMIT; + }); + + if ($is_apply) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + # a small (non-streamed) transaction with DDL and DML + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab VALUES (2001, md5(2001::text), -2001, 2*2001); + ALTER TABLE test_tab ADD COLUMN e INT; + SAVEPOINT s1; + INSERT INTO test_tab VALUES (2002, md5(2002::text), -2002, 2*2002, -3*2002); + COMMIT; + }); + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d), count(e) FROM test_tab"); + is($result, qq(2002|1999|1002|1), + 'check data was copied to subscriber in streaming mode and extra columns contain local defaults' + ); + + # A large (streamed) transaction with DDL and DML. One of the DDL is performed + # after DML to ensure that we invalidate the schema sent for test_tab so that + # the next transaction has to send the schema again. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(2003,5000) s(i); + ALTER TABLE test_tab ADD COLUMN f INT; + COMMIT; + }); + + # A small transaction that won't get streamed. This is just to ensure that we + # send the schema again to reflect the last column added in the previous test. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i, 4*i FROM generate_series(5001,5005) s(i); + COMMIT; + }); + + $node_publisher->wait_for_catchup($appname); + + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d), count(e), count(f) FROM test_tab" + ); + is($result, qq(5005|5002|4005|3004|5), + 'check data was copied to subscriber for both streaming and non-streaming transactions' + ); + + # Cleanup the test data + $node_publisher->safe_psql( + 'postgres', q{ + DELETE FROM test_tab WHERE (a > 2); + ALTER TABLE test_tab DROP COLUMN c, DROP COLUMN d, DROP COLUMN e, DROP COLUMN f; + }); + $node_publisher->wait_for_catchup($appname); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -37,6 +142,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -54,76 +163,25 @@ my $result = "SELECT count(*), count(c), count(d = 999) FROM test_tab"); is($result, qq(2|0|0), 'check initial data was copied to subscriber'); -# a small (non-streamed) transaction with DDL and DML -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab VALUES (3, md5(3::text)); -ALTER TABLE test_tab ADD COLUMN c INT; -SAVEPOINT s1; -INSERT INTO test_tab VALUES (4, md5(4::text), -4); -COMMIT; -}); - -# large (streamed) transaction with DDL and DML -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(5, 1000) s(i); -ALTER TABLE test_tab ADD COLUMN d INT; -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001, 2000) s(i); -COMMIT; -}); - -# a small (non-streamed) transaction with DDL and DML -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab VALUES (2001, md5(2001::text), -2001, 2*2001); -ALTER TABLE test_tab ADD COLUMN e INT; -SAVEPOINT s1; -INSERT INTO test_tab VALUES (2002, md5(2002::text), -2002, 2*2002, -3*2002); -COMMIT; -}); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d), count(e) FROM test_tab"); -is($result, qq(2002|1999|1002|1), - 'check data was copied to subscriber in streaming mode and extra columns contain local defaults' +################################### +# Test using streaming mode 'apply' +################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" ); -# A large (streamed) transaction with DDL and DML. One of the DDL is performed -# after DML to ensure that we invalidate the schema sent for test_tab so that -# the next transaction has to send the schema again. -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(2003,5000) s(i); -ALTER TABLE test_tab ADD COLUMN f INT; -COMMIT; -}); - -# A small transaction that won't get streamed. This is just to ensure that we -# send the schema again to reflect the last column added in the previous test. -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i, 4*i FROM generate_series(5001,5005) s(i); -COMMIT; -}); +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = apply)"); -$node_publisher->wait_for_catchup($appname); +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d), count(e), count(f) FROM test_tab"); -is($result, qq(5005|5002|4005|3004|5), - 'check data was copied to subscriber for both streaming and non-streaming transactions' -); +test_streaming($node_publisher, $node_subscriber, $appname, 1); $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/018_stream_subxact_abort.pl b/src/test/subscription/t/018_stream_subxact_abort.pl index 7155442e76..71f8c1dd05 100644 --- a/src/test/subscription/t/018_stream_subxact_abort.pl +++ b/src/test/subscription/t/018_stream_subxact_abort.pl @@ -8,6 +8,113 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" option +# so the same code can be run both for the streaming=on and streaming=apply +# cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_apply) = @_; + + # Check that a background worker starts if "streaming" option is + # specified as "apply". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_apply) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + # large (streamed) transaction with DDL, DML and ROLLBACKs + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i); + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501,1000) s(i); + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001,1500) s(i); + SAVEPOINT s3; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501,2000) s(i); + ROLLBACK TO s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001,2500) s(i); + ROLLBACK TO s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2501,3000) s(i); + SAVEPOINT s4; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3001,3500) s(i); + SAVEPOINT s5; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3501,4000) s(i); + COMMIT; + }); + + if ($is_apply) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c) FROM test_tab"); + is($result, qq(2000|0), + 'check rollback to savepoint was reflected on subscriber and extra columns contain local defaults' + ); + + # large (streamed) transaction with subscriber receiving out of order + # subtransaction ROLLBACKs + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(4001,4500) s(i); + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001,5500) s(i); + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(6001,6500) s(i); + SAVEPOINT s3; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(7001,7500) s(i); + RELEASE s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(8001,8500) s(i); + ROLLBACK TO s1; + COMMIT; + }); + + $node_publisher->wait_for_catchup($appname); + + $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c) FROM test_tab"); + is($result, qq(2500|0), + 'check rollback to savepoint was reflected on subscriber'); + + # large (streamed) transaction with subscriber receiving rollback + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(8501,9000) s(i); + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(9001,9500) s(i); + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(9501,10000) s(i); + ROLLBACK; + }); + + $node_publisher->wait_for_catchup($appname); + + $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c) FROM test_tab"); + is($result, qq(2500|0), 'check rollback was reflected on subscriber'); + + # Cleanup the test data + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE (a > 2)"); + $node_publisher->wait_for_catchup($appname); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -36,6 +143,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -53,81 +164,25 @@ my $result = "SELECT count(*), count(c) FROM test_tab"); is($result, qq(2|0), 'check initial data was copied to subscriber'); -# large (streamed) transaction with DDL, DML and ROLLBACKs -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i); -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501,1000) s(i); -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001,1500) s(i); -SAVEPOINT s3; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501,2000) s(i); -ROLLBACK TO s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001,2500) s(i); -ROLLBACK TO s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2501,3000) s(i); -SAVEPOINT s4; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3001,3500) s(i); -SAVEPOINT s5; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3501,4000) s(i); -COMMIT; -}); - -$node_publisher->wait_for_catchup($appname); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c) FROM test_tab"); -is($result, qq(2000|0), - 'check rollback to savepoint was reflected on subscriber and extra columns contain local defaults' +################################### +# Test using streaming mode 'apply' +################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" ); -# large (streamed) transaction with subscriber receiving out of order -# subtransaction ROLLBACKs -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(4001,4500) s(i); -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001,5500) s(i); -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(6001,6500) s(i); -SAVEPOINT s3; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(7001,7500) s(i); -RELEASE s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(8001,8500) s(i); -ROLLBACK TO s1; -COMMIT; -}); - -$node_publisher->wait_for_catchup($appname); +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = apply)"); -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c) FROM test_tab"); -is($result, qq(2500|0), - 'check rollback to savepoint was reflected on subscriber'); - -# large (streamed) transaction with subscriber receiving rollback -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(8501,9000) s(i); -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(9001,9500) s(i); -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(9501,10000) s(i); -ROLLBACK; -}); +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c) FROM test_tab"); -is($result, qq(2500|0), 'check rollback was reflected on subscriber'); +test_streaming($node_publisher, $node_subscriber, $appname, 1); $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/019_stream_subxact_ddl_abort.pl b/src/test/subscription/t/019_stream_subxact_ddl_abort.pl index dbd0fca4d1..d0cd869430 100644 --- a/src/test/subscription/t/019_stream_subxact_ddl_abort.pl +++ b/src/test/subscription/t/019_stream_subxact_ddl_abort.pl @@ -9,6 +9,69 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" option +# so the same code can be run both for the streaming=on and streaming=apply +# cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_apply) = @_; + + # Check that a background worker starts if "streaming" option is + # specified as "apply". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_apply) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + # large (streamed) transaction with DDL, DML and ROLLBACKs + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i); + ALTER TABLE test_tab ADD COLUMN c INT; + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(501,1000) s(i); + ALTER TABLE test_tab ADD COLUMN d INT; + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001,1500) s(i); + ALTER TABLE test_tab ADD COLUMN e INT; + SAVEPOINT s3; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(1501,2000) s(i); + ALTER TABLE test_tab DROP COLUMN c; + ROLLBACK TO s1; + INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(501,1000) s(i); + COMMIT; + }); + + if ($is_apply) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c) FROM test_tab"); + is($result, qq(1000|500), + 'check rollback to savepoint was reflected on subscriber and extra columns contain local defaults' + ); + + # Cleanup the test data + $node_publisher->safe_psql( + 'postgres', q{ + DELETE FROM test_tab WHERE (a > 2); + ALTER TABLE test_tab DROP COLUMN c; + }); + $node_publisher->wait_for_catchup($appname); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -37,6 +100,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -54,35 +121,26 @@ my $result = "SELECT count(*), count(c) FROM test_tab"); is($result, qq(2|0), 'check initial data was copied to subscriber'); -# large (streamed) transaction with DDL, DML and ROLLBACKs -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i); -ALTER TABLE test_tab ADD COLUMN c INT; -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(501,1000) s(i); -ALTER TABLE test_tab ADD COLUMN d INT; -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001,1500) s(i); -ALTER TABLE test_tab ADD COLUMN e INT; -SAVEPOINT s3; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(1501,2000) s(i); -ALTER TABLE test_tab DROP COLUMN c; -ROLLBACK TO s1; -INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(501,1000) s(i); -COMMIT; -}); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c) FROM test_tab"); -is($result, qq(1000|500), - 'check rollback to savepoint was reflected on subscriber and extra columns contain local defaults' +################################### +# Test using streaming mode 'apply' +################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" ); +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = apply)"); + +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; + +test_streaming($node_publisher, $node_subscriber, $appname, 1); + $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/022_twophase_cascade.pl b/src/test/subscription/t/022_twophase_cascade.pl index 7a797f37ba..b52af74e35 100644 --- a/src/test/subscription/t/022_twophase_cascade.pl +++ b/src/test/subscription/t/022_twophase_cascade.pl @@ -11,6 +11,208 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" option +# so the same code can be run both for the streaming=on and streaming=apply +# cases. +sub test_streaming +{ + my ($node_A, $node_B, $node_C, $appname_B, $appname_C, $streaming_mode) = + @_; + + my $oldpid_B = $node_A->safe_psql( + 'postgres', " + SELECT pid FROM pg_stat_replication + WHERE application_name = '$appname_B' AND state = 'streaming';"); + my $oldpid_C = $node_B->safe_psql( + 'postgres', " + SELECT pid FROM pg_stat_replication + WHERE application_name = '$appname_C' AND state = 'streaming';"); + + # Setup logical replication streaming mode + + $node_B->safe_psql( + 'postgres', " + ALTER SUBSCRIPTION tap_sub_B + SET (streaming = $streaming_mode);"); + $node_C->safe_psql( + 'postgres', " + ALTER SUBSCRIPTION tap_sub_C + SET (streaming = $streaming_mode)"); + + # Wait for subscribers to finish initialization + + $node_A->poll_query_until( + 'postgres', " + SELECT pid != $oldpid_B FROM pg_stat_replication + WHERE application_name = '$appname_B' AND state = 'streaming';" + ) or die "Timed out while waiting for apply to restart"; + $node_B->poll_query_until( + 'postgres', " + SELECT pid != $oldpid_C FROM pg_stat_replication + WHERE application_name = '$appname_C' AND state = 'streaming';" + ) or die "Timed out while waiting for apply to restart"; + + ############################### + # Test 2PC PREPARE / COMMIT PREPARED. + # 1. Data is streamed as a 2PC transaction. + # 2. Then do commit prepared. + # + # Expect all data is replicated on subscriber(s) after the commit. + ############################### + + # Check that a background worker starts if "streaming" option is + # specified as "apply". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($streaming_mode eq 'apply') + { + $node_B->append_conf('postgresql.conf', "log_min_messages = debug1"); + $node_B->reload; + + $node_C->append_conf('postgresql.conf', "log_min_messages = debug1"); + $node_C->reload; + } + + # Insert, update and delete enough rows to exceed the 64kB limit. + # Then 2PC PREPARE + $node_A->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + if ($streaming_mode eq 'apply') + { + $node_B->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_B->append_conf('postgresql.conf', "log_min_messages = warning"); + $node_B->reload; + + $node_C->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_C->append_conf('postgresql.conf', "log_min_messages = warning"); + $node_C->reload; + } + + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); + + # check the transaction state is prepared on subscriber(s) + my $result = + $node_B->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber B'); + $result = + $node_C->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber C'); + + # 2PC COMMIT + $node_A->safe_psql('postgres', "COMMIT PREPARED 'test_prepared_tab';"); + + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); + + # check that transaction was committed on subscriber(s) + $result = $node_B->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), + 'Rows inserted by 2PC have committed on subscriber B, and extra columns have local defaults' + ); + $result = $node_C->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), + 'Rows inserted by 2PC have committed on subscriber C, and extra columns have local defaults' + ); + + # check the transaction state is ended on subscriber(s) + $result = + $node_B->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is committed on subscriber B'); + $result = + $node_C->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is committed on subscriber C'); + + ############################### + # Test 2PC PREPARE with a nested ROLLBACK TO SAVEPOINT. + # 0. Cleanup from previous test leaving only 2 rows. + # 1. Insert one more row. + # 2. Record a SAVEPOINT. + # 3. Data is streamed using 2PC. + # 4. Do rollback to SAVEPOINT prior to the streamed inserts. + # 5. Then COMMIT PREPARED. + # + # Expect data after the SAVEPOINT is aborted leaving only 3 rows (= 2 original + 1 from step 1). + ############################### + + # First, delete the data except for 2 rows (delete will be replicated) + $node_A->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); + + # 2PC PREPARE with a nested ROLLBACK TO SAVEPOINT + $node_A->safe_psql( + 'postgres', " + BEGIN; + INSERT INTO test_tab VALUES (9999, 'foobar'); + SAVEPOINT sp_inner; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + ROLLBACK TO SAVEPOINT sp_inner; + PREPARE TRANSACTION 'outer'; + "); + + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); + + # check the transaction state prepared on subscriber(s) + $result = + $node_B->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber B'); + $result = + $node_C->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber C'); + + # 2PC COMMIT + $node_A->safe_psql('postgres', "COMMIT PREPARED 'outer';"); + + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); + + # check the transaction state is ended on subscriber + $result = + $node_B->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is ended on subscriber B'); + $result = + $node_C->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is ended on subscriber C'); + + # check inserts are visible at subscriber(s). + # All the streamed data (prior to the SAVEPOINT) should be rolled back. + # (9999, 'foobar') should be committed. + $result = $node_B->safe_psql('postgres', + "SELECT count(*) FROM test_tab where b = 'foobar';"); + is($result, qq(1), 'Rows committed are present on subscriber B'); + $result = + $node_B->safe_psql('postgres', "SELECT count(*) FROM test_tab;"); + is($result, qq(3), 'Rows committed are present on subscriber B'); + $result = $node_C->safe_psql('postgres', + "SELECT count(*) FROM test_tab where b = 'foobar';"); + is($result, qq(1), 'Rows committed are present on subscriber C'); + $result = + $node_C->safe_psql('postgres', "SELECT count(*) FROM test_tab;"); + is($result, qq(3), 'Rows committed are present on subscriber C'); + + # Cleanup the test data + $node_A->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); +} + ############################### # Setup a cascade of pub/sub nodes. # node_A -> node_B -> node_C @@ -260,160 +462,15 @@ is($result, qq(21), 'Rows committed are present on subscriber C'); # 2PC + STREAMING TESTS # --------------------- -my $oldpid_B = $node_A->safe_psql( - 'postgres', " - SELECT pid FROM pg_stat_replication - WHERE application_name = '$appname_B' AND state = 'streaming';"); -my $oldpid_C = $node_B->safe_psql( - 'postgres', " - SELECT pid FROM pg_stat_replication - WHERE application_name = '$appname_C' AND state = 'streaming';"); - -# Setup logical replication (streaming = on) - -$node_B->safe_psql( - 'postgres', " - ALTER SUBSCRIPTION tap_sub_B - SET (streaming = on);"); -$node_C->safe_psql( - 'postgres', " - ALTER SUBSCRIPTION tap_sub_C - SET (streaming = on)"); - -# Wait for subscribers to finish initialization - -$node_A->poll_query_until( - 'postgres', " - SELECT pid != $oldpid_B FROM pg_stat_replication - WHERE application_name = '$appname_B' AND state = 'streaming';" -) or die "Timed out while waiting for apply to restart"; -$node_B->poll_query_until( - 'postgres', " - SELECT pid != $oldpid_C FROM pg_stat_replication - WHERE application_name = '$appname_C' AND state = 'streaming';" -) or die "Timed out while waiting for apply to restart"; - -############################### -# Test 2PC PREPARE / COMMIT PREPARED. -# 1. Data is streamed as a 2PC transaction. -# 2. Then do commit prepared. -# -# Expect all data is replicated on subscriber(s) after the commit. -############################### - -# Insert, update and delete enough rows to exceed the 64kB limit. -# Then 2PC PREPARE -$node_A->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_A->wait_for_catchup($appname_B); -$node_B->wait_for_catchup($appname_C); +################################ +# Test using streaming mode 'on' +################################ +test_streaming($node_A, $node_B, $node_C, $appname_B, $appname_C, 'on'); -# check the transaction state is prepared on subscriber(s) -$result = - $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber B'); -$result = - $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber C'); - -# 2PC COMMIT -$node_A->safe_psql('postgres', "COMMIT PREPARED 'test_prepared_tab';"); - -$node_A->wait_for_catchup($appname_B); -$node_B->wait_for_catchup($appname_C); - -# check that transaction was committed on subscriber(s) -$result = $node_B->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), - 'Rows inserted by 2PC have committed on subscriber B, and extra columns have local defaults' -); -$result = $node_C->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), - 'Rows inserted by 2PC have committed on subscriber C, and extra columns have local defaults' -); - -# check the transaction state is ended on subscriber(s) -$result = - $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is committed on subscriber B'); -$result = - $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is committed on subscriber C'); - -############################### -# Test 2PC PREPARE with a nested ROLLBACK TO SAVEPOINT. -# 0. Cleanup from previous test leaving only 2 rows. -# 1. Insert one more row. -# 2. Record a SAVEPOINT. -# 3. Data is streamed using 2PC. -# 4. Do rollback to SAVEPOINT prior to the streamed inserts. -# 5. Then COMMIT PREPARED. -# -# Expect data after the SAVEPOINT is aborted leaving only 3 rows (= 2 original + 1 from step 1). -############################### - -# First, delete the data except for 2 rows (delete will be replicated) -$node_A->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); - -# 2PC PREPARE with a nested ROLLBACK TO SAVEPOINT -$node_A->safe_psql( - 'postgres', " - BEGIN; - INSERT INTO test_tab VALUES (9999, 'foobar'); - SAVEPOINT sp_inner; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - ROLLBACK TO SAVEPOINT sp_inner; - PREPARE TRANSACTION 'outer'; - "); - -$node_A->wait_for_catchup($appname_B); -$node_B->wait_for_catchup($appname_C); - -# check the transaction state prepared on subscriber(s) -$result = - $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber B'); -$result = - $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber C'); - -# 2PC COMMIT -$node_A->safe_psql('postgres', "COMMIT PREPARED 'outer';"); - -$node_A->wait_for_catchup($appname_B); -$node_B->wait_for_catchup($appname_C); - -# check the transaction state is ended on subscriber -$result = - $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is ended on subscriber B'); -$result = - $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is ended on subscriber C'); - -# check inserts are visible at subscriber(s). -# All the streamed data (prior to the SAVEPOINT) should be rolled back. -# (9999, 'foobar') should be committed. -$result = $node_B->safe_psql('postgres', - "SELECT count(*) FROM test_tab where b = 'foobar';"); -is($result, qq(1), 'Rows committed are present on subscriber B'); -$result = $node_B->safe_psql('postgres', "SELECT count(*) FROM test_tab;"); -is($result, qq(3), 'Rows committed are present on subscriber B'); -$result = $node_C->safe_psql('postgres', - "SELECT count(*) FROM test_tab where b = 'foobar';"); -is($result, qq(1), 'Rows committed are present on subscriber C'); -$result = $node_C->safe_psql('postgres', "SELECT count(*) FROM test_tab;"); -is($result, qq(3), 'Rows committed are present on subscriber C'); +################################### +# Test using streaming mode 'apply' +################################### +test_streaming($node_A, $node_B, $node_C, $appname_B, $appname_C, 'apply'); ############################### # check all the cleanup diff --git a/src/test/subscription/t/023_twophase_stream.pl b/src/test/subscription/t/023_twophase_stream.pl index d8475d25a4..f55c5c95e7 100644 --- a/src/test/subscription/t/023_twophase_stream.pl +++ b/src/test/subscription/t/023_twophase_stream.pl @@ -8,6 +8,266 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" option +# so the same code can be run both for the streaming=on and streaming=apply +# cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_apply) = @_; + + ############################### + # Test 2PC PREPARE / COMMIT PREPARED. + # 1. Data is streamed as a 2PC transaction. + # 2. Then do commit prepared. + # + # Expect all data is replicated on subscriber side after the commit. + ############################### + + # Check that a background worker starts if "streaming" option is + # specified as "apply". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_apply) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + # check that 2PC gets replicated to subscriber + # Insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + if ($is_apply) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is in prepared state on subscriber + my $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber'); + + # 2PC transaction gets committed + $node_publisher->safe_psql('postgres', + "COMMIT PREPARED 'test_prepared_tab';"); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is committed on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), + 'Rows inserted by 2PC have committed on subscriber, and extra columns contain local defaults' + ); + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is committed on subscriber'); + + ############################### + # Test 2PC PREPARE / ROLLBACK PREPARED. + # 1. Table is deleted back to 2 rows which are replicated on subscriber. + # 2. Data is streamed using 2PC. + # 3. Do rollback prepared. + # + # Expect data rolls back leaving only the original 2 rows. + ############################### + + # First, delete the data except for 2 rows (will be replicated) + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE a > 2;"); + + # Then insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is in prepared state on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber'); + + # 2PC transaction gets aborted + $node_publisher->safe_psql('postgres', + "ROLLBACK PREPARED 'test_prepared_tab';"); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is aborted on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(2|2|2), + 'Rows inserted by 2PC are rolled back, leaving only the original 2 rows' + ); + + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is aborted on subscriber'); + + ############################### + # Check that 2PC COMMIT PREPARED is decoded properly on crash restart. + # 1. insert, update and delete enough rows to exceed the 64kB limit. + # 2. Then server crashes before the 2PC transaction is committed. + # 3. After servers are restarted the pending transaction is committed. + # + # Expect all data is replicated on subscriber side after the commit. + # Note: both publisher and subscriber do crash/restart. + ############################### + + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + $node_subscriber->stop('immediate'); + $node_publisher->stop('immediate'); + + $node_publisher->start; + $node_subscriber->start; + + # commit post the restart + $node_publisher->safe_psql('postgres', + "COMMIT PREPARED 'test_prepared_tab';"); + $node_publisher->wait_for_catchup($appname); + + # check inserts are visible + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), + 'Rows inserted by 2PC have committed on subscriber, and extra columns contain local defaults' + ); + + ############################### + # Do INSERT after the PREPARE but before ROLLBACK PREPARED. + # 1. Table is deleted back to 2 rows which are replicated on subscriber. + # 2. Data is streamed using 2PC. + # 3. A single row INSERT is done which is after the PREPARE. + # 4. Then do a ROLLBACK PREPARED. + # + # Expect the 2PC data rolls back leaving only 3 rows on the subscriber + # (the original 2 + inserted 1). + ############################### + + # First, delete the data except for 2 rows (will be replicated) + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE a > 2;"); + + # Then insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is in prepared state on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber'); + + # Insert a different record (now we are outside of the 2PC transaction) + # Note: the 2PC transaction still holds row locks so make sure this insert is for a separate primary key + $node_publisher->safe_psql('postgres', + "INSERT INTO test_tab VALUES (99999, 'foobar')"); + + # 2PC transaction gets aborted + $node_publisher->safe_psql('postgres', + "ROLLBACK PREPARED 'test_prepared_tab';"); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is aborted on subscriber, + # but the extra INSERT outside of the 2PC still was replicated + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3|3|3), + 'check the outside insert was copied to subscriber'); + + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is aborted on subscriber'); + + ############################### + # Do INSERT after the PREPARE but before COMMIT PREPARED. + # 1. Table is deleted back to 2 rows which are replicated on subscriber. + # 2. Data is streamed using 2PC. + # 3. A single row INSERT is done which is after the PREPARE. + # 4. Then do a COMMIT PREPARED. + # + # Expect 2PC data + the extra row are on the subscriber + # (the 3334 + inserted 1 = 3335). + ############################### + + # First, delete the data except for 2 rows (will be replicated) + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE a > 2;"); + + # Then insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is in prepared state on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber'); + + # Insert a different record (now we are outside of the 2PC transaction) + # Note: the 2PC transaction still holds row locks so make sure this insert is for a separare primary key + $node_publisher->safe_psql('postgres', + "INSERT INTO test_tab VALUES (99999, 'foobar')"); + + # 2PC transaction gets committed + $node_publisher->safe_psql('postgres', + "COMMIT PREPARED 'test_prepared_tab';"); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is committed on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3335|3335|3335), + 'Rows inserted by 2PC (as well as outside insert) have committed on subscriber, and extra columns contain local defaults' + ); + + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is committed on subscriber'); + + # Cleanup the test data + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE a > 2;"); + $node_publisher->wait_for_catchup($appname); +} + ############################### # Setup ############################### @@ -48,6 +308,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql( 'postgres', " CREATE SUBSCRIPTION tap_sub @@ -70,236 +334,30 @@ my $twophase_query = $node_subscriber->poll_query_until('postgres', $twophase_query) or die "Timed out while waiting for subscriber to enable twophase"; -############################### # Check initial data was copied to subscriber -############################### my $result = $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab"); is($result, qq(2|2|2), 'check initial data was copied to subscriber'); -############################### -# Test 2PC PREPARE / COMMIT PREPARED. -# 1. Data is streamed as a 2PC transaction. -# 2. Then do commit prepared. -# -# Expect all data is replicated on subscriber side after the commit. -############################### - -# check that 2PC gets replicated to subscriber -# Insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is in prepared state on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber'); - -# 2PC transaction gets committed -$node_publisher->safe_psql('postgres', - "COMMIT PREPARED 'test_prepared_tab';"); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is committed on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), - 'Rows inserted by 2PC have committed on subscriber, and extra columns contain local defaults' -); -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is committed on subscriber'); - -############################### -# Test 2PC PREPARE / ROLLBACK PREPARED. -# 1. Table is deleted back to 2 rows which are replicated on subscriber. -# 2. Data is streamed using 2PC. -# 3. Do rollback prepared. -# -# Expect data rolls back leaving only the original 2 rows. -############################### - -# First, delete the data except for 2 rows (will be replicated) -$node_publisher->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); - -# Then insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is in prepared state on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber'); - -# 2PC transaction gets aborted -$node_publisher->safe_psql('postgres', - "ROLLBACK PREPARED 'test_prepared_tab';"); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is aborted on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(2|2|2), - 'Rows inserted by 2PC are rolled back, leaving only the original 2 rows'); - -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is aborted on subscriber'); - -############################### -# Check that 2PC COMMIT PREPARED is decoded properly on crash restart. -# 1. insert, update and delete enough rows to exceed the 64kB limit. -# 2. Then server crashes before the 2PC transaction is committed. -# 3. After servers are restarted the pending transaction is committed. -# -# Expect all data is replicated on subscriber side after the commit. -# Note: both publisher and subscriber do crash/restart. -############################### - -$node_publisher->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_subscriber->stop('immediate'); -$node_publisher->stop('immediate'); - -$node_publisher->start; -$node_subscriber->start; - -# commit post the restart -$node_publisher->safe_psql('postgres', - "COMMIT PREPARED 'test_prepared_tab';"); -$node_publisher->wait_for_catchup($appname); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -# check inserts are visible -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), - 'Rows inserted by 2PC have committed on subscriber, and extra columns contain local defaults' +################################### +# Test using streaming mode 'apply' +################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" ); -############################### -# Do INSERT after the PREPARE but before ROLLBACK PREPARED. -# 1. Table is deleted back to 2 rows which are replicated on subscriber. -# 2. Data is streamed using 2PC. -# 3. A single row INSERT is done which is after the PREPARE. -# 4. Then do a ROLLBACK PREPARED. -# -# Expect the 2PC data rolls back leaving only 3 rows on the subscriber -# (the original 2 + inserted 1). -############################### - -# First, delete the data except for 2 rows (will be replicated) -$node_publisher->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); - -# Then insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is in prepared state on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber'); - -# Insert a different record (now we are outside of the 2PC transaction) -# Note: the 2PC transaction still holds row locks so make sure this insert is for a separate primary key -$node_publisher->safe_psql('postgres', - "INSERT INTO test_tab VALUES (99999, 'foobar')"); - -# 2PC transaction gets aborted -$node_publisher->safe_psql('postgres', - "ROLLBACK PREPARED 'test_prepared_tab';"); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is aborted on subscriber, -# but the extra INSERT outside of the 2PC still was replicated -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3|3|3), 'check the outside insert was copied to subscriber'); - -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is aborted on subscriber'); - -############################### -# Do INSERT after the PREPARE but before COMMIT PREPARED. -# 1. Table is deleted back to 2 rows which are replicated on subscriber. -# 2. Data is streamed using 2PC. -# 3. A single row INSERT is done which is after the PREPARE. -# 4. Then do a COMMIT PREPARED. -# -# Expect 2PC data + the extra row are on the subscriber -# (the 3334 + inserted 1 = 3335). -############################### - -# First, delete the data except for 2 rows (will be replicated) -$node_publisher->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); - -# Then insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is in prepared state on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber'); - -# Insert a different record (now we are outside of the 2PC transaction) -# Note: the 2PC transaction still holds row locks so make sure this insert is for a separare primary key -$node_publisher->safe_psql('postgres', - "INSERT INTO test_tab VALUES (99999, 'foobar')"); - -# 2PC transaction gets committed -$node_publisher->safe_psql('postgres', - "COMMIT PREPARED 'test_prepared_tab';"); - -$node_publisher->wait_for_catchup($appname); +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = apply)"); -# check that transaction is committed on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3335|3335|3335), - 'Rows inserted by 2PC (as well as outside insert) have committed on subscriber, and extra columns contain local defaults' -); +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is committed on subscriber'); +test_streaming($node_publisher, $node_subscriber, $appname, 1); ############################### # check all the cleanup -- 2.23.0.windows.1 [application/octet-stream] v12-0003-A-temporary-patch-that-includes-patch-in-another.patch (8.8K, ../../OS3PR01MB62753CDFD0DF1FCBC978AFB39EAF9@OS3PR01MB6275.jpnprd01.prod.outlook.com/4-v12-0003-A-temporary-patch-that-includes-patch-in-another.patch) download | inline diff: From 93507672247938ef22bc4ff15c4d23f7140d84b0 Mon Sep 17 00:00:00 2001 From: wangw <[email protected]> Date: Thu, 16 Jun 2022 13:31:59 +0800 Subject: [PATCH v12 3/5] A temporary patch that includes patch in another bugfix thread. The partition cache map on subscriber have several bugs (see thread [1]). Because patch 0004 is developed based on the patches in [1], so I use latest 0001-patch (see [2]) as a temporary patch 0003 here. After the patch in [1] is committed, I will delete this temporary patch. [1] - https://www.postgresql.org/message-id/OSZPR01MB6310F46CD425A967E4AEF736FDA49%40OSZPR01MB6310.jpnprd01.prod.outlook.com [2] - https://www.postgresql.org/message-id/OSZPR01MB6310898F35D480D58689A991FDAC9%40OSZPR01MB6310.jpnprd01.prod.outlook.com --- src/backend/replication/logical/relation.c | 121 ++++++++++++--------- src/backend/replication/logical/worker.c | 20 +++- src/test/subscription/t/013_partition.pl | 14 +++ 3 files changed, 98 insertions(+), 57 deletions(-) diff --git a/src/backend/replication/logical/relation.c b/src/backend/replication/logical/relation.c index 34c55c04e3..18e657cdfe 100644 --- a/src/backend/replication/logical/relation.c +++ b/src/backend/replication/logical/relation.c @@ -249,6 +249,72 @@ logicalrep_report_missing_attrs(LogicalRepRelation *remoterel, } } +/* + * Check if replica identity matches and mark the updatable flag. + * + * We allow for stricter replica identity (fewer columns) on subscriber as + * that will not stop us from finding unique tuple. IE, if publisher has + * identity (id,timestamp) and subscriber just (id) this will not be a + * problem, but in the opposite scenario it will. + * + * Don't throw any error here just mark the relation entry as not updatable, + * as replica identity is only for updates and deletes but inserts can be + * replicated even without it. + */ +static void +logicalrep_rel_mark_updatable(LogicalRepRelMapEntry *entry) +{ + Bitmapset *idkey; + LogicalRepRelation *remoterel = &entry->remoterel; + int i; + + entry->updatable = true; + + /* + * If it is a partitioned table, we don't check it, we will check its + * partition later. + */ + if (entry->localrel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + return; + + idkey = RelationGetIndexAttrBitmap(entry->localrel, + INDEX_ATTR_BITMAP_IDENTITY_KEY); + /* fallback to PK if no replica identity */ + if (idkey == NULL) + { + idkey = RelationGetIndexAttrBitmap(entry->localrel, + INDEX_ATTR_BITMAP_PRIMARY_KEY); + /* + * If no replica identity index and no PK, the published table + * must have replica identity FULL. + */ + if (idkey == NULL && remoterel->replident != REPLICA_IDENTITY_FULL) + entry->updatable = false; + } + + i = -1; + while ((i = bms_next_member(idkey, i)) >= 0) + { + int attnum = i + FirstLowInvalidHeapAttributeNumber; + + if (!AttrNumberIsForUserDefinedAttr(attnum)) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("logical replication target relation \"%s.%s\" uses " + "system columns in REPLICA IDENTITY index", + remoterel->nspname, remoterel->relname))); + + attnum = AttrNumberGetAttrOffset(attnum); + + if (entry->attrmap->attnums[attnum] < 0 || + !bms_is_member(entry->attrmap->attnums[attnum], remoterel->attkeys)) + { + entry->updatable = false; + break; + } + } +} + /* * Open the local relation associated with the remote one. * @@ -307,7 +373,6 @@ logicalrep_rel_open(LogicalRepRelId remoteid, LOCKMODE lockmode) if (!entry->localrelvalid) { Oid relid; - Bitmapset *idkey; TupleDesc desc; MemoryContext oldctx; int i; @@ -365,55 +430,8 @@ logicalrep_rel_open(LogicalRepRelId remoteid, LOCKMODE lockmode) /* be tidy */ bms_free(missingatts); - /* - * Check that replica identity matches. We allow for stricter replica - * identity (fewer columns) on subscriber as that will not stop us - * from finding unique tuple. IE, if publisher has identity - * (id,timestamp) and subscriber just (id) this will not be a problem, - * but in the opposite scenario it will. - * - * Don't throw any error here just mark the relation entry as not - * updatable, as replica identity is only for updates and deletes but - * inserts can be replicated even without it. - */ - entry->updatable = true; - idkey = RelationGetIndexAttrBitmap(entry->localrel, - INDEX_ATTR_BITMAP_IDENTITY_KEY); - /* fallback to PK if no replica identity */ - if (idkey == NULL) - { - idkey = RelationGetIndexAttrBitmap(entry->localrel, - INDEX_ATTR_BITMAP_PRIMARY_KEY); - - /* - * If no replica identity index and no PK, the published table - * must have replica identity FULL. - */ - if (idkey == NULL && remoterel->replident != REPLICA_IDENTITY_FULL) - entry->updatable = false; - } - - i = -1; - while ((i = bms_next_member(idkey, i)) >= 0) - { - int attnum = i + FirstLowInvalidHeapAttributeNumber; - - if (!AttrNumberIsForUserDefinedAttr(attnum)) - ereport(ERROR, - (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), - errmsg("logical replication target relation \"%s.%s\" uses " - "system columns in REPLICA IDENTITY index", - remoterel->nspname, remoterel->relname))); - - attnum = AttrNumberGetAttrOffset(attnum); - - if (entry->attrmap->attnums[attnum] < 0 || - !bms_is_member(entry->attrmap->attnums[attnum], remoterel->attkeys)) - { - entry->updatable = false; - break; - } - } + /* Check that replica identity matches. */ + logicalrep_rel_mark_updatable(entry); entry->localrelvalid = true; } @@ -651,7 +669,8 @@ logicalrep_partition_open(LogicalRepRelMapEntry *root, attrmap->maplen * sizeof(AttrNumber)); } - entry->updatable = root->updatable; + /* Check that replica identity matches. */ + logicalrep_rel_mark_updatable(entry); entry->localrelvalid = true; diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c index 18d6f33479..7d05254bde 100644 --- a/src/backend/replication/logical/worker.c +++ b/src/backend/replication/logical/worker.c @@ -2358,6 +2358,8 @@ apply_handle_tuple_routing(ApplyExecutionData *edata, TupleTableSlot *remoteslot_part; TupleConversionMap *map; MemoryContext oldctx; + LogicalRepRelMapEntry *part_entry = NULL; + AttrMap *attrmap = NULL; /* ModifyTableState is needed for ExecFindPartition(). */ edata->mtstate = mtstate = makeNode(ModifyTableState); @@ -2389,8 +2391,11 @@ apply_handle_tuple_routing(ApplyExecutionData *edata, remoteslot_part = table_slot_create(partrel, &estate->es_tupleTable); map = partrelinfo->ri_RootToPartitionMap; if (map != NULL) - remoteslot_part = execute_attr_map_slot(map->attrMap, remoteslot, + { + attrmap = map->attrMap; + remoteslot_part = execute_attr_map_slot(attrmap, remoteslot, remoteslot_part); + } else { remoteslot_part = ExecCopySlot(remoteslot_part, remoteslot); @@ -2398,6 +2403,14 @@ apply_handle_tuple_routing(ApplyExecutionData *edata, } MemoryContextSwitchTo(oldctx); + /* Check if we can do the update or delete on the leaf partition. */ + if(operation == CMD_UPDATE || operation == CMD_DELETE) + { + part_entry = logicalrep_partition_open(relmapentry, partrel, + attrmap); + check_relation_updatable(part_entry); + } + switch (operation) { case CMD_INSERT: @@ -2419,15 +2432,10 @@ apply_handle_tuple_routing(ApplyExecutionData *edata, * suitable partition. */ { - AttrMap *attrmap = map ? map->attrMap : NULL; - LogicalRepRelMapEntry *part_entry; TupleTableSlot *localslot; ResultRelInfo *partrelinfo_new; bool found; - part_entry = logicalrep_partition_open(relmapentry, partrel, - attrmap); - /* Get the matching local tuple from the partition. */ found = FindReplTupleInLocalRel(estate, partrel, &part_entry->remoterel, diff --git a/src/test/subscription/t/013_partition.pl b/src/test/subscription/t/013_partition.pl index 69f4009a14..e07a9217d7 100644 --- a/src/test/subscription/t/013_partition.pl +++ b/src/test/subscription/t/013_partition.pl @@ -868,4 +868,18 @@ $result = $node_subscriber2->safe_psql('postgres', "SELECT a, b, c FROM tab5 ORDER BY 1"); is($result, qq(3||1), 'updates of tab5 replicated correctly after altering table on publisher'); +# Alter REPLICA IDENTITY on subscriber. +# No REPLICA IDENTITY in the partitioned table on subscriber, but what we check +# is the partition, so it works fine. +$node_subscriber2->safe_psql('postgres', + "ALTER TABLE tab5 REPLICA IDENTITY NOTHING"); + +$node_publisher->safe_psql('postgres', "UPDATE tab5 SET a = 4 WHERE a = 3"); + +$node_publisher->wait_for_catchup('sub2'); + +$result = $node_subscriber2->safe_psql('postgres', + "SELECT a, b, c FROM tab5_1 ORDER BY 1"); +is($result, qq(4||1), 'updates of tab5 replicated correctly'); + done_testing(); -- 2.23.0.windows.1 [application/octet-stream] v12-0004-Add-some-checks-before-using-apply-background-wo.patch (35.8K, ../../OS3PR01MB62753CDFD0DF1FCBC978AFB39EAF9@OS3PR01MB6275.jpnprd01.prod.outlook.com/5-v12-0004-Add-some-checks-before-using-apply-background-wo.patch) download | inline diff: From e99cbbf254246eeb0a441e9320f999c8b85f6cdf Mon Sep 17 00:00:00 2001 From: wangw <[email protected]> Date: Tue, 14 Jun 2022 11:23:52 +0800 Subject: [PATCH v12 4/5] Add some checks before using apply background worker to apply changes. If any of the following checks are violated, an error will be reported. 1. The unique columns between publisher and subscriber are difference. 2. There is any non-immutable function present in expression in subscriber's relation. Check from the following 4 items: a. The function in triggers; b. Column default value expressions and domain constraints; c. Constraint expressions. d. The foreign keys. --- doc/src/sgml/ref/create_subscription.sgml | 4 + .../replication/logical/applybgwroker.c | 44 ++ src/backend/replication/logical/proto.c | 62 ++- src/backend/replication/logical/relation.c | 196 +++++++++ src/backend/replication/logical/tablesync.c | 1 + src/backend/replication/logical/worker.c | 25 +- src/backend/utils/cache/typcache.c | 17 + src/include/replication/logicalproto.h | 1 + src/include/replication/logicalrelation.h | 16 + src/include/replication/worker_internal.h | 1 + src/include/utils/typcache.h | 2 + .../subscription/t/022_twophase_cascade.pl | 7 + .../subscription/t/032_streaming_apply.pl | 391 ++++++++++++++++++ 13 files changed, 759 insertions(+), 8 deletions(-) create mode 100644 src/test/subscription/t/032_streaming_apply.pl diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml index d4caae0222..8de1a23ce4 100644 --- a/doc/src/sgml/ref/create_subscription.sgml +++ b/doc/src/sgml/ref/create_subscription.sgml @@ -240,6 +240,10 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl the transaction is committed. Note that if an error happens when applying changes in a background worker, it might not report the finish LSN of the remote transaction in the server log. + To run in this mode, there are following two requirements. The first + is that the unique column should be the same between publisher and + subscriber; the second is that there should not be any non-immutable + function in subscriber-side replicated table. </para> </listitem> </varlistentry> diff --git a/src/backend/replication/logical/applybgwroker.c b/src/backend/replication/logical/applybgwroker.c index 7e8681590f..1c8dbfa9b5 100644 --- a/src/backend/replication/logical/applybgwroker.c +++ b/src/backend/replication/logical/applybgwroker.c @@ -773,3 +773,47 @@ apply_bgworker_subxact_info_add(TransactionId current_xid) MemoryContextSwitchTo(oldctx); } } + +/* + * Check if changes on this logical replication relation can be applied by + * apply background worker. + * + * Although we maintains the commit order by allowing only one process to + * commit at a time, our access order to the relation has changed. + * This could cause unexpected problems if the unique column on the replicated + * table is inconsistent with the publisher-side or contains non-immutable + * functions when applying transactions in the apply background worker. + */ +void +apply_bgworker_relation_check(LogicalRepRelMapEntry *rel) +{ + /* Check only we are in apply bgworker. */ + if (!am_apply_bgworker()) + return; + + /* + * If it is a partitioned table, we do not check it, we will check its + * partition later. + */ + if (rel->localrel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + return; + + /* + * If any unique index exist, check that they are same as remoterel. + */ + if (!rel->sameunique) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot replicate relation with different unique index"), + errhint("Please change the streaming option to 'on' instead of 'apply'."))); + + /* + * Check if there is any non-immutable function present in expression in + * this relation. + */ + if (rel->volatility == FUNCTION_NONIMMUTABLE) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot replicate relation. There is at least one non-immutable function"), + errhint("Please change the streaming option to 'on' instead of 'apply'."))); +} diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c index a888038eb4..6af3302270 100644 --- a/src/backend/replication/logical/proto.c +++ b/src/backend/replication/logical/proto.c @@ -23,7 +23,8 @@ /* * Protocol message flags. */ -#define LOGICALREP_IS_REPLICA_IDENTITY 1 +#define LOGICALREP_IS_REPLICA_IDENTITY 0x0001 +#define LOGICALREP_IS_UNIQUE 0x0002 #define MESSAGE_TRANSACTIONAL (1<<0) #define TRUNCATE_CASCADE (1<<0) @@ -933,11 +934,55 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns) TupleDesc desc; int i; uint16 nliveatts = 0; - Bitmapset *idattrs = NULL; + Bitmapset *idattrs = NULL, + *attunique = NULL; bool replidentfull; desc = RelationGetDescr(rel); + if (rel->rd_rel->relhasindex) + { + List *indexoidlist = RelationGetIndexList(rel); + ListCell *indexoidscan; + + foreach(indexoidscan, indexoidlist) + { + Oid indexoid = lfirst_oid(indexoidscan); + Relation indexRel; + + /* Look up the description for index */ + indexRel = RelationIdGetRelation(indexoid); + + if (!RelationIsValid(indexRel)) + elog(ERROR, "could not open relation with OID %u", indexoid); + + if (indexRel->rd_index->indisunique) + { + int i; + + /* Add referenced attributes to idindexattrs */ + for (i = 0; i < indexRel->rd_index->indnatts; i++) + { + int attrnum = indexRel->rd_index->indkey.values[i]; + + /* + * We don't include non-key columns into idindexattrs + * bitmaps. See RelationGetIndexAttrBitmap. + */ + if (attrnum != 0) + { + if (i < indexRel->rd_index->indnkeyatts && + !bms_is_member(attrnum - FirstLowInvalidHeapAttributeNumber, attunique)) + attunique = bms_add_member(attunique, + attrnum - FirstLowInvalidHeapAttributeNumber); + } + } + } + RelationClose(indexRel); + } + list_free(indexoidlist); + } + /* send number of live attributes */ for (i = 0; i < desc->natts; i++) { @@ -976,6 +1021,10 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns) idattrs)) flags |= LOGICALREP_IS_REPLICA_IDENTITY; + if (bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber, + attunique)) + flags |= LOGICALREP_IS_UNIQUE; + pq_sendbyte(out, flags); /* attribute name */ @@ -1001,7 +1050,8 @@ logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel) int natts; char **attnames; Oid *atttyps; - Bitmapset *attkeys = NULL; + Bitmapset *attkeys = NULL, + *attunique = NULL; natts = pq_getmsgint(in, 2); attnames = palloc(natts * sizeof(char *)); @@ -1012,11 +1062,14 @@ logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel) { uint8 flags; - /* Check for replica identity column */ + /* Check for replica identity and unique column */ flags = pq_getmsgbyte(in); if (flags & LOGICALREP_IS_REPLICA_IDENTITY) attkeys = bms_add_member(attkeys, i); + if (flags & LOGICALREP_IS_UNIQUE) + attunique = bms_add_member(attunique, i); + /* attribute name */ attnames[i] = pstrdup(pq_getmsgstring(in)); @@ -1030,6 +1083,7 @@ logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel) rel->attnames = attnames; rel->atttyps = atttyps; rel->attkeys = attkeys; + rel->attunique = attunique; rel->natts = natts; } diff --git a/src/backend/replication/logical/relation.c b/src/backend/replication/logical/relation.c index 18e657cdfe..a36007a209 100644 --- a/src/backend/replication/logical/relation.c +++ b/src/backend/replication/logical/relation.c @@ -19,12 +19,19 @@ #include "access/table.h" #include "catalog/namespace.h" +#include "catalog/pg_proc.h" #include "catalog/pg_subscription_rel.h" +#include "commands/trigger.h" #include "executor/executor.h" #include "nodes/makefuncs.h" +#include "optimizer/optimizer.h" #include "replication/logicalrelation.h" #include "replication/worker_internal.h" +#include "rewrite/rewriteHandler.h" #include "utils/inval.h" +#include "utils/lsyscache.h" +#include "utils/syscache.h" +#include "utils/typcache.h" static MemoryContext LogicalRepRelMapContext = NULL; @@ -91,6 +98,23 @@ logicalrep_relmap_invalidate_cb(Datum arg, Oid reloid) } } +/* + * Reset the flag volatility of all existing entry in the relation map cache. + */ +static void +logicalrep_relmap_reset_volatility_cb(Datum arg, int cacheid, uint32 hashvalue) +{ + HASH_SEQ_STATUS hash_seq; + LogicalRepRelMapEntry *entry; + + if (LogicalRepRelMap == NULL) + return; + + hash_seq_init(&hash_seq, LogicalRepRelMap); + while ((entry = hash_seq_search(&hash_seq)) != NULL) + entry->volatility = FUNCTION_UNKNOWN; +} + /* * Initialize the relation map cache. */ @@ -116,6 +140,9 @@ logicalrep_relmap_init(void) /* Watch for invalidation events. */ CacheRegisterRelcacheCallback(logicalrep_relmap_invalidate_cb, (Datum) 0); + CacheRegisterSyscacheCallback(PROCOID, + logicalrep_relmap_reset_volatility_cb, + (Datum) 0); } /* @@ -142,6 +169,7 @@ logicalrep_relmap_free_entry(LogicalRepRelMapEntry *entry) pfree(remoterel->atttyps); } bms_free(remoterel->attkeys); + bms_free(remoterel->attunique); if (entry->attrmap) pfree(entry->attrmap); @@ -190,6 +218,7 @@ logicalrep_relmap_update(LogicalRepRelation *remoterel) } entry->remoterel.replident = remoterel->replident; entry->remoterel.attkeys = bms_copy(remoterel->attkeys); + entry->remoterel.attunique = bms_copy(remoterel->attunique); MemoryContextSwitchTo(oldctx); } @@ -315,6 +344,160 @@ logicalrep_rel_mark_updatable(LogicalRepRelMapEntry *entry) } } +/* + * Check if unique index/constraint matches and mark sameunique and volatility + * flag. + * + * Don't throw any error here just mark the relation entry as not sameunique or + * FUNCTION_NONIMMUTABLE as we only check these in apply background worker. + */ +static void +logicalrep_rel_mark_safe_in_apply_bgworker(LogicalRepRelMapEntry *entry) +{ + Bitmapset *ukey; + int i; + TupleDesc tupdesc; + int attnum; + List *fkeys = NIL; + + /* + * Check that the unique column in the relation on the subscriber-side is + * also the unique column on the publisher-side. + */ + entry->sameunique = true; + ukey = RelationGetIndexAttrBitmap(entry->localrel, + INDEX_ATTR_BITMAP_KEY); + + if (ukey) + { + i = -1; + while ((i = bms_next_member(ukey, i)) >= 0) + { + attnum = AttrNumberGetAttrOffset(i + FirstLowInvalidHeapAttributeNumber); + + if (entry->attrmap->attnums[attnum] < 0 || + !bms_is_member(entry->attrmap->attnums[attnum], entry->remoterel.attunique)) + { + entry->sameunique = false; + break; + } + } + } + + /* + * Check whether there is any non-immutable function in the local table. + * + * a. The function in triggers; + * b. Column default value expressions and domain constraints; + * c. Constraint expressions; + * d. Foreign keys. + */ + if (entry->volatility != FUNCTION_UNKNOWN) + return; + + /* Initialize the flag. */ + entry->volatility = FUNCTION_IMMUTABLE; + + /* Check the trigger functions. */ + if (entry->localrel->trigdesc != NULL) + { + for (i = 0; i < entry->localrel->trigdesc->numtriggers; i++) + { + Trigger *trig = entry->localrel->trigdesc->triggers + i; + + if (trig->tgenabled != TRIGGER_FIRES_ALWAYS && + trig->tgenabled != TRIGGER_FIRES_ON_REPLICA) + continue; + + if (func_volatile(trig->tgfoid) != PROVOLATILE_IMMUTABLE) + { + entry->volatility = FUNCTION_NONIMMUTABLE; + return; + } + } + } + + /* Check the columns. */ + tupdesc = RelationGetDescr(entry->localrel); + for (attnum = 0; attnum < tupdesc->natts; attnum++) + { + Form_pg_attribute att = TupleDescAttr(tupdesc, attnum); + + /* We don't need info for dropped or generated attributes */ + if (att->attisdropped || att->attgenerated) + continue; + + /* + * We don't need to check columns that only exist on the + * subscriber + */ + if (entry->attrmap->attnums[attnum] < 0) + continue; + + if (att->atthasdef) + { + Node *defaultexpr; + + defaultexpr = build_column_default(entry->localrel, attnum + 1); + if (contain_mutable_functions(defaultexpr)) + { + entry->volatility = FUNCTION_NONIMMUTABLE; + return; + } + } + + /* + * If the column is of a DOMAIN type, determine whether + * that domain has any CHECK expressions that are not + * immutable. + */ + if (get_typtype(att->atttypid) == TYPTYPE_DOMAIN) + { + List *domain_constraints; + ListCell *lc; + + domain_constraints = GetDomainConstraints(att->atttypid); + + foreach(lc, domain_constraints) + { + DomainConstraintState *con = (DomainConstraintState *) lfirst(lc); + + if (con->check_expr && contain_mutable_functions((Node *) con->check_expr)) + { + entry->volatility = FUNCTION_NONIMMUTABLE; + return; + } + } + } + } + + /* Check the constraints. */ + if (tupdesc->constr) + { + ConstrCheck *check = tupdesc->constr->check; + + /* + * Determine if there are any CHECK constraints which + * contains non-immutable function. + */ + for (i = 0; i < tupdesc->constr->num_check; i++) + { + Expr *check_expr = stringToNode(check[i].ccbin); + + if (contain_mutable_functions((Node *) check_expr)) + { + entry->volatility = FUNCTION_NONIMMUTABLE; + return; + } + } + } + + /* Check the foreign keys. */ + fkeys = RelationGetFKeyList(entry->localrel); + if (fkeys) + entry->volatility = FUNCTION_NONIMMUTABLE; +} + /* * Open the local relation associated with the remote one. * @@ -433,6 +616,12 @@ logicalrep_rel_open(LogicalRepRelId remoteid, LOCKMODE lockmode) /* Check that replica identity matches. */ logicalrep_rel_mark_updatable(entry); + /* + * Set flags to check later whether changes could be applied in the + * apply background worker. + */ + logicalrep_rel_mark_safe_in_apply_bgworker(entry); + entry->localrelvalid = true; } @@ -629,6 +818,7 @@ logicalrep_partition_open(LogicalRepRelMapEntry *root, } entry->remoterel.replident = remoterel->replident; entry->remoterel.attkeys = bms_copy(remoterel->attkeys); + entry->remoterel.attunique = bms_copy(remoterel->attunique); } entry->localrel = partrel; @@ -672,6 +862,12 @@ logicalrep_partition_open(LogicalRepRelMapEntry *root, /* Check that replica identity matches. */ logicalrep_rel_mark_updatable(entry); + /* + * Set flags to check later whether changes could be applied in the apply + * background worker. + */ + logicalrep_rel_mark_safe_in_apply_bgworker(entry); + entry->localrelvalid = true; /* state and statelsn are left set to 0. */ diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c index 8ffba7e2e5..3cdbf8b457 100644 --- a/src/backend/replication/logical/tablesync.c +++ b/src/backend/replication/logical/tablesync.c @@ -884,6 +884,7 @@ fetch_remote_table_info(char *nspname, char *relname, lrel->attnames = palloc0(MaxTupleAttributeNumber * sizeof(char *)); lrel->atttyps = palloc0(MaxTupleAttributeNumber * sizeof(Oid)); lrel->attkeys = NULL; + lrel->attunique = NULL; /* * Store the columns as a list of names. Ignore those that are not diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c index 7d05254bde..acf6494cd6 100644 --- a/src/backend/replication/logical/worker.c +++ b/src/backend/replication/logical/worker.c @@ -1365,6 +1365,16 @@ apply_handle_stream_stop(StringInfo s) { char action = LOGICAL_REP_MSG_STREAM_STOP; + /* + * There is no need to wait here to allow stream_stop to complete by + * background worker to avoid deadlocks. + * + * The deadlock problem only occurs if relation's unique + * index/constraint is different between publisher and subscriber. But + * in these cases, we do not allow to apply streamed transaction in the + * apply background worker (see function + * apply_bgworker_relation_check). + */ apply_bgworker_send_data(stream_apply_worker, 1, &action); elog(DEBUG1, "stopped streaming of xid %u, %u changes streamed", stream_xid, nchanges); @@ -1914,6 +1924,8 @@ apply_handle_insert(StringInfo s) /* Set relation for error callback */ apply_error_callback_arg.rel = rel; + apply_bgworker_relation_check(rel); + /* Initialize the executor state. */ edata = create_edata_for_relation(rel); estate = edata->estate; @@ -2050,6 +2062,8 @@ apply_handle_update(StringInfo s) /* Check if we can do the update. */ check_relation_updatable(rel); + apply_bgworker_relation_check(rel); + /* Initialize the executor state. */ edata = create_edata_for_relation(rel); estate = edata->estate; @@ -2218,6 +2232,8 @@ apply_handle_delete(StringInfo s) /* Check if we can do the delete. */ check_relation_updatable(rel); + apply_bgworker_relation_check(rel); + /* Initialize the executor state. */ edata = create_edata_for_relation(rel); estate = edata->estate; @@ -2403,13 +2419,14 @@ apply_handle_tuple_routing(ApplyExecutionData *edata, } MemoryContextSwitchTo(oldctx); + part_entry = logicalrep_partition_open(relmapentry, partrel, + attrmap); + + apply_bgworker_relation_check(part_entry); + /* Check if we can do the update or delete on the leaf partition. */ if(operation == CMD_UPDATE || operation == CMD_DELETE) - { - part_entry = logicalrep_partition_open(relmapentry, partrel, - attrmap); check_relation_updatable(part_entry); - } switch (operation) { diff --git a/src/backend/utils/cache/typcache.c b/src/backend/utils/cache/typcache.c index 808f9ebd0d..b248899d82 100644 --- a/src/backend/utils/cache/typcache.c +++ b/src/backend/utils/cache/typcache.c @@ -2540,6 +2540,23 @@ compare_values_of_enum(TypeCacheEntry *tcache, Oid arg1, Oid arg2) return 0; } +/* + * GetDomainConstraints --- get DomainConstraintState list of specified domain type + */ +List * +GetDomainConstraints(Oid type_id) +{ + TypeCacheEntry *typentry; + List *constraints = NIL; + + typentry = lookup_type_cache(type_id, TYPECACHE_DOMAIN_CONSTR_INFO); + + if(typentry->domainData != NULL) + constraints = typentry->domainData->constraints; + + return constraints; +} + /* * Load (or re-load) the enumData member of the typcache entry. */ diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h index 7bba77c9e7..a503eb62c2 100644 --- a/src/include/replication/logicalproto.h +++ b/src/include/replication/logicalproto.h @@ -108,6 +108,7 @@ typedef struct LogicalRepRelation char replident; /* replica identity */ char relkind; /* remote relation kind */ Bitmapset *attkeys; /* Bitmap of key columns */ + Bitmapset *attunique; /* Bitmap of unique columns */ } LogicalRepRelation; /* Type mapping info */ diff --git a/src/include/replication/logicalrelation.h b/src/include/replication/logicalrelation.h index 78cd7e77f5..4476cf7cec 100644 --- a/src/include/replication/logicalrelation.h +++ b/src/include/replication/logicalrelation.h @@ -15,6 +15,17 @@ #include "access/attmap.h" #include "replication/logicalproto.h" +/* + * States to determine volatility of the function in expressions in one + * relation. + */ +typedef enum RelFuncVolatility +{ + FUNCTION_UNKNOWN = 0, /* initializing */ + FUNCTION_IMMUTABLE, /* all functions are immutable function */ + FUNCTION_NONIMMUTABLE /* at least one non-immutable function */ +} RelFuncVolatility; + typedef struct LogicalRepRelMapEntry { LogicalRepRelation remoterel; /* key is remoterel.remoteid */ @@ -31,6 +42,11 @@ typedef struct LogicalRepRelMapEntry Relation localrel; /* relcache entry (NULL when closed) */ AttrMap *attrmap; /* map of local attributes to remote ones */ bool updatable; /* Can apply updates/deletes? */ + bool sameunique; /* Are all unique columns of the local + relation contained by the unique columns in + remote? */ + RelFuncVolatility volatility; /* all functions in localrel are + immutable function? */ /* Sync state. */ char state; diff --git a/src/include/replication/worker_internal.h b/src/include/replication/worker_internal.h index 0302836dee..ac08a40d42 100644 --- a/src/include/replication/worker_internal.h +++ b/src/include/replication/worker_internal.h @@ -195,6 +195,7 @@ extern void apply_bgworker_free(ApplyBgworkerState *wstate); extern void apply_bgworker_check_status(void); extern void apply_bgworker_set_status(ApplyBgworkerStatus status); extern void apply_bgworker_subxact_info_add(TransactionId current_xid); +extern void apply_bgworker_relation_check(LogicalRepRelMapEntry *rel); static inline bool am_tablesync_worker(void) diff --git a/src/include/utils/typcache.h b/src/include/utils/typcache.h index 431ad7f1b3..ed7c2e7f48 100644 --- a/src/include/utils/typcache.h +++ b/src/include/utils/typcache.h @@ -199,6 +199,8 @@ extern uint64 assign_record_type_identifier(Oid type_id, int32 typmod); extern int compare_values_of_enum(TypeCacheEntry *tcache, Oid arg1, Oid arg2); +extern List *GetDomainConstraints(Oid type_id); + extern size_t SharedRecordTypmodRegistryEstimate(void); extern void SharedRecordTypmodRegistryInit(SharedRecordTypmodRegistry *, diff --git a/src/test/subscription/t/022_twophase_cascade.pl b/src/test/subscription/t/022_twophase_cascade.pl index b52af74e35..81791c8d3a 100644 --- a/src/test/subscription/t/022_twophase_cascade.pl +++ b/src/test/subscription/t/022_twophase_cascade.pl @@ -39,6 +39,13 @@ sub test_streaming ALTER SUBSCRIPTION tap_sub_C SET (streaming = $streaming_mode)"); + if ($streaming_mode eq 'apply') + { + $node_C->safe_psql( + 'postgres', " + ALTER TABLE test_tab ALTER c DROP DEFAULT"); + } + # Wait for subscribers to finish initialization $node_A->poll_query_until( diff --git a/src/test/subscription/t/032_streaming_apply.pl b/src/test/subscription/t/032_streaming_apply.pl new file mode 100644 index 0000000000..84a6900b33 --- /dev/null +++ b/src/test/subscription/t/032_streaming_apply.pl @@ -0,0 +1,391 @@ +# Copyright (c) 2022, PostgreSQL Global Development Group + +# Test the restrictions of streaming mode "apply" in logical replication + +use strict; +use warnings; +use PostgreSQL::Test::Cluster; +use PostgreSQL::Test::Utils; +use Test::More; + +my $offset = 0; + +# Create publisher node +my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); +$node_publisher->init(allows_streaming => 'logical'); +$node_publisher->append_conf('postgresql.conf', + 'logical_decoding_work_mem = 64kB'); +$node_publisher->start; + +# Create subscriber node +my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber'); +$node_subscriber->init; +$node_subscriber->start; + +# Setup structure on publisher +$node_publisher->safe_psql('postgres', + "CREATE TABLE test_tab (a int primary key, b varchar)"); +$node_publisher->safe_psql('postgres', + "CREATE TABLE test_tab_partitioned (a int primary key, b varchar)"); + +# Setup structure on subscriber +# We need to test normal table and partition table. +$node_subscriber->safe_psql('postgres', + "CREATE TABLE test_tab (a int primary key, b varchar)"); +$node_subscriber->safe_psql('postgres', + "CREATE TABLE test_tab_partitioned (a int primary key, b varchar) PARTITION BY RANGE(a)"); +$node_subscriber->safe_psql('postgres', + "CREATE TABLE test_tab_partition (LIKE test_tab_partitioned)"); +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partitioned ATTACH PARTITION test_tab_partition DEFAULT" +); + +# Setup logical replication +my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres'; +$node_publisher->safe_psql('postgres', + "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); +$node_publisher->safe_psql('postgres', + "CREATE PUBLICATION tap_pub_partitioned FOR TABLE test_tab_partitioned"); + +my $appname = 'tap_sub'; +$node_subscriber->safe_psql( + 'postgres', " + CREATE SUBSCRIPTION tap_sub + CONNECTION '$publisher_connstr application_name=$appname' + PUBLICATION tap_pub, tap_pub_partitioned + WITH (streaming = apply, copy_data = false)"); + +$node_publisher->wait_for_catchup($appname); + +# It is not allowed that the unique index on the publisher and the subscriber +# is different. Check the error reported by background worker in this case. +# First we check the unique index on normal table. +$node_subscriber->safe_psql('postgres', + "CREATE UNIQUE INDEX test_tab_b_idx ON test_tab (b)"); + +# Check that a background worker starts if "streaming" option is +# specified as "apply". We have to look for the DEBUG1 log messages +# about that, so temporarily bump up the log verbosity. +$node_subscriber->append_conf('postgresql.conf', "log_min_messages = debug1"); +$node_subscriber->reload; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); +$node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); +$node_subscriber->reload; + +$node_subscriber->wait_for_log( + qr/ERROR: cannot replicate relation with different unique index/, + $offset); + +# Drop the unique index on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', "DROP INDEX test_tab_b_idx"); + +$node_publisher->wait_for_catchup($appname); + +my $result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); +is($result, qq(5000), 'data replicated to subscriber after dropping index'); + +# Then we check the unique index on partition table. +$node_subscriber->safe_psql('postgres', + "CREATE UNIQUE INDEX test_tab_b_partition_idx ON test_tab_partition (b)"); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab_partitioned SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log( + qr/ERROR: cannot replicate relation with different unique index/, + $offset); + +# Drop the unique index on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', "DROP INDEX test_tab_b_partition_idx"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); +is($result, qq(5000), 'data replicated to subscriber after dropping index'); + +# Triggers which execute non-immutable function are not allowed on the +# subscriber side. Check the error reported by background worker in this case. +# First we check the trigger function on normal table. +$node_subscriber->safe_psql( + 'postgres', qq{ +CREATE FUNCTION trigger_func() RETURNS TRIGGER AS \$\$ + BEGIN + RETURN NULL; + END +\$\$ language plpgsql; +CREATE TRIGGER insert_trig +BEFORE INSERT ON test_tab +FOR EACH ROW EXECUTE PROCEDURE trigger_func(); +ALTER TABLE test_tab ENABLE REPLICA TRIGGER insert_trig; +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', "DELETE FROM test_tab"); + +$node_subscriber->wait_for_log( + qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, + $offset); + +# Drop the trigger on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "DROP TRIGGER insert_trig ON test_tab"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); +is($result, qq(0), 'data replicated to subscriber after dropping trigger'); + +# Then we check the unique index on partition table. +$node_subscriber->safe_psql( + 'postgres', qq{ +CREATE TRIGGER insert_trig +BEFORE INSERT ON test_tab_partition +FOR EACH ROW EXECUTE PROCEDURE trigger_func(); +ALTER TABLE test_tab_partition ENABLE REPLICA TRIGGER insert_trig; +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', "DELETE FROM test_tab_partitioned"); + +$node_subscriber->wait_for_log( + qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, + $offset); + +# Drop the trigger on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "DROP TRIGGER insert_trig ON test_tab_partition"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); +is($result, qq(0), 'data replicated to subscriber after dropping trigger'); + +# It is not allowed that column default value expression contains a +# non-immutable function on the subscriber side. Check the error reported by +# background worker in this case. +# First we check the column default value expression on normal table. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab ALTER COLUMN b SET DEFAULT random()"); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log( + qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, + $offset); + +# Drop default value on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab ALTER COLUMN b DROP DEFAULT"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); +is($result, qq(5000), + 'data replicated to subscriber after dropping default value expression'); + +# Then we check the column default value expression on partition table. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partition ALTER COLUMN b SET DEFAULT random()"); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab_partitioned SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log( + qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, + $offset); + +# Drop default value on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partition ALTER COLUMN b DROP DEFAULT"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); +is($result, qq(5000), + 'data replicated to subscriber after dropping default value expression'); + +# It is not allowed that domain constraint expression contains a non-immutable +# function on the subscriber side. Check the error reported by background +# worker in this case. +# Because the column type of the partition table must be the same as its parent +# table, only test normal table here. +$node_subscriber->safe_psql( + 'postgres', qq{ +CREATE DOMAIN test_domain AS int CHECK(VALUE > random()); +ALTER TABLE test_tab ALTER COLUMN a TYPE test_domain; +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', "DELETE FROM test_tab"); + +$node_subscriber->wait_for_log( + qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, + $offset); + +# Drop domain constraint expression on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab ALTER COLUMN a TYPE int"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); +is($result, qq(0), + 'data replicated to subscriber after dropping domain constraint expression' +); + +# It is not allowed that constraint expression contains a non-immutable function +# on the subscriber side. Check the error reported by background worker in this +# case. +# First we check the constraint expression on normal table. +$node_subscriber->safe_psql( + 'postgres', qq{ +ALTER TABLE test_tab ADD CONSTRAINT test_tab_con check (a > random()); +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log( + qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, + $offset); + +# Drop constraint on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab DROP CONSTRAINT test_tab_con"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); +is($result, qq(5000), + 'data replicated to subscriber after dropping constraint expression'); + +# Then we check the constraint expression on partition table. +$node_subscriber->safe_psql( + 'postgres', qq{ +ALTER TABLE test_tab_partition ADD CONSTRAINT test_tab_con check (a > random()); +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', "DELETE FROM test_tab_partitioned"); + +$node_subscriber->wait_for_log( + qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, + $offset); + +# Drop constraint on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partition DROP CONSTRAINT test_tab_con"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); +is($result, qq(0), + 'data replicated to subscriber after dropping constraint expression'); + +# It is not allowed that foreign key on the subscriber side. Check the error +# reported by background worker in this case. +# First we check the foreign key on normal table. +$node_publisher->safe_psql('postgres', "DELETE FROM test_tab"); +$node_publisher->wait_for_catchup($appname); +$node_subscriber->safe_psql( + 'postgres', qq{ +CREATE TABLE test_tab_f (a int primary key); +ALTER TABLE test_tab ADD CONSTRAINT test_tabfk FOREIGN KEY(a) REFERENCES test_tab_f(a); +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log( + qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, + $offset); + +# Drop the foreign key constraint on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab DROP CONSTRAINT test_tabfk"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); +is($result, qq(5000), + 'data replicated to subscriber after dropping the foreign key'); + +# Then we check the foreign key on partition table. +$node_publisher->wait_for_catchup($appname); +$node_subscriber->safe_psql( + 'postgres', qq{ +CREATE TABLE test_tab_partition_f (a int primary key); +ALTER TABLE test_tab_partition ADD CONSTRAINT test_tab_patition_fk FOREIGN KEY(a) REFERENCES test_tab_partition_f(a); +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab_partitioned SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log( + qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, + $offset); + +# Drop the foreign key constraint on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partition DROP CONSTRAINT test_tab_patition_fk"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); +is($result, qq(5000), + 'data replicated to subscriber after dropping the foreign key'); + +$node_subscriber->stop; +$node_publisher->stop; + +done_testing(); -- 2.23.0.windows.1 [application/octet-stream] v12-0005-Retry-to-apply-streaming-xact-only-in-apply-work.patch (25.5K, ../../OS3PR01MB62753CDFD0DF1FCBC978AFB39EAF9@OS3PR01MB6275.jpnprd01.prod.outlook.com/6-v12-0005-Retry-to-apply-streaming-xact-only-in-apply-work.patch) download | inline diff: From 4740660f490462d58d647f0d8cae50eb0cd78ae3 Mon Sep 17 00:00:00 2001 From: wangw <[email protected]> Date: Wed, 15 Jun 2022 11:02:08 +0800 Subject: [PATCH v12 5/5] Retry to apply streaming xact only in apply worker. --- doc/src/sgml/catalogs.sgml | 11 ++ doc/src/sgml/ref/create_subscription.sgml | 4 + src/backend/catalog/pg_subscription.c | 1 + src/backend/catalog/system_views.sql | 4 +- src/backend/commands/subscriptioncmds.c | 1 + .../replication/logical/applybgwroker.c | 19 ++- src/backend/replication/logical/worker.c | 89 +++++++++++ src/bin/pg_dump/pg_dump.c | 5 +- src/include/catalog/pg_subscription.h | 4 + .../subscription/t/032_streaming_apply.pl | 139 ++++++++++-------- 10 files changed, 209 insertions(+), 68 deletions(-) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index 5cb39b18bd..928ad02ace 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -7907,6 +7907,17 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>subretry</structfield> <type>bool</type> + </para> + <para> + If true, the subscription will not try to apply streaming transaction + in <literal>apply</literal> mode. See + <xref linkend="sql-createsubscription"/> for more information. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>subconninfo</structfield> <type>text</type> diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml index 8de1a23ce4..70fbf81668 100644 --- a/doc/src/sgml/ref/create_subscription.sgml +++ b/doc/src/sgml/ref/create_subscription.sgml @@ -244,6 +244,10 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl is that the unique column should be the same between publisher and subscriber; the second is that there should not be any non-immutable function in subscriber-side replicated table. + When applying a streaming transaction, if either requirement is not + met, the background worker will exit with an error. And when retrying + later, we will try to apply this transaction in <literal>on</literal> + mode. </para> </listitem> </varlistentry> diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c index add51caadf..1824390d7d 100644 --- a/src/backend/catalog/pg_subscription.c +++ b/src/backend/catalog/pg_subscription.c @@ -71,6 +71,7 @@ GetSubscription(Oid subid, bool missing_ok) sub->stream = subform->substream; sub->twophasestate = subform->subtwophasestate; sub->disableonerr = subform->subdisableonerr; + sub->retry = subform->subretry; /* Get conninfo */ datum = SysCacheGetAttr(SUBSCRIPTIONOID, diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql index fedaed533b..10f4dd6785 100644 --- a/src/backend/catalog/system_views.sql +++ b/src/backend/catalog/system_views.sql @@ -1298,8 +1298,8 @@ REVOKE ALL ON pg_replication_origin_status FROM public; -- All columns of pg_subscription except subconninfo are publicly readable. REVOKE ALL ON pg_subscription FROM public; GRANT SELECT (oid, subdbid, subskiplsn, subname, subowner, subenabled, - subbinary, substream, subtwophasestate, subdisableonerr, subslotname, - subsynccommit, subpublications) + subbinary, substream, subtwophasestate, subdisableonerr, + subretry, subslotname, subsynccommit, subpublications) ON pg_subscription TO public; CREATE VIEW pg_stat_subscription_stats AS diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c index 4080bba987..38697184d9 100644 --- a/src/backend/commands/subscriptioncmds.c +++ b/src/backend/commands/subscriptioncmds.c @@ -663,6 +663,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt, LOGICALREP_TWOPHASE_STATE_PENDING : LOGICALREP_TWOPHASE_STATE_DISABLED); values[Anum_pg_subscription_subdisableonerr - 1] = BoolGetDatum(opts.disableonerr); + values[Anum_pg_subscription_subretry - 1] = BoolGetDatum(false); values[Anum_pg_subscription_subconninfo - 1] = CStringGetTextDatum(conninfo); if (opts.slot_name) diff --git a/src/backend/replication/logical/applybgwroker.c b/src/backend/replication/logical/applybgwroker.c index 1c8dbfa9b5..603d0605a7 100644 --- a/src/backend/replication/logical/applybgwroker.c +++ b/src/backend/replication/logical/applybgwroker.c @@ -99,6 +99,19 @@ canstartapplybgworker(TransactionId xid, bool start) if (start && !XLogRecPtrIsInvalid(MySubscription->skiplsn)) return false; + /* + * We don't start new background worker if retry was set as it's possible + * that the last time we tried to apply a transaction in background worker + * and the check failed (see function apply_bgworker_relation_check). So + * we will try to apply this transaction in apply worker. + */ + if (start && MySubscription->retry) + { + elog(DEBUG1, "retry to apply an streaming transaction in apply " + "background worker"); + return false; + } + /* * For streaming transactions that are being applied in apply background * worker, we cannot decide whether to apply the change for a relation @@ -804,8 +817,7 @@ apply_bgworker_relation_check(LogicalRepRelMapEntry *rel) if (!rel->sameunique) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("cannot replicate relation with different unique index"), - errhint("Please change the streaming option to 'on' instead of 'apply'."))); + errmsg("cannot replicate relation with different unique index"))); /* * Check if there is any non-immutable function present in expression in @@ -814,6 +826,5 @@ apply_bgworker_relation_check(LogicalRepRelMapEntry *rel) if (rel->volatility == FUNCTION_NONIMMUTABLE) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("cannot replicate relation. There is at least one non-immutable function"), - errhint("Please change the streaming option to 'on' instead of 'apply'."))); + errmsg("cannot replicate relation. There is at least one non-immutable function"))); } diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c index acf6494cd6..be1da014c4 100644 --- a/src/backend/replication/logical/worker.c +++ b/src/backend/replication/logical/worker.c @@ -372,6 +372,8 @@ static void clear_subscription_skip_lsn(XLogRecPtr finish_lsn); static inline void set_apply_error_context_xact(TransactionId xid, XLogRecPtr lsn); static inline void reset_apply_error_context_info(void); +static void set_subscription_retry(bool retry); + /* * Should this worker apply changes for given relation. * @@ -888,6 +890,9 @@ apply_handle_commit(StringInfo s) apply_handle_commit_internal(&commit_data); + /* Set the flag that we will not retry later. */ + set_subscription_retry(false); + /* Check the status of apply background worker if any. */ apply_bgworker_check_status(); @@ -999,6 +1004,9 @@ apply_handle_prepare(StringInfo s) in_remote_transaction = false; + /* Set the flag that we will not retry later. */ + set_subscription_retry(false); + /* Check the status of apply background worker if any. */ apply_bgworker_check_status(); @@ -1052,6 +1060,9 @@ apply_handle_commit_prepared(StringInfo s) store_flush_position(prepare_data.end_lsn); in_remote_transaction = false; + /* Set the flag that we will not retry later. */ + set_subscription_retry(false); + /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(prepare_data.end_lsn); @@ -1107,6 +1118,9 @@ apply_handle_rollback_prepared(StringInfo s) store_flush_position(rollback_data.rollback_end_lsn); in_remote_transaction = false; + /* Set the flag that we will not retry later. */ + set_subscription_retry(false); + /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(rollback_data.rollback_end_lsn); @@ -1193,6 +1207,9 @@ apply_handle_stream_prepare(StringInfo s) /* unlink the files with serialized changes and subxact info. */ stream_cleanup_files(MyLogicalRepWorker->subid, prepare_data.xid); } + + /* Set the flag that we will not retry later. */ + set_subscription_retry(false); } in_remote_transaction = false; @@ -1621,6 +1638,9 @@ apply_handle_stream_abort(StringInfo s) */ else serialize_stream_abort(xid, subxid); + + /* Set the flag that we will not retry later. */ + set_subscription_retry(false); } } @@ -2792,6 +2812,9 @@ apply_handle_stream_commit(StringInfo s) /* unlink the files with serialized changes and subxact info */ stream_cleanup_files(MyLogicalRepWorker->subid, xid); } + + /* Set the flag that we will not retry later. */ + set_subscription_retry(false); } /* Check the status of apply background worker if any. */ @@ -3860,6 +3883,9 @@ start_table_sync(XLogRecPtr *origin_startpos, char **myslotname) } PG_CATCH(); { + /* Set the flag that we will retry later. */ + set_subscription_retry(true); + if (MySubscription->disableonerr) DisableSubscriptionAndExit(); else @@ -3898,6 +3924,9 @@ start_apply(XLogRecPtr origin_startpos) } PG_CATCH(); { + /* Set the flag that we will retry later. */ + set_subscription_retry(true); + if (MySubscription->disableonerr) DisableSubscriptionAndExit(); else @@ -4399,3 +4428,63 @@ reset_apply_error_context_info(void) apply_error_callback_arg.remote_attnum = -1; set_apply_error_context_xact(InvalidTransactionId, InvalidXLogRecPtr); } + +/* + * Set subretry of pg_subscription catalog. + * + * If retry is true, subscriber is about to exit with an error. Otherwise, it + * means that the changes was applied successfully. + */ +static void +set_subscription_retry(bool retry) +{ + Relation rel; + HeapTuple tup; + bool started_tx = false; + bool nulls[Natts_pg_subscription]; + bool replaces[Natts_pg_subscription]; + Datum values[Natts_pg_subscription]; + + if (MySubscription->retry == retry || + am_apply_bgworker()) + return; + + if (!IsTransactionState()) + { + StartTransactionCommand(); + started_tx = true; + } + + /* Look up the subscription in the catalog */ + rel = table_open(SubscriptionRelationId, RowExclusiveLock); + tup = SearchSysCacheCopy1(SUBSCRIPTIONOID, + ObjectIdGetDatum(MySubscription->oid)); + + if (!HeapTupleIsValid(tup)) + elog(ERROR, "subscription \"%s\" does not exist", MySubscription->name); + + LockSharedObject(SubscriptionRelationId, MySubscription->oid, 0, + AccessShareLock); + + /* Form a new tuple. */ + memset(values, 0, sizeof(values)); + memset(nulls, false, sizeof(nulls)); + memset(replaces, false, sizeof(replaces)); + + /* reset subretry */ + values[Anum_pg_subscription_subretry - 1] = BoolGetDatum(retry); + replaces[Anum_pg_subscription_subretry - 1] = true; + + tup = heap_modify_tuple(tup, RelationGetDescr(rel), values, nulls, + replaces); + + /* Update the catalog. */ + CatalogTupleUpdate(rel, &tup->t_self, tup); + + /* Cleanup. */ + heap_freetuple(tup); + table_close(rel, NoLock); + + if (started_tx) + CommitTransactionCommand(); +} diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 6f8b30abb0..9b3ffe0888 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -4472,8 +4472,9 @@ getSubscriptions(Archive *fout) ntups = PQntuples(res); /* - * Get subscription fields. We don't include subskiplsn in the dump as - * after restoring the dump this value may no longer be relevant. + * Get subscription fields. We don't include subskiplsn and subretry in + * the dump as after restoring the dump this value may no longer be + * relevant. */ i_tableoid = PQfnumber(res, "tableoid"); i_oid = PQfnumber(res, "oid"); diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h index 9b394a45fe..dc7597bf80 100644 --- a/src/include/catalog/pg_subscription.h +++ b/src/include/catalog/pg_subscription.h @@ -76,6 +76,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW bool subdisableonerr; /* True if a worker error should cause the * subscription to be disabled */ + bool subretry; /* True if the previous apply change failed. */ + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* Connection string to the publisher */ text subconninfo BKI_FORCE_NOT_NULL; @@ -115,6 +117,8 @@ typedef struct Subscription bool disableonerr; /* Indicates if the subscription should be * automatically disabled if a worker error * occurs */ + bool retry; /* Indicates if the previous apply change + * failed. */ char *conninfo; /* Connection string to the publisher */ char *slotname; /* Name of the replication slot */ char *synccommit; /* Synchronous commit setting for worker */ diff --git a/src/test/subscription/t/032_streaming_apply.pl b/src/test/subscription/t/032_streaming_apply.pl index 84a6900b33..8f2e254b39 100644 --- a/src/test/subscription/t/032_streaming_apply.pl +++ b/src/test/subscription/t/032_streaming_apply.pl @@ -58,7 +58,9 @@ $node_subscriber->safe_psql( $node_publisher->wait_for_catchup($appname); # It is not allowed that the unique index on the publisher and the subscriber -# is different. Check the error reported by background worker in this case. +# is different. Check the error reported by background worker in this case. And +# after retrying in apply worker, we check if the data is replicated +# successfully. # First we check the unique index on normal table. $node_subscriber->safe_psql('postgres', "CREATE UNIQUE INDEX test_tab_b_idx ON test_tab (b)"); @@ -82,14 +84,15 @@ $node_subscriber->wait_for_log( qr/ERROR: cannot replicate relation with different unique index/, $offset); -# Drop the unique index on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', "DROP INDEX test_tab_b_idx"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); my $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); -is($result, qq(5000), 'data replicated to subscriber after dropping index'); +is($result, qq(5000), 'data replicated to subscribers after retrying because of unique index'); + +# Drop the unique index on the subscriber. +$node_subscriber->safe_psql('postgres', "DROP INDEX test_tab_b_idx"); # Then we check the unique index on partition table. $node_subscriber->safe_psql('postgres', @@ -106,17 +109,20 @@ $node_subscriber->wait_for_log( qr/ERROR: cannot replicate relation with different unique index/, $offset); -# Drop the unique index on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', "DROP INDEX test_tab_b_partition_idx"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); -is($result, qq(5000), 'data replicated to subscriber after dropping index'); +is($result, qq(5000), 'data replicated to subscribers after retrying because of unique index'); + +# Drop the unique index on the subscriber. +$node_subscriber->safe_psql('postgres', "DROP INDEX test_tab_b_partition_idx"); # Triggers which execute non-immutable function are not allowed on the # subscriber side. Check the error reported by background worker in this case. +# And after retrying in apply worker, we check if the data is replicated +# successfully. # First we check the trigger function on normal table. $node_subscriber->safe_psql( 'postgres', qq{ @@ -140,15 +146,16 @@ $node_subscriber->wait_for_log( qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, $offset); -# Drop the trigger on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "DROP TRIGGER insert_trig ON test_tab"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); -is($result, qq(0), 'data replicated to subscriber after dropping trigger'); +is($result, qq(0), 'data replicated to subscribers after retrying because of trigger'); + +# Drop the trigger on the subscriber. +$node_subscriber->safe_psql('postgres', + "DROP TRIGGER insert_trig ON test_tab"); # Then we check the unique index on partition table. $node_subscriber->safe_psql( @@ -168,19 +175,21 @@ $node_subscriber->wait_for_log( qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, $offset); -# Drop the trigger on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "DROP TRIGGER insert_trig ON test_tab_partition"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); -is($result, qq(0), 'data replicated to subscriber after dropping trigger'); +is($result, qq(0), 'data replicated to subscribers after retrying because of trigger'); + +# Drop the trigger on the subscriber. +$node_subscriber->safe_psql('postgres', + "DROP TRIGGER insert_trig ON test_tab_partition"); # It is not allowed that column default value expression contains a # non-immutable function on the subscriber side. Check the error reported by -# background worker in this case. +# background worker in this case. And after retrying in apply worker, we check +# if the data is replicated successfully. # First we check the column default value expression on normal table. $node_subscriber->safe_psql('postgres', "ALTER TABLE test_tab ALTER COLUMN b SET DEFAULT random()"); @@ -196,16 +205,17 @@ $node_subscriber->wait_for_log( qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, $offset); -# Drop default value on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "ALTER TABLE test_tab ALTER COLUMN b DROP DEFAULT"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); is($result, qq(5000), - 'data replicated to subscriber after dropping default value expression'); + 'data replicated to subscribers after retrying because of column default value'); + +# Drop default value on the subscriber. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab ALTER COLUMN b DROP DEFAULT"); # Then we check the column default value expression on partition table. $node_subscriber->safe_psql('postgres', @@ -222,20 +232,22 @@ $node_subscriber->wait_for_log( qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, $offset); -# Drop default value on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "ALTER TABLE test_tab_partition ALTER COLUMN b DROP DEFAULT"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); is($result, qq(5000), - 'data replicated to subscriber after dropping default value expression'); + 'data replicated to subscribers after retrying because of column default value'); + +# Drop default value on the subscriber. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partition ALTER COLUMN b DROP DEFAULT"); # It is not allowed that domain constraint expression contains a non-immutable # function on the subscriber side. Check the error reported by background -# worker in this case. +# worker in this case. And after retrying in apply worker, we check if the data +# is replicated successfully. # Because the column type of the partition table must be the same as its parent # table, only test normal table here. $node_subscriber->safe_psql( @@ -253,21 +265,23 @@ $node_subscriber->wait_for_log( qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, $offset); -# Drop domain constraint expression on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "ALTER TABLE test_tab ALTER COLUMN a TYPE int"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); is($result, qq(0), - 'data replicated to subscriber after dropping domain constraint expression' + 'data replicated to subscribers after retrying because of domain' ); -# It is not allowed that constraint expression contains a non-immutable function -# on the subscriber side. Check the error reported by background worker in this -# case. +# Drop domain constraint expression on the subscriber. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab ALTER COLUMN a TYPE int"); + +# It is not allowed that constraint expression contains a non-immutable +# function on the subscriber side. Check the error reported by background +# worker in this case. And after retrying in apply worker, we check if the data +# is replicated successfully. # First we check the constraint expression on normal table. $node_subscriber->safe_psql( 'postgres', qq{ @@ -285,16 +299,17 @@ $node_subscriber->wait_for_log( qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, $offset); -# Drop constraint on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "ALTER TABLE test_tab DROP CONSTRAINT test_tab_con"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); is($result, qq(5000), - 'data replicated to subscriber after dropping constraint expression'); + 'data replicated to subscribers after retrying because of constraint'); + +# Drop constraint on the subscriber. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab DROP CONSTRAINT test_tab_con"); # Then we check the constraint expression on partition table. $node_subscriber->safe_psql( @@ -311,19 +326,21 @@ $node_subscriber->wait_for_log( qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, $offset); -# Drop constraint on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "ALTER TABLE test_tab_partition DROP CONSTRAINT test_tab_con"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); is($result, qq(0), - 'data replicated to subscriber after dropping constraint expression'); + 'data replicated to subscribers after retrying because of constraint'); + +# Drop constraint on the subscriber. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partition DROP CONSTRAINT test_tab_con"); # It is not allowed that foreign key on the subscriber side. Check the error -# reported by background worker in this case. +# reported by background worker in this case. And after retrying in apply +# worker, we check if the data is replicated successfully. # First we check the foreign key on normal table. $node_publisher->safe_psql('postgres', "DELETE FROM test_tab"); $node_publisher->wait_for_catchup($appname); @@ -344,16 +361,17 @@ $node_subscriber->wait_for_log( qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, $offset); -# Drop the foreign key constraint on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "ALTER TABLE test_tab DROP CONSTRAINT test_tabfk"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); is($result, qq(5000), - 'data replicated to subscriber after dropping the foreign key'); + 'data replicated to subscribers after retrying because of foreign key'); + +# Drop the foreign key constraint on the subscriber. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab DROP CONSTRAINT test_tabfk"); # Then we check the foreign key on partition table. $node_publisher->wait_for_catchup($appname); @@ -374,16 +392,17 @@ $node_subscriber->wait_for_log( qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, $offset); -# Drop the foreign key constraint on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "ALTER TABLE test_tab_partition DROP CONSTRAINT test_tab_patition_fk"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); is($result, qq(5000), - 'data replicated to subscriber after dropping the foreign key'); + 'data replicated to subscribers after retrying because of foreign key'); + +# Drop the foreign key constraint on the subscriber. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partition DROP CONSTRAINT test_tab_patition_fk"); $node_subscriber->stop; $node_publisher->stop; -- 2.23.0.windows.1 ^ permalink raw reply [nested|flat] 66+ messages in thread
* Re: Perform streaming logical transactions by background workers and parallel apply @ 2022-06-17 08:57 Amit Kapila <[email protected]> parent: [email protected] <[email protected]> 3 siblings, 0 replies; 66+ messages in thread From: Amit Kapila @ 2022-06-17 08:57 UTC (permalink / raw) To: [email protected] <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Peter Smith <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; PostgreSQL Hackers <[email protected]> On Fri, Jun 17, 2022 at 12:47 PM [email protected] <[email protected]> wrote: > > Attach the new patches. > Only changed patches 0001, 0004. > Few more comments on the previous version of patch: =========================================== 1. +/* + * Count the number of registered (not necessarily running) apply background + * worker for a subscription. + */ /worker/workers 2. +static void +apply_bgworker_setup_dsm(ApplyBgworkerState *wstate) +{ ... ... + int64 queue_size = 160000000; /* 16 MB for now */ I think it would be better to use define for this rather than a hard-coded value. 3. +/* + * Status for apply background worker. + */ +typedef enum ApplyBgworkerStatus +{ + APPLY_BGWORKER_ATTACHED = 0, + APPLY_BGWORKER_READY, + APPLY_BGWORKER_BUSY, + APPLY_BGWORKER_FINISHED, + APPLY_BGWORKER_EXIT +} ApplyBgworkerStatus; It would be better if you can add comments to explain each of these states. 4. + /* Set up one message queue per worker, plus one. */ + mq = shm_mq_create(shm_toc_allocate(toc, (Size) queue_size), + (Size) queue_size); + shm_toc_insert(toc, APPLY_BGWORKER_KEY_MQ, mq); + shm_mq_set_sender(mq, MyProc); I don't understand the meaning of 'plus one' in the above comment as the patch seems to be setting up just one queue here? 5. + + /* Attach the queues. */ + wstate->mq_handle = shm_mq_attach(mq, seg, NULL); Similar to above. If there is only one queue then the comment should say queue instead of queues. 6. snprintf(bgw.bgw_name, BGW_MAXLEN, "logical replication worker for subscription %u", subid); + else + snprintf(bgw.bgw_name, BGW_MAXLEN, + "logical replication background apply worker for subscription %u ", subid); No need for extra space after %u in the above code. 7. + launched = logicalrep_worker_launch(MyLogicalRepWorker->dbid, + MySubscription->oid, + MySubscription->name, + MyLogicalRepWorker->userid, + InvalidOid, + dsm_segment_handle(wstate->dsm_seg)); + + if (launched) + { + /* Wait for worker to attach. */ + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_ATTACHED); In logicalrep_worker_launch(), we already seem to be waiting for workers to attach via WaitForReplicationWorkerAttach(), so it is not clear to me why we need to wait again? If there is a genuine reason then it is better to add some comments to explain it. I think in some way, we need to know if the worker is successfully attached and we may not get that via WaitForReplicationWorkerAttach, so there needs to be some way to know that but this doesn't sound like a very good idea. If that understanding is correct then can we think of a better way? 8. I think we can simplify apply_bgworker_find_or_start by having separate APIs for find and start. Most of the places need to use find API except for the first stream. If we do that then I think you don't need to make a hash entry unless we established ApplyBgworkerState which currently looks odd as you need to remove the entry if we fail to allocate the state. 9. + /* + * TO IMPROVE: Do we need to display the apply background worker's + * information in pg_stat_replication ? + */ + UpdateWorkerStats(last_received, send_time, false); In this do you mean to say pg_stat_subscription? If so, then to decide whether we need to update stats here we should see what additional information we can update here which is not possible via the main apply worker? 10. ApplyBgworkerMain { ... + /* Load the subscription into persistent memory context. */ + ApplyContext = AllocSetContextCreate(TopMemoryContext, ... This comment seems to be copied from ApplyWorkerMain but doesn't apply here. -- With Regards, Amit Kapila. ^ permalink raw reply [nested|flat] 66+ messages in thread
* Re: Perform streaming logical transactions by background workers and parallel apply @ 2022-06-20 02:59 Amit Kapila <[email protected]> parent: [email protected] <[email protected]> 3 siblings, 1 reply; 66+ messages in thread From: Amit Kapila @ 2022-06-20 02:59 UTC (permalink / raw) To: [email protected] <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Peter Smith <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; PostgreSQL Hackers <[email protected]> On Fri, Jun 17, 2022 at 12:47 PM [email protected] <[email protected]> wrote: > > On Wed, Jun 15, 2022 at 8:13 PM Amit Kapila <[email protected]> wrote: > > Few questions/comments on 0001 > > =========================== > Thanks for your comments. > > > 1. > > In the commit message, I see: "We also need to allow stream_stop to > > complete by the apply background worker to avoid deadlocks because > > T-1's current stream of changes can update rows in conflicting order > > with T-2's next stream of changes." > > > > Thinking about this, won't the T-1 and T-2 deadlock on the publisher > > node as well if the above statement is true? > Yes, I think so. > I think if table's unique index/constraint of the publisher and the subscriber > are consistent, the deadlock will occur on the publisher-side. > If it is inconsistent, deadlock may only occur in the subscriber. But since we > added the check for these (see patch 0004), so it seems okay to not handle this > at STREAM_STOP. > > BTW, I made the following improvements to the code (#a, #c are improved in 0004 > patch, #b, #d and #e are improved in 0001 patch.) : > a. > I added some comments in the function apply_handle_stream_stop to explain why > we do not need to allow stream_stop to complete by the apply background worker. > I have improved the comments in this and other related sections of the patch. See attached. > > > > 3. > > + > > + <para> > > + Setting streaming mode to <literal>apply</literal> could export invalid LSN > > + as finish LSN of failed transaction. Changing the streaming mode and making > > + the same conflict writes the finish LSN of the failed transaction in the > > + server log if required. > > + </para> > > > > How will the user identify that this is an invalid LSN value and she > > shouldn't use it to SKIP the transaction? Can we change the second > > sentence to: "User should change the streaming mode to 'on' if they > > would instead wish to see the finish LSN on error. Users can use > > finish LSN to SKIP applying the transaction." I think we can give > > reference to docs where the SKIP feature is explained. > Improved the sentence as suggested. > You haven't answered first part of the comment: "How will the user identify that this is an invalid LSN value and she shouldn't use it to SKIP the transaction?". Have you checked what value it displays? For example, in one of the case in apply_error_callback as shown in below code, we don't even display finish LSN if it is invalid. else if (XLogRecPtrIsInvalid(errarg->finish_lsn)) errcontext("processing remote data for replication origin \"%s\" during \"%s\" in transaction %u", errarg->origin_name, logicalrep_message_type(errarg->command), errarg->remote_xid); -- With Regards, Amit Kapila. Attachments: [application/octet-stream] improve_comments_1.patch (2.5K, ../../CAA4eK1+iiwpfmaoNPPktTkVTGjprv_Fjpr_fu7yoOUKEfDtv_A@mail.gmail.com/2-improve_comments_1.patch) download | inline diff: diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c index acf6494cd6..e621eb5e44 100644 --- a/src/backend/replication/logical/worker.c +++ b/src/backend/replication/logical/worker.c @@ -32,7 +32,12 @@ * transaction commit is received and also wait for the worker to finish at * commit. This preserves commit ordering and avoids writing to and reading * from file in most cases. We still need to spill if there is no worker - * available. + * available. It is important to maintain commit order to avoid failures + * due to (a) transaction dependencies, say if we insert a row in the first + * transaction and update it in the second transaction then allowing to apply + * both in parallel can lead to failure in the update. (b) deadlocks, allowing + * transactions that update the same set of rows/tables in opposite order to be + * applied in parallel can lead to deadlocks. * * 2) Write to temporary files and apply when the final commit arrives * @@ -1366,14 +1371,12 @@ apply_handle_stream_stop(StringInfo s) char action = LOGICAL_REP_MSG_STREAM_STOP; /* - * There is no need to wait here to allow stream_stop to complete by - * background worker to avoid deadlocks. - * - * The deadlock problem only occurs if relation's unique - * index/constraint is different between publisher and subscriber. But - * in these cases, we do not allow to apply streamed transaction in the - * apply background worker (see function - * apply_bgworker_relation_check). + * Unlike stream_commit, we don't need to wait here for stream_stop to + * finish. Allowing the other transaction to be applied before stream_stop + * is finished can only lead to failures if the unique index/constraint is + * different between publisher and subscriber. But for such cases, we don't + * allow streamed transactions to be applied in parallel. See + * apply_bgworker_relation_check. */ apply_bgworker_send_data(stream_apply_worker, 1, &action); @@ -2764,7 +2767,11 @@ apply_handle_stream_commit(StringInfo s) /* Send commit message */ apply_bgworker_send_data(wstate, s->len, s->data); - /* Wait for apply background worker to finish */ + /* + * Wait for apply background worker to finish. This is required to + * maintain commit order which avoids failures due to transaction + * dependencies and deadlocks. + */ apply_bgworker_wait_for(wstate, APPLY_BGWORKER_FINISHED); pgstat_report_stat(false); ^ permalink raw reply [nested|flat] 66+ messages in thread
* Re: Perform streaming logical transactions by background workers and parallel apply @ 2022-06-21 01:41 Peter Smith <[email protected]> parent: [email protected] <[email protected]> 0 siblings, 2 replies; 66+ messages in thread From: Peter Smith @ 2022-06-21 01:41 UTC (permalink / raw) To: [email protected] <[email protected]>; +Cc: Amit Kapila <[email protected]>; Masahiko Sawada <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; PostgreSQL Hackers <[email protected]> Here are some review comments for the v11-0001 patch. (I will review the remaining patches 0002-0005 and post any comments later) ====== 1. General I still feel that 'apply' seems like a meaningless enum value for this feature because from a user point-of-view every replicated change gets "applied". IMO something like 'streaming = parallel' or 'streaming = background' (etc) might have more meaning for a user. ====== 2. Commit message We also need to allow stream_stop to complete by the apply background worker to avoid deadlocks because T-1's current stream of changes can update rows in conflicting order with T-2's next stream of changes. Did this mean to say? "allow stream_stop to complete by" -> "allow stream_stop to be performed by" ~~~ 3. Commit message This patch also extends the SUBSCRIPTION 'streaming' option so that the user can control whether to apply the streaming transaction in an apply background worker or spill the change to disk. User can set the streaming option to 'on/off', 'apply'. For now, 'apply' means the streaming will be applied via a apply background worker if available. 'on' means the streaming transaction will be spilled to disk. 3a. "option" -> "parameter" (2x) 3b. "User can" -> "The user can" 3c. I think this part should also mention that the stream parameter default is unchanged... ====== 4. doc/src/sgml/config.sgml + <para> + Maximum number of apply background workers per subscription. This + parameter controls the amount of parallelism of the streaming of + in-progress transactions if we set subscription option + <literal>streaming</literal> to <literal>apply</literal>. + </para> "if we set subscription option <literal>streaming</literal> to <literal>apply</literal>." -> "when subscription parameter <literal>streaming = apply</literal>. ====== 5. doc/src/sgml/config.sgml + <para> + Setting streaming mode to <literal>apply</literal> could export invalid LSN + as finish LSN of failed transaction. Changing the streaming mode and making + the same conflict writes the finish LSN of the failed transaction in the + server log if required. + </para> This text made no sense to me. Can you reword it? IIUC it means something like this: When the streaming mode is 'apply', the finish LSN of failed transactions may not be logged. In that case, it may be necessary to change the streaming mode and cause the same conflicts again so the finish LSN of the failed transaction will be written to the server log. ====== 6. doc/src/sgml/protocol.sgml Since there are protocol changes made here, shouldn’t there also be some corresponding LOGICALREP_PROTO_XXX constants and special checking added in the worker.c? ====== 7. doc/src/sgml/ref/create_subscription.sgml + for this subscription. The default value is <literal>off</literal>, + all transactions are fully decoded on the publisher and only then + sent to the subscriber as a whole. + </para> SUGGESTION The default value is off, meaning all transactions are fully decoded on the publisher and only then sent to the subscriber as a whole. ~~~ 8. doc/src/sgml/ref/create_subscription.sgml + <para> + If set to <literal>on</literal>, the changes of transaction are + written to temporary files and then applied at once after the + transaction is committed on the publisher. + </para> SUGGESTION If set to on, the incoming changes are written to a temporary file and then applied only after the transaction is committed on the publisher. ~~~ 9. doc/src/sgml/ref/create_subscription.sgml + <para> + If set to <literal>apply</literal> incoming + changes are directly applied via one of the background workers, if + available. If no background worker is free to handle streaming + transaction then the changes are written to a file and applied after + the transaction is committed. Note that if an error happens when + applying changes in a background worker, it might not report the + finish LSN of the remote transaction in the server log. </para> SUGGESTION If set to apply, the incoming changes are directly applied via one of the apply background workers, if available. If no background worker is free to handle streaming transactions then the changes are written to a file and applied after the transaction is committed. Note that if an error happens when applying changes in a background worker, the finish LSN of the remote transaction might not be reported in the server log. ====== 10. src/backend/access/transam/xact.c @@ -1741,6 +1742,13 @@ RecordTransactionAbort(bool isSubXact) elog(PANIC, "cannot abort transaction %u, it was already committed", xid); + /* + * Are we using the replication origins feature? Or, in other words, + * are we replaying remote actions? + */ + replorigin = (replorigin_session_origin != InvalidRepOriginId && + replorigin_session_origin != DoNotReplicateId); + /* Fetch the data we need for the abort record */ nrels = smgrGetPendingDeletes(false, &rels); nchildren = xactGetCommittedChildren(&children); @@ -1765,6 +1773,11 @@ RecordTransactionAbort(bool isSubXact) MyXactFlags, InvalidTransactionId, NULL); + if (replorigin) + /* Move LSNs forward for this replication origin */ + replorigin_session_advance(replorigin_session_origin_lsn, + XactLastRecEnd); + I did not see any reason why the code assigning the 'replorigin' and the code checking the 'replorigin' are separated like they are. I thought these 2 new code fragments should be kept together. Perhaps it was decided this assignment must be outside the critical section? But if that’s the case maybe a comment explaining so would be good. ~~~ 11. src/backend/access/transam/xact.c + if (replorigin) + /* Move LSNs forward for this replication origin */ + replorigin_session_advance(replorigin_session_origin_lsn, + The positioning of that comment is unusual. Maybe better before the check? ====== 12. src/backend/commands/subscriptioncmds.c - defGetStreamingMode + /* + * If no parameter given, assume "true" is meant. + */ + if (def->arg == NULL) + return SUBSTREAM_ON; SUGGESTION for comment If the streaming parameter is given but no parameter value is specified, then assume "true" is meant. ~~~ 13. src/backend/commands/subscriptioncmds.c - defGetStreamingMode + /* + * Allow 0, 1, "true", "false", "on", "off" or "apply". + */ IMO these should be in a order consistent with the code. SUGGESTION Allow 0, 1, “false”, "true", “off”, "on", or "apply". ====== 14. src/backend/replication/logical/Makefile - worker.o + worker.o \ + applybgwroker.o typo "applybgwroker" -> "applybgworker" ====== 15. .../replication/logical/applybgwroker.c +/*------------------------------------------------------------------------- + * applybgwroker.c + * Support routines for applying xact by apply background worker + * + * Copyright (c) 2016-2022, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/replication/logical/applybgwroker.c 15a. Typo in filename: "applybgwroker" -> "applybgworker" 15b. Typo in file header comment: "applybgwroker" -> "applybgworker" ~~~ 16. .../replication/logical/applybgwroker.c +/* + * entry for a hash table we use to map from xid to our apply background worker + * state. + */ +typedef struct ApplyBgworkerEntry Comment should start uppercase. ~~~ 17. .../replication/logical/applybgwroker.c +/* + * Fields to record the share informations between main apply worker and apply + * background worker. + */ SUGGESTION Information shared between main apply worker and apply background worker. ~~~ 18. .../replication/logical/applybgwroker.c +/* apply background worker setup */ +static ApplyBgworkerState *apply_bgworker_setup(void); +static void apply_bgworker_setup_dsm(ApplyBgworkerState *wstate); IMO there was not really any need for this comment – these are just function forward declares. ~~~ 19. .../replication/logical/applybgwroker.c - find_or_start_apply_bgworker + if (found) + { + entry->wstate->pstate->status = APPLY_BGWORKER_BUSY; + return entry->wstate; + } + else if (!start) + return NULL; I felt this might be more readable without the else: if (found) { entry->wstate->pstate->status = APPLY_BGWORKER_BUSY; return entry->wstate; } Assert(!found) if (!start) return NULL; ~~~ 20. .../replication/logical/applybgwroker.c - find_or_start_apply_bgworker + /* + * Now, we try to get a apply background worker. If there is at least one + * worker in the idle list, then take one. Otherwise, we try to start a + * new apply background worker. + */ 20a. "a apply" -> "an apply" 20b. IMO it's better to call this the free list (not the idle list) ~~~ 21. .../replication/logical/applybgwroker.c - find_or_start_apply_bgworker + /* + * If the apply background worker cannot be launched, remove entry + * in hash table. + */ "remove entry in hash table" -> "remove the entry from the hash table" ~~~ 22. .../replication/logical/applybgwroker.c - apply_bgworker_free +/* + * Add the worker to the free list and remove the entry from hash table. + */ "from hash table" -> "from the hash table" ~~~ 23. .../replication/logical/applybgwroker.c - apply_bgworker_free + elog(DEBUG1, "adding finished apply worker #%u for xid %u to the idle list", + wstate->pstate->n, wstate->pstate->stream_xid); IMO it's better to call this the free list (not the idle list) ~~~ 24. .../replication/logical/applybgwroker.c - LogicalApplyBgwLoop +/* Apply Background Worker main loop */ +static void +LogicalApplyBgwLoop(shm_mq_handle *mqh, volatile ApplyBgworkerShared *pst) Why is the name incosistent with other function names in the file? Should it be apply_bgworker_loop? ~~~ 25. .../replication/logical/applybgwroker.c - LogicalApplyBgwLoop + /* + * Push apply error context callback. Fields will be filled during + * applying a change. + */ "during" -> "when" ~~~ 26. .../replication/logical/applybgwroker.c - LogicalApplyBgwLoop + /* + * We use first byte of message for additional communication between + * main Logical replication worker and apply bgworkers, so if it + * differs from 'w', then process it first. + */ "bgworkers" -> "background workers" ~~~ 27. .../replication/logical/applybgwroker.c - ApplyBgwShutdown For consistency should it be called apply_bgworker_shutdown? ~~~ 28. .../replication/logical/applybgwroker.c - LogicalApplyBgwMain For consistency should it be called apply_bgworker_main? ~~~ 29. .../replication/logical/applybgwroker.c - apply_bgworker_check_status + errdetail("Cannot handle streamed replication transaction by apply " + "bgworkers until all tables are synchronized"))); "bgworkers" -> "background workers" ====== 30. src/backend/replication/logical/decode.c @@ -651,9 +651,10 @@ DecodeCommit(LogicalDecodingContext *ctx, XLogRecordBuffer *buf, { for (i = 0; i < parsed->nsubxacts; i++) { - ReorderBufferForget(ctx->reorder, parsed->subxacts[i], buf->origptr); + ReorderBufferForget(ctx->reorder, parsed->subxacts[i], buf->origptr, + commit_time); } - ReorderBufferForget(ctx->reorder, xid, buf->origptr); + ReorderBufferForget(ctx->reorder, xid, buf->origptr, commit_time); ReorderBufferForget was declared with 'abort_time' param. So it makes these calls a bit confusing looking to be passing 'commit_time' Maybe better to do like below and pass 'forget_time' (inside that 'if') along with an explanatory comment: TimestampTz forget_time = commit_time; ====== 31. src/backend/replication/logical/launcher.c - logicalrep_worker_find + /* We only need main apply worker or table sync worker here */ "need" -> "are interested in the" ~~~ 32. src/backend/replication/logical/launcher.c - logicalrep_worker_launch + if (!is_subworker) + snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyWorkerMain"); + else + snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyBgworkerMain"); IMO better to reverse this and express the condition as 'if (is_subworker)' ~~~ 33. src/backend/replication/logical/launcher.c - logicalrep_worker_launch + else if (!is_subworker) snprintf(bgw.bgw_name, BGW_MAXLEN, "logical replication worker for subscription %u", subid); + else + snprintf(bgw.bgw_name, BGW_MAXLEN, + "logical replication background apply worker for subscription %u ", subid); 33a. Ditto. IMO better to reverse this and express the condition as 'if (is_subworker)' 33b. "background apply worker" -> "apply background worker" ~~~ 34. src/backend/replication/logical/launcher.c - logicalrep_worker_stop IMO this code logic should be rewritten to be simpler to have a common LWLockRelease. This also makes the code more like logicalrep_worker_detach, which seems like a good thing. SUGGESTION logicalrep_worker_stop(Oid subid, Oid relid) { LogicalRepWorker *worker; LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); worker = logicalrep_worker_find(subid, relid, false); if (worker) logicalrep_worker_stop_internal(worker); LWLockRelease(LogicalRepWorkerLock); } ~~~ 35. src/backend/replication/logical/launcher.c - logicalrep_apply_background_worker_count +/* + * Count the number of registered (not necessarily running) apply background + * worker for a subscription. + */ "worker" -> "workers" ~~~ 36. src/backend/replication/logical/launcher.c - logicalrep_apply_background_worker_count + int res = 0; + A better variable name here would be 'count', or even 'n'. ====== 36. src/backend/replication/logical/origin.c + * However, If must_acquire is false, we allow process to get the slot which is + * already acquired by other process. SUGGESTION However, if the function parameter 'must_acquire' is false, we allow the process to use the same slot already acquired by another process. ~~~ 37. src/backend/replication/logical/origin.c + ereport(ERROR, + (errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED), + errmsg("could not find correct replication state slot for replication origin with OID %u for apply background worker", + node), + errhint("There is no replication state slot set by its main apply worker."))); 37a. Somehow, I felt the errmsg and the errhint could be clearer. Maybe like this? " apply background worker could not find replication state slot for replication origin with OID %u", "There is no replication state slot set by the main apply worker." 37b. Also, I think thet generally the 'errhint' informs some advice or some action that the user can take to fix the problem. But is this errhint actually saying anything useful for the user? Perhaps you meant to say 'errdetail' here? ====== 38. src/backend/replication/logical/proto.c - logicalrep_read_stream_abort + /* + * If the version of the publisher is lower than the version of the + * subscriber, it may not support sending these two fields, so only take + * these fields when include_abort_lsn is true. + */ + if (include_abort_lsn) + { + abort_data->abort_lsn = pq_getmsgint64(in); + abort_data->abort_time = pq_getmsgint64(in); + } + else + { + abort_data->abort_lsn = InvalidXLogRecPtr; + abort_data->abort_time = 0; + } This comment is documenting a decision that was made elsewhere. But it somehow feels wrong to me that the decision to read or not read the abort time/lsn is made by the caller of this function. IMO it might make more sense if the server version was simply passed as a param and then this function can be in control of its own destiny and make the decision does it need to read those extra fields or not. An extra member flag can be added to LogicalRepStreamAbortData to indicate if abort_data read these values or not. ====== 39. src/backend/replication/logical/worker.c * Streamed transactions (large transactions exceeding a memory limit on the - * upstream) are not applied immediately, but instead, the data is written - * to temporary files and then applied at once when the final commit arrives. + * upstream) are applied via one of two approaches. "via" -> "using" ~~~ 40. src/backend/replication/logical/worker.c + * Assign a new apply background worker (if available) as soon as the xact's + * first stream is received and the main apply worker will send changes to this + * new worker via shared memory. We keep this worker assigned till the + * transaction commit is received and also wait for the worker to finish at + * commit. This preserves commit ordering and avoids writing to and reading + * from file in most cases. We still need to spill if there is no worker + * available. We also need to allow stream_stop to complete by the background + * worker to avoid deadlocks because T-1's current stream of changes can update + * rows in conflicting order with T-2's next stream of changes. 40a. "and the main apply -> ". The main apply" 40b. "and avoids writing to and reading from file in most cases." -> "and avoids file I/O in most cases." 40c. "We still need to spill if" -> "We still need to spill to a file if" 40d. "We also need to allow stream_stop to complete by the background worker" -> "We also need to allow stream_stop to be performed by the background worker" ~~~ 41. src/backend/replication/logical/worker.c -static ApplyErrorCallbackArg apply_error_callback_arg = +ApplyErrorCallbackArg apply_error_callback_arg = { .command = 0, .rel = NULL, @@ -242,7 +246,7 @@ static ApplyErrorCallbackArg apply_error_callback_arg = .origin_name = NULL, }; Maybe it is still a good idea to at least keep the old comment here: /* Struct for saving and restoring apply errcontext information */ ~~ 42. src/backend/replication/logical/worker.c +/* check if we are applying the transaction in apply background worker */ +#define apply_bgworker_active() (in_streamed_transaction && stream_apply_worker != NULL) 42a. Uppercase comment. 42b. "in apply background worker" -> "in apply background worker" ~~~ 43. src/backend/replication/logical/worker.c - handle_streamed_transaction @@ -426,41 +437,76 @@ end_replication_step(void) } /* - * Handle streamed transactions. + * Handle streamed transactions for both main apply worker and apply background + * worker. * - * If in streaming mode (receiving a block of streamed transaction), we - * simply redirect it to a file for the proper toplevel transaction. + * In streaming case (receiving a block of streamed transaction), for + * SUBSTREAM_ON mode, we simply redirect it to a file for the proper toplevel + * transaction, and for SUBSTREAM_APPLY mode, we send the changes to background + * apply worker (LOGICAL_REP_MSG_RELATION or LOGICAL_REP_MSG_TYPE changes will + * also be applied in main apply worker). * - * Returns true for streamed transactions, false otherwise (regular mode). + * For non-streamed transactions, returns false; + * For streamed transactions, returns true if in main apply worker (except we + * apply streamed transaction in "apply" mode and address + * LOGICAL_REP_MSG_RELATION or LOGICAL_REP_MSG_TYPE changes), false otherwise. */ Maybe it is accurate (I don’t know), but this header comment seems excessively complicated with so many quirks about when to return true/false. Can it be reworded into plainer language? ~~~ 44. src/backend/replication/logical/worker.c - handle_streamed_transaction Because there are so many returns for each of these conditions, consider refactoring the logic to change all the if/else to just be "if" and then you can comment each separate cases better. I think it may be clearer. SUGGESTION /* This is the apply background worker */ if (am_apply_bgworker()) { ... return false; } /* This is the main apply, but there is an apply background worker */ if (apply_bgworker_active()) { ... return true; } /* This is the main apply, and there is no apply background worker */ ... return true; ~~~ 45. src/backend/replication/logical/worker.c - apply_handle_stream_prepare + /* + * This is the main apply worker. Check if we are processing this + * transaction in a apply background worker. + */ + if (wstate) I think the part that says "This is the main apply worker" should be at the top of the 'else' ~~~ 46. src/backend/replication/logical/worker.c - apply_handle_stream_prepare + /* + * This is the main apply worker and the transaction has been + * serialized to file, replay all the spooled operations. + */ SUGGESTION The transaction has been serialized to file. Replay all the spooled operations. ~~~ 47. src/backend/replication/logical/worker.c - apply_handle_stream_prepare + /* unlink the files with serialized changes and subxact info. */ + stream_cleanup_files(MyLogicalRepWorker->subid, prepare_data.xid); Start comment with capital letter. ~~~ 48. src/backend/replication/logical/worker.c - apply_handle_stream_start + /* If we are in a apply background worker, begin the transaction */ + AcceptInvalidationMessages(); + maybe_reread_subscription(); The "if we are" part of the comment is not needed because the fact the code is inside am_apply_bgworker() makes this obvious anyway/ ~~~ 49. src/backend/replication/logical/worker.c - apply_handle_stream_start + /* open the spool file for this transaction */ + stream_open_file(MyLogicalRepWorker->subid, stream_xid, first_segment); + Start the comment uppercase. + /* if this is not the first segment, open existing subxact file */ + if (!first_segment) + subxact_info_read(MyLogicalRepWorker->subid, stream_xid); Start the comment uppercase. ~~~ 50. src/backend/replication/logical/worker.c - apply_handle_stream_abort + /* Check whether the publisher sends abort_lsn and abort_time. */ + if (am_apply_bgworker()) + include_abort_lsn = MyParallelState->server_version >= 150000; + + logicalrep_read_stream_abort(s, &abort_data, include_abort_lsn); Here is where I felt maybe just the server version could be passed so the logicalrep_read_stream_abort could decide itself what message parts needed to be read. Basically it seems strange that the message contain parts which might not be read. I felt it is better to always read the whole message then later you can choose what parts you are interested in. ~~~ 51. src/backend/replication/logical/worker.c - apply_handle_stream_abort + /* + * This is the main apply worker. Check if we are processing this + * transaction in a apply background worker. + */ + /* + * We are in main apply worker and the transaction has been serialized + * to file. + */ 51a. I thought the "This is the main apply worker" and "We are in main apply worker" should just be be a comment top of this "else" 51b. "a apply worker" -> "an apply worker" 51c. There seemed to be some missing comment to say this logic is telling the bgworker to abort and then waiting for it to do so. ~~~ 52. src/backend/replication/logical/worker.c - apply_handle_stream_commit I did not really understand why the patch relocates this function to another place in the file. Can't it be left in the same place? ~~~ 53. src/backend/replication/logical/worker.c - apply_handle_stream_commit + /* + * This is the main apply worker. Check if we are processing this + * transaction in an apply background worker. + */ I thought the top of the else should just say "This is the main apply worker." Then the if (wstate) part should say “Check if we are processing this transaction in an apply background worker, and if so tell it to comment the message”/ ~~~ 54. src/backend/replication/logical/worker.c - apply_handle_stream_commit + /* + * This is the main apply worker and the transaction has been + * serialized to file, replay all the spooled operations. + */ SUGGESTION The transaction has been serialized to file, so replay all the spooled operations. ~~~ 55. src/backend/replication/logical/worker.c - apply_handle_stream_commit + /* unlink the files with serialized changes and subxact info */ + stream_cleanup_files(MyLogicalRepWorker->subid, xid); Uppercase comment. ====== 56. src/backend/utils/misc/guc.c @@ -3220,6 +3220,18 @@ static struct config_int ConfigureNamesInt[] = NULL, NULL, NULL }, + { + {"max_apply_bgworkers_per_subscription", + PGC_SIGHUP, + REPLICATION_SUBSCRIBERS, + gettext_noop("Maximum number of apply backgrand workers per subscription."), + NULL, + }, + &max_apply_bgworkers_per_subscription, + 3, 0, MAX_BACKENDS, + NULL, NULL, NULL + }, + "backgrand" -> "background" ====== 57. src/include/catalog/pg_subscription.h @@ -109,7 +110,7 @@ typedef struct Subscription bool enabled; /* Indicates if the subscription is enabled */ bool binary; /* Indicates if the subscription wants data in * binary format */ - bool stream; /* Allow streaming in-progress transactions. */ + char stream; /* Allow streaming in-progress transactions. */ char twophasestate; /* Allow streaming two-phase transactions */ bool disableonerr; /* Indicates if the subscription should be * automatically disabled if a worker error I felt probably this 'stream' comment should be the same as for 'substream'. ====== 58. src/include/replication/worker_internal.h +/* + * Shared information among apply workers. + */ +typedef struct ApplyBgworkerShared SUGGESTION (maybe you can do better than this) Struct for sharing information between apply main and apply background workers. ~~~ 59. src/include/replication/worker_internal.h + /* Status for apply background worker. */ + ApplyBgworkerStatus status; "Status for" -> "Status of" ~~~ 60. src/include/replication/worker_internal.h +extern PGDLLIMPORT MemoryContext ApplyMessageContext; + +extern PGDLLIMPORT ApplyErrorCallbackArg apply_error_callback_arg; + +extern PGDLLIMPORT bool MySubscriptionValid; + +extern PGDLLIMPORT volatile ApplyBgworkerShared *MyParallelState; +extern PGDLLIMPORT List *subxactlist; + I did not recognise the significance why are the last 2 externs grouped togeth but the others are not. ~~~ 61. src/include/replication/worker_internal.h +/* prototype needed because of stream_commit */ +extern void apply_dispatch(StringInfo s); 61a. I was unsure if this comment is useful to anyone... 61b. If you decide to keep it, please use uppercase. ~~~ 62. src/include/replication/worker_internal.h +/* apply background worker setup and interactions */ +extern ApplyBgworkerState *apply_bgworker_find_or_start(TransactionId xid, + bool start); Uppercase comment. ====== 63. I also did a quick check of all the new debug logging added. Here is everyhing from patch v11-0001. apply_bgworker_free: + elog(DEBUG1, "adding finished apply worker #%u for xid %u to the idle list", + wstate->pstate->n, wstate->pstate->stream_xid); LogicalApplyBgwLoop: + elog(DEBUG1, "[Apply BGW #%u] ended processing streaming chunk," + "waiting on shm_mq_receive", pst->n); + elog(DEBUG1, "[Apply BGW #%u] exiting", pst->n); ApplyBgworkerMain: + elog(DEBUG1, "[Apply BGW #%u] started", pst->n); apply_bgworker_setup: + elog(DEBUG1, "setting up apply worker #%u", list_length(ApplyWorkersList) + 1); apply_bgworker_set_status: + elog(DEBUG1, "[Apply BGW #%u] set status to %d", MyParallelState->n, status); apply_bgworker_subxact_info_add: + elog(DEBUG1, "[Apply BGW #%u] defining savepoint %s", + MyParallelState->n, spname); apply_handle_stream_prepare: + elog(DEBUG1, "received prepare for streamed transaction %u", + prepare_data.xid); apply_handle_stream_start: + elog(DEBUG1, "starting streaming of xid %u", stream_xid); apply_handle_stream_stop: + elog(DEBUG1, "stopped streaming of xid %u, %u changes streamed", stream_xid, nchanges); apply_handle_stream_abort: + elog(DEBUG1, "[Apply BGW #%u] aborting current transaction xid=%u, subxid=%u", + MyParallelState->n, GetCurrentTransactionIdIfAny(), + GetCurrentSubTransactionId()); + elog(DEBUG1, "[Apply BGW #%u] rolling back to savepoint %s", + MyParallelState->n, spname); apply_handle_stream_commit: + elog(DEBUG1, "received commit for streamed transaction %u", xid); Observations: 63a. Every new introduced message is at level DEBUG1 (not DEBUG). AFAIK this is OK, because the messages are all protocol related and every other existing debug message of the current replication worker.c was also at the same DEBUG1 level. 63b. The prefix "[Apply BGW #%u]" is used to indicate the bgworker is executing the code, but it does not seem to be used 100% consistently - e.g. there are some apply_bgworker_XXX functions not using this prefix. Is that OK or a mistake? ------ Kind Regards, Peter Smith. Fujitsu Austrlia ^ permalink raw reply [nested|flat] 66+ messages in thread
* Re: Perform streaming logical transactions by background workers and parallel apply @ 2022-06-21 04:24 Amit Kapila <[email protected]> parent: Peter Smith <[email protected]> 1 sibling, 0 replies; 66+ messages in thread From: Amit Kapila @ 2022-06-21 04:24 UTC (permalink / raw) To: Peter Smith <[email protected]>; +Cc: [email protected] <[email protected]>; Masahiko Sawada <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; PostgreSQL Hackers <[email protected]> On Tue, Jun 21, 2022 at 7:11 AM Peter Smith <[email protected]> wrote: > > Here are some review comments for the v11-0001 patch. > > (I will review the remaining patches 0002-0005 and post any comments later) > > ====== > > 1. General > > I still feel that 'apply' seems like a meaningless enum value for this > feature because from a user point-of-view every replicated change gets > "applied". IMO something like 'streaming = parallel' or 'streaming = > background' (etc) might have more meaning for a user. > +1. I would prefer 'streaming = parallel' as that suits here because we allow streams (set of changes) of a transaction to be applied in parallel to other transactions or in parallel to a stream of changes from another streaming transaction. > ====== > > 10. src/backend/access/transam/xact.c > > @@ -1741,6 +1742,13 @@ RecordTransactionAbort(bool isSubXact) > elog(PANIC, "cannot abort transaction %u, it was already committed", > xid); > > + /* > + * Are we using the replication origins feature? Or, in other words, > + * are we replaying remote actions? > + */ > + replorigin = (replorigin_session_origin != InvalidRepOriginId && > + replorigin_session_origin != DoNotReplicateId); > + > /* Fetch the data we need for the abort record */ > nrels = smgrGetPendingDeletes(false, &rels); > nchildren = xactGetCommittedChildren(&children); > @@ -1765,6 +1773,11 @@ RecordTransactionAbort(bool isSubXact) > MyXactFlags, InvalidTransactionId, > NULL); > > + if (replorigin) > + /* Move LSNs forward for this replication origin */ > + replorigin_session_advance(replorigin_session_origin_lsn, > + XactLastRecEnd); > + > > I did not see any reason why the code assigning the 'replorigin' and > the code checking the 'replorigin' are separated like they are. I > thought these 2 new code fragments should be kept together. Perhaps it > was decided this assignment must be outside the critical section? But > if that’s the case maybe a comment explaining so would be good. > I also don't see any particular reason for this apart from being similar to RecordTransactionCommit(). I think it should be fine either way. > ~~~ > > 11. src/backend/access/transam/xact.c > > + if (replorigin) > + /* Move LSNs forward for this replication origin */ > + replorigin_session_advance(replorigin_session_origin_lsn, > + > > The positioning of that comment is unusual. Maybe better before the check? > This again seems to be due to a similar code in RecordTransactionCommit(). I would suggest let's keep the code consistent. -- With Regards, Amit Kapila. ^ permalink raw reply [nested|flat] 66+ messages in thread
* Re: Perform streaming logical transactions by background workers and parallel apply @ 2022-06-23 02:47 Peter Smith <[email protected]> parent: [email protected] <[email protected]> 3 siblings, 0 replies; 66+ messages in thread From: Peter Smith @ 2022-06-23 02:47 UTC (permalink / raw) To: [email protected] <[email protected]>; +Cc: Amit Kapila <[email protected]>; Masahiko Sawada <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; PostgreSQL Hackers <[email protected]> FYI - the latest patch set v12* on this thread no longer applies. [postgres@CentOS7-x64 oss_postgres_misc]$ git apply v12-0003-A-temporary-patch-that-includes-patch-in-another.patch error: patch failed: src/backend/replication/logical/relation.c:307 error: src/backend/replication/logical/relation.c: patch does not apply error: patch failed: src/backend/replication/logical/worker.c:2358 error: src/backend/replication/logical/worker.c: patch does not apply error: patch failed: src/test/subscription/t/013_partition.pl:868 error: src/test/subscription/t/013_partition.pl: patch does not apply [postgres@CentOS7-x64 oss_postgres_misc]$ ~~ I know the v12-0003 was meant just a temporary patch for something that may now already be pushed, but it cannot be just skipped either because then v12-0004 will also fail. [postgres@CentOS7-x64 oss_postgres_misc]$ git apply v12-0004-Add-some-checks-before-using-apply-background-wo.patch error: patch failed: src/backend/replication/logical/relation.c:433 error: src/backend/replication/logical/relation.c: patch does not apply error: patch failed: src/backend/replication/logical/worker.c:2403 error: src/backend/replication/logical/worker.c: patch does not apply [postgres@CentOS7-x64 oss_postgres_misc]$ ------ Kind Regards, Peter Smith. Fujitsu Australia ^ permalink raw reply [nested|flat] 66+ messages in thread
* Re: Perform streaming logical transactions by background workers and parallel apply @ 2022-06-23 06:50 Peter Smith <[email protected]> parent: [email protected] <[email protected]> 3 siblings, 1 reply; 66+ messages in thread From: Peter Smith @ 2022-06-23 06:50 UTC (permalink / raw) To: [email protected] <[email protected]>; +Cc: Amit Kapila <[email protected]>; Masahiko Sawada <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; PostgreSQL Hackers <[email protected]> Here are some review comments for v12-0002 ====== 1. Commit message "streaming" option -> "streaming" parameter ~~~ 2. General (every file in this patch) "streaming" option -> "streaming" parameter ~~~ 3. .../subscription/t/022_twophase_cascade.pl For every test file in this patch the new function is passed $is_apply = 0/1 to indicate to use 'on' or 'apply' parameter value. But in this test file the parameter is passed as $streaming_mode = 'on'/'apply'. I was wondering if (for the sake of consistency) it might be better to use the same parameter kind for all of the test files. Actually, I don't care if you choose to do nothing and leave this as-is; I am just posting this review comment in case it was not a deliberate decision to implement them differently. e.g. + my ($node_publisher, $node_subscriber, $appname, $is_apply) = @_; versus + my ($node_A, $node_B, $node_C, $appname_B, $appname_C, $streaming_mode) = + @_; ------ Kind Regards, Peter Smith. Fujitsu Australia ^ permalink raw reply [nested|flat] 66+ messages in thread
* RE: Perform streaming logical transactions by background workers and parallel apply @ 2022-06-23 07:21 [email protected] <[email protected]> parent: Amit Kapila <[email protected]> 0 siblings, 1 reply; 66+ messages in thread From: [email protected] @ 2022-06-23 07:21 UTC (permalink / raw) To: Amit Kapila <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Peter Smith <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; PostgreSQL Hackers <[email protected]> On Mon, Jun 20, 2022 at 11:00 AM Amit Kapila <[email protected]> wrote: > I have improved the comments in this and other related sections of the > patch. See attached. Thanks for your comments and patch! Improved the comments as you suggested. > > > 3. > > > + > > > + <para> > > > + Setting streaming mode to <literal>apply</literal> could export invalid > LSN > > > + as finish LSN of failed transaction. Changing the streaming mode and > making > > > + the same conflict writes the finish LSN of the failed transaction in the > > > + server log if required. > > > + </para> > > > > > > How will the user identify that this is an invalid LSN value and she > > > shouldn't use it to SKIP the transaction? Can we change the second > > > sentence to: "User should change the streaming mode to 'on' if they > > > would instead wish to see the finish LSN on error. Users can use > > > finish LSN to SKIP applying the transaction." I think we can give > > > reference to docs where the SKIP feature is explained. > > Improved the sentence as suggested. > > > > You haven't answered first part of the comment: "How will the user > identify that this is an invalid LSN value and she shouldn't use it to > SKIP the transaction?". Have you checked what value it displays? For > example, in one of the case in apply_error_callback as shown in below > code, we don't even display finish LSN if it is invalid. > else if (XLogRecPtrIsInvalid(errarg->finish_lsn)) > errcontext("processing remote data for replication origin \"%s\" > during \"%s\" in transaction %u", > errarg->origin_name, > logicalrep_message_type(errarg->command), > errarg->remote_xid); I am sorry that I missed something in my previous reply. The invalid LSN value here is to say InvalidXLogRecPtr (0/0). Here is an example : ``` 2022-06-23 14:30:11.343 CST [822333] logical replication worker CONTEXT: processing remote data for replication origin "pg_16389" during "INSERT" for replication target relation "public.tab" in transaction 727 finished at 0/0 ``` So I try to improve the sentence in pg-doc by changing from ``` Setting streaming mode to <literal>apply</literal> could export invalid LSN as finish LSN of failed transaction. ``` to ``` Setting streaming mode to <literal>apply</literal> could export invalid LSN (0/0) as finish LSN of failed transaction. ``` I also improved the patches as you suggested in [1]: > 1. > +/* > + * Count the number of registered (not necessarily running) apply background > + * worker for a subscription. > + */ > > /worker/workers Improved as suggested. > 2. > +static void > +apply_bgworker_setup_dsm(ApplyBgworkerState *wstate) > +{ > ... > ... > + int64 queue_size = 160000000; /* 16 MB for now */ > > I think it would be better to use define for this rather than a > hard-coded value. Improved as suggested. Added a macro like this: ``` /* queue size of DSM, 16 MB for now. */ #define DSM_QUEUE_SIZE 160000000 ``` > 3. > +/* > + * Status for apply background worker. > + */ > +typedef enum ApplyBgworkerStatus > +{ > + APPLY_BGWORKER_ATTACHED = 0, > + APPLY_BGWORKER_READY, > + APPLY_BGWORKER_BUSY, > + APPLY_BGWORKER_FINISHED, > + APPLY_BGWORKER_EXIT > +} ApplyBgworkerStatus; > > It would be better if you can add comments to explain each of these states. Improved as suggested. Added the comments like below: ``` APPLY_BGWORKER_BUSY = 0, /* assigned to a transaction */ APPLY_BGWORKER_FINISHED, /* transaction is completed */ APPLY_BGWORKER_EXIT /* exit */ ``` In addition, after improving the point #7 as you suggested, I removed "APPLY_BGWORKER_ATTACHED". And I removed "APPLY_BGWORKER_READY" in v12. > 4. > + /* Set up one message queue per worker, plus one. */ > + mq = shm_mq_create(shm_toc_allocate(toc, (Size) queue_size), > + (Size) queue_size); > + shm_toc_insert(toc, APPLY_BGWORKER_KEY_MQ, mq); > + shm_mq_set_sender(mq, MyProc); > > > I don't understand the meaning of 'plus one' in the above comment as > the patch seems to be setting up just one queue here? Yes, you are right. Improved as below: ``` /* Set up message queue for the worker. */ ``` > 5. > + > + /* Attach the queues. */ > + wstate->mq_handle = shm_mq_attach(mq, seg, NULL); > > Similar to above. If there is only one queue then the comment should > say queue instead of queues. Improved as suggested. > 6. > snprintf(bgw.bgw_name, BGW_MAXLEN, > "logical replication worker for subscription %u", subid); > + else > + snprintf(bgw.bgw_name, BGW_MAXLEN, > + "logical replication background apply worker for subscription %u ", subid); > > No need for extra space after %u in the above code. Improved as suggested. > 7. > + launched = logicalrep_worker_launch(MyLogicalRepWorker->dbid, > + MySubscription->oid, > + MySubscription->name, > + MyLogicalRepWorker->userid, > + InvalidOid, > + dsm_segment_handle(wstate->dsm_seg)); > + > + if (launched) > + { > + /* Wait for worker to attach. */ > + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_ATTACHED); > > In logicalrep_worker_launch(), we already seem to be waiting for > workers to attach via WaitForReplicationWorkerAttach(), so it is not > clear to me why we need to wait again? If there is a genuine reason > then it is better to add some comments to explain it. I think in some > way, we need to know if the worker is successfully attached and we may > not get that via WaitForReplicationWorkerAttach, so there needs to be > some way to know that but this doesn't sound like a very good idea. If > that understanding is correct then can we think of a better way? Improved the related logic. The reason we wait again here in previous version is to wait for apply bgworker to attach the memory queue, but function WaitForReplicationWorkerAttach could not do that. Now to improve this, we invoke the function logicalrep_worker_attach after the attaching the memory queue instead of before. Also to make sure worker has not die due to error or some reasons, I modified the function logicalrep_worker_launch and function WaitForReplicationWorkerAttach. And then, we could judge whether the worker started successfully or died according to the return value of the function logicalrep_worker_launch. > 8. I think we can simplify apply_bgworker_find_or_start by having > separate APIs for find and start. Most of the places need to use find > API except for the first stream. If we do that then I think you don't > need to make a hash entry unless we established ApplyBgworkerState > which currently looks odd as you need to remove the entry if we fail > to allocate the state. Improved as suggested. > 9. > + /* > + * TO IMPROVE: Do we need to display the apply background worker's > + * information in pg_stat_replication ? > + */ > + UpdateWorkerStats(last_received, send_time, false); > > In this do you mean to say pg_stat_subscription? If so, then to decide > whether we need to update stats here we should see what additional > information we can update here which is not possible via the main > apply worker? Yes, it should be pg_stat_subscription. I think we do not need to update these statistics here. I think the messages received in function LogicalApplyBgwLoop in apply bgworker have handled in function LogicalRepApplyLoop in apply worker, these statistics have been updated. (see function LogicalRepApplyLoop) > 10. > ApplyBgworkerMain > { > ... > + /* Load the subscription into persistent memory context. */ > + ApplyContext = AllocSetContextCreate(TopMemoryContext, > ... > > This comment seems to be copied from ApplyWorkerMain but doesn't apply > here. Yes, you are right. Improved as below: ``` /* Init the memory context for the apply background worker to work in. */ ``` In addition, I also tried to improve the patches by following points: a. In the function apply_handle_stream_abort, when invoking the function set_apply_error_context_xact, I forgot to change the second input parameter. So changed "InvalidXLogRecPtr" to "abort_lsn". b. Improved the function name from "canstartapplybgworker" to "apply_bgworker_can_start". c. Detach the dsm segment if we fail to launch a apply bgworker. (see function apply_bgworker_setup) BTW, I deleted the temporary patch 0003 (v12) and rebased patches because the commit 26b3455afa and ac0e2d387a in HEAD. And now, I am improving the patches as suggested by Peter-san in [3]. I will send new patches soon. Attach the new patches. [1] - https://www.postgresql.org/message-id/CAA4eK1%2BQQHGb0afmM_Cf2qu%3DUJoCnvs3VcZ%2B1xTiySx205fU1w%40ma... [2] - https://www.postgresql.org/message-id/OS3PR01MB6275208A2F8ED832710F65E09EA49%40OS3PR01MB6275.jpnprd0... [3] - https://www.postgresql.org/message-id/CAHut%2BPtu_eWOVWAKrwkUFdTAh_r-RZsbDFkFmKwEAmxws%3DSh5w%40mail... Regards, Wang wei Attachments: [application/octet-stream] v13-0001-Perform-streaming-logical-transactions-by-backgr.patch (100.3K, ../../OS3PR01MB6275F61CC74C5F3E47C7DD229EB59@OS3PR01MB6275.jpnprd01.prod.outlook.com/2-v13-0001-Perform-streaming-logical-transactions-by-backgr.patch) download | inline diff: From 587013ddcca61c97df54239db93815689c751f90 Mon Sep 17 00:00:00 2001 From: "houzj.fnst" <[email protected]> Date: Wed, 20 Apr 2022 16:45:07 +0800 Subject: [PATCH v13 1/4] Perform streaming logical transactions by background workers Currently, for large transactions, the publisher sends the data in multiple streams (changes divided into chunks depending upon logical_decoding_work_mem), and then on the subscriber-side, the apply worker writes the changes into temporary files and once it receives the commit, it reads from the file and applies the entire transaction. To improve the performance of such transactions, we can instead allow them to be applied via background workers. In this approach, we assign a new apply background worker (if available) as soon as the xact's first stream is received and the main apply worker will send changes to this new worker via shared memory. The apply background worker will directly apply the change instead of writing it to temporary files. We keep this worker assigned till the transaction commit is received and also wait for the worker to finish at commit. This preserves commit ordering and avoids writing to and reading from file in most cases. We still need to spill if there is no worker available. This patch also extends the SUBSCRIPTION 'streaming' option so that the user can control whether to apply the streaming transaction in an apply background worker or spill the change to disk. User can set the streaming option to 'on/off', 'apply'. For now, 'apply' means the streaming will be applied via a apply background worker if available. 'on' means the streaming transaction will be spilled to disk. --- doc/src/sgml/catalogs.sgml | 10 +- doc/src/sgml/config.sgml | 25 + doc/src/sgml/logical-replication.sgml | 9 + doc/src/sgml/protocol.sgml | 19 + doc/src/sgml/ref/create_subscription.sgml | 24 +- src/backend/access/transam/xact.c | 13 + src/backend/commands/subscriptioncmds.c | 66 +- src/backend/postmaster/bgworker.c | 3 + src/backend/replication/logical/Makefile | 3 +- .../replication/logical/applybgwroker.c | 777 ++++++++++++++++++ src/backend/replication/logical/decode.c | 10 +- src/backend/replication/logical/launcher.c | 122 ++- src/backend/replication/logical/origin.c | 26 +- src/backend/replication/logical/proto.c | 35 +- .../replication/logical/reorderbuffer.c | 10 +- src/backend/replication/logical/tablesync.c | 10 +- src/backend/replication/logical/worker.c | 660 +++++++++++---- src/backend/replication/pgoutput/pgoutput.c | 2 +- src/backend/utils/activity/wait_event.c | 3 + src/backend/utils/misc/guc.c | 12 + src/backend/utils/misc/postgresql.conf.sample | 1 + src/bin/pg_dump/pg_dump.c | 6 +- src/include/catalog/pg_subscription.h | 17 +- src/include/replication/logicallauncher.h | 1 + src/include/replication/logicalproto.h | 19 +- src/include/replication/logicalworker.h | 1 + src/include/replication/origin.h | 2 +- src/include/replication/reorderbuffer.h | 7 +- src/include/replication/worker_internal.h | 102 ++- src/include/utils/wait_event.h | 1 + src/test/regress/expected/subscription.out | 2 +- src/tools/pgindent/typedefs.list | 5 + 32 files changed, 1776 insertions(+), 227 deletions(-) create mode 100644 src/backend/replication/logical/applybgwroker.c diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index c00c93dd7b..5cb39b18bd 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -7873,11 +7873,15 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l <row> <entry role="catalog_table_entry"><para role="column_definition"> - <structfield>substream</structfield> <type>bool</type> + <structfield>substream</structfield> <type>char</type> </para> <para> - If true, the subscription will allow streaming of in-progress - transactions + Controls how to handle the streaming of in-progress transactions: + <literal>f</literal> = disallow streaming of in-progress transactions, + <literal>t</literal> = spill the changes of in-progress transactions to + disk and apply at once after the transaction is committed on the + publisher, + <literal>a</literal> = apply changes directly using a background worker </para></entry> </row> diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml index 48478b1024..3974ba1bf6 100644 --- a/doc/src/sgml/config.sgml +++ b/doc/src/sgml/config.sgml @@ -4970,6 +4970,31 @@ ANY <replaceable class="parameter">num_sync</replaceable> ( <replaceable class=" </listitem> </varlistentry> + <varlistentry id="guc-max-apply-bgworkers-per-subscription" xreflabel="max_apply_bgworkers_per_subscription"> + <term><varname>max_apply_bgworkers_per_subscription</varname> (<type>integer</type>) + <indexterm> + <primary><varname>max_apply_bgworkers_per_subscription</varname> configuration parameter</primary> + </indexterm> + </term> + <listitem> + <para> + Maximum number of apply background workers per subscription. This + parameter controls the amount of parallelism of the streaming of + in-progress transactions if we set subscription option + <literal>streaming</literal> to <literal>apply</literal>. + </para> + <para> + The apply background workers are taken from the pool defined by + <varname>max_logical_replication_workers</varname>. + </para> + <para> + The default value is 2. This parameter can only be set in the + <filename>postgresql.conf</filename> file or on the server command + line. + </para> + </listitem> + </varlistentry> + </variablelist> </sect2> diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml index 145ea71d61..7dd78a67da 100644 --- a/doc/src/sgml/logical-replication.sgml +++ b/doc/src/sgml/logical-replication.sgml @@ -948,6 +948,15 @@ CONTEXT: processing remote data for replication origin "pg_16395" during "INSER might not violate any constraint. This can easily make the subscriber inconsistent. </para> + + <para> + Setting streaming mode to <literal>apply</literal> could export invalid LSN + (0/0) as finish LSN of failed transaction. User should change the streaming + mode to 'on' if they would instead wish to see the finish LSN on error. + Users can use finish LSN to SKIP applying the transaction by running <link + linkend="sql-altersubscription"><command>ALTER SUBSCRIPTION ... + SKIP</command></link>. + </para> </sect1> <sect1 id="logical-replication-restrictions"> diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml index a94743b587..29f3a698b7 100644 --- a/doc/src/sgml/protocol.sgml +++ b/doc/src/sgml/protocol.sgml @@ -6809,6 +6809,25 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;" </listitem> </varlistentry> + <varlistentry> + <term>Int64 (XLogRecPtr)</term> + <listitem> + <para> + The LSN of the abort. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Int64 (TimestampTz)</term> + <listitem> + <para> + Abort timestamp of the transaction. The value is in number + of microseconds since PostgreSQL epoch (2000-01-01). + </para> + </listitem> + </varlistentry> + <varlistentry> <term>Int32 (TransactionId)</term> <listitem> diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml index 35b39c28da..d4caae0222 100644 --- a/doc/src/sgml/ref/create_subscription.sgml +++ b/doc/src/sgml/ref/create_subscription.sgml @@ -217,13 +217,29 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl </varlistentry> <varlistentry> - <term><literal>streaming</literal> (<type>boolean</type>)</term> + <term><literal>streaming</literal> (<type>enum</type>)</term> <listitem> <para> Specifies whether to enable streaming of in-progress transactions - for this subscription. By default, all transactions - are fully decoded on the publisher and only then sent to the - subscriber as a whole. + for this subscription. The default value is <literal>off</literal>, + all transactions are fully decoded on the publisher and only then + sent to the subscriber as a whole. + </para> + + <para> + If set to <literal>on</literal>, the changes of transaction are + written to temporary files and then applied at once after the + transaction is committed on the publisher. + </para> + + <para> + If set to <literal>apply</literal> incoming + changes are directly applied via one of the background workers, if + available. If no background worker is free to handle streaming + transaction then the changes are written to a file and applied after + the transaction is committed. Note that if an error happens when + applying changes in a background worker, it might not report the + finish LSN of the remote transaction in the server log. </para> </listitem> </varlistentry> diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c index 47d80b0d25..678540ba25 100644 --- a/src/backend/access/transam/xact.c +++ b/src/backend/access/transam/xact.c @@ -1711,6 +1711,7 @@ RecordTransactionAbort(bool isSubXact) int nchildren; TransactionId *children; TimestampTz xact_time; + bool replorigin; /* * If we haven't been assigned an XID, nobody will care whether we aborted @@ -1741,6 +1742,13 @@ RecordTransactionAbort(bool isSubXact) elog(PANIC, "cannot abort transaction %u, it was already committed", xid); + /* + * Are we using the replication origins feature? Or, in other words, + * are we replaying remote actions? + */ + replorigin = (replorigin_session_origin != InvalidRepOriginId && + replorigin_session_origin != DoNotReplicateId); + /* Fetch the data we need for the abort record */ nrels = smgrGetPendingDeletes(false, &rels); nchildren = xactGetCommittedChildren(&children); @@ -1765,6 +1773,11 @@ RecordTransactionAbort(bool isSubXact) MyXactFlags, InvalidTransactionId, NULL); + if (replorigin) + /* Move LSNs forward for this replication origin */ + replorigin_session_advance(replorigin_session_origin_lsn, + XactLastRecEnd); + /* * Report the latest async abort LSN, so that the WAL writer knows to * flush this abort. There's nothing to be gained by delaying this, since diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c index 83e6eae855..4080bba987 100644 --- a/src/backend/commands/subscriptioncmds.c +++ b/src/backend/commands/subscriptioncmds.c @@ -83,7 +83,7 @@ typedef struct SubOpts bool copy_data; bool refresh; bool binary; - bool streaming; + char streaming; bool twophase; bool disableonerr; XLogRecPtr lsn; @@ -95,6 +95,62 @@ static List *merge_publications(List *oldpublist, List *newpublist, bool addpub, static void ReportSlotConnectionError(List *rstates, Oid subid, char *slotname, char *err); +/* + * Extract the streaming mode value from a DefElem. This is like + * defGetBoolean() but also accepts the special value of "apply". + */ +static char +defGetStreamingMode(DefElem *def) +{ + /* + * If no parameter given, assume "true" is meant. + */ + if (def->arg == NULL) + return SUBSTREAM_ON; + + /* + * Allow 0, 1, "true", "false", "on", "off" or "apply". + */ + switch (nodeTag(def->arg)) + { + case T_Integer: + switch (intVal(def->arg)) + { + case 0: + return SUBSTREAM_OFF; + case 1: + return SUBSTREAM_ON; + default: + /* otherwise, error out below */ + break; + } + break; + default: + { + char *sval = defGetString(def); + + /* + * The set of strings accepted here should match up with the + * grammar's opt_boolean_or_string production. + */ + if (pg_strcasecmp(sval, "false") == 0 || + pg_strcasecmp(sval, "off") == 0) + return SUBSTREAM_OFF; + if (pg_strcasecmp(sval, "true") == 0 || + pg_strcasecmp(sval, "on") == 0) + return SUBSTREAM_ON; + if (pg_strcasecmp(sval, "apply") == 0) + return SUBSTREAM_APPLY; + } + break; + } + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s requires a Boolean value or \"apply\"", + def->defname))); + return SUBSTREAM_OFF; /* keep compiler quiet */ +} + /* * Common option parsing function for CREATE and ALTER SUBSCRIPTION commands. * @@ -132,7 +188,7 @@ parse_subscription_options(ParseState *pstate, List *stmt_options, if (IsSet(supported_opts, SUBOPT_BINARY)) opts->binary = false; if (IsSet(supported_opts, SUBOPT_STREAMING)) - opts->streaming = false; + opts->streaming = SUBSTREAM_OFF; if (IsSet(supported_opts, SUBOPT_TWOPHASE_COMMIT)) opts->twophase = false; if (IsSet(supported_opts, SUBOPT_DISABLE_ON_ERR)) @@ -233,7 +289,7 @@ parse_subscription_options(ParseState *pstate, List *stmt_options, errorConflictingDefElem(defel, pstate); opts->specified_opts |= SUBOPT_STREAMING; - opts->streaming = defGetBoolean(defel); + opts->streaming = defGetStreamingMode(defel); } else if (strcmp(defel->defname, "two_phase") == 0) { @@ -601,7 +657,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt, values[Anum_pg_subscription_subowner - 1] = ObjectIdGetDatum(owner); values[Anum_pg_subscription_subenabled - 1] = BoolGetDatum(opts.enabled); values[Anum_pg_subscription_subbinary - 1] = BoolGetDatum(opts.binary); - values[Anum_pg_subscription_substream - 1] = BoolGetDatum(opts.streaming); + values[Anum_pg_subscription_substream - 1] = CharGetDatum(opts.streaming); values[Anum_pg_subscription_subtwophasestate - 1] = CharGetDatum(opts.twophase ? LOGICALREP_TWOPHASE_STATE_PENDING : @@ -1060,7 +1116,7 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt, if (IsSet(opts.specified_opts, SUBOPT_STREAMING)) { values[Anum_pg_subscription_substream - 1] = - BoolGetDatum(opts.streaming); + CharGetDatum(opts.streaming); replaces[Anum_pg_subscription_substream - 1] = true; } diff --git a/src/backend/postmaster/bgworker.c b/src/backend/postmaster/bgworker.c index 40601aefd9..40ccb8993c 100644 --- a/src/backend/postmaster/bgworker.c +++ b/src/backend/postmaster/bgworker.c @@ -128,6 +128,9 @@ static const struct }, { "ApplyWorkerMain", ApplyWorkerMain + }, + { + "ApplyBgworkerMain", ApplyBgworkerMain } }; diff --git a/src/backend/replication/logical/Makefile b/src/backend/replication/logical/Makefile index c4e2fdeb71..a24a41969e 100644 --- a/src/backend/replication/logical/Makefile +++ b/src/backend/replication/logical/Makefile @@ -26,6 +26,7 @@ OBJS = \ reorderbuffer.o \ snapbuild.o \ tablesync.o \ - worker.o + worker.o \ + applybgwroker.o include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/replication/logical/applybgwroker.c b/src/backend/replication/logical/applybgwroker.c new file mode 100644 index 0000000000..3e2f615f20 --- /dev/null +++ b/src/backend/replication/logical/applybgwroker.c @@ -0,0 +1,777 @@ +/*------------------------------------------------------------------------- + * applybgwroker.c + * Support routines for applying xact by apply background worker + * + * Copyright (c) 2016-2022, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/replication/logical/applybgwroker.c + * + * This file contains routines that are intended to support setting up, using, + * and tearing down a ApplyBgworkerState. + * + * Refer to the comments in file header of logical/worker.c to see more + * information about apply background worker. + * + *------------------------------------------------------------------------- + */ + +#include "postgres.h" + +#include "libpq/pqformat.h" +#include "mb/pg_wchar.h" +#include "pgstat.h" +#include "postmaster/interrupt.h" +#include "replication/logicallauncher.h" +#include "replication/logicalworker.h" +#include "replication/origin.h" +#include "replication/walreceiver.h" +#include "replication/worker_internal.h" +#include "storage/ipc.h" +#include "storage/procarray.h" +#include "tcop/tcopprot.h" +#include "utils/inval.h" +#include "utils/memutils.h" +#include "utils/resowner.h" +#include "utils/syscache.h" + +#define PG_LOGICAL_APPLY_SHM_MAGIC 0x79fb2447 // TODO Consider change + +/* + * DSM keys for apply background worker. Unlike other parallel execution code, + * since we don't need to worry about DSM keys conflicting with plan_node_id we + * can use small integers. + */ +#define APPLY_BGWORKER_KEY_SHARED 1 +#define APPLY_BGWORKER_KEY_MQ 2 + +/* queue size of DSM, 16 MB for now. */ +#define DSM_QUEUE_SIZE 160000000 + +/* + * There are three fields in message: start_lsn, end_lsn and send_time. Because + * we have updated these statistics in apply worker, we could ignore these + * fields in apply background worker. (see function LogicalRepApplyLoop) + */ +#define IGNORE_SIZE_IN_MESSAGE (3 * sizeof(uint64)) + +/* + * entry for a hash table we use to map from xid to our apply background worker + * state. + */ +typedef struct ApplyBgworkerEntry +{ + TransactionId xid; + ApplyBgworkerState *wstate; +} ApplyBgworkerEntry; + +/* Apply background workers hash table (initialized on first use) */ +static HTAB *ApplyWorkersHash = NULL; +static List *ApplyWorkersFreeList = NIL; +static List *ApplyWorkersList = NIL; + +/* + * Fields to record the share informations between main apply worker and apply + * background worker. + */ +volatile ApplyBgworkerShared *MyParallelState = NULL; + +List *subxactlist = NIL; + +/* apply background worker setup */ +static bool apply_bgworker_can_start(TransactionId xid); +static ApplyBgworkerState *apply_bgworker_setup(void); +static void apply_bgworker_setup_dsm(ApplyBgworkerState *wstate); + +/* + * Confirm if we can try to start a new apply background worker. + */ +static bool +apply_bgworker_can_start(TransactionId xid) +{ + if (!TransactionIdIsValid(xid)) + return false; + + /* + * We don't start new background worker if we are not in streaming apply + * mode. + */ + if (MySubscription->stream != SUBSTREAM_APPLY) + return false; + + /* + * We don't start new background worker if user has set skiplsn as it's + * possible that user want to skip the streaming transaction. For + * streaming transaction, we need to spill the transaction to disk so that + * we can get the last LSN of the transaction to judge whether to skip + * before starting to apply the change. + */ + if (!XLogRecPtrIsInvalid(MySubscription->skiplsn)) + return false; + + /* + * For streaming transactions that are being applied in apply background + * worker, we cannot decide whether to apply the change for a relation + * that is not in the READY state (see should_apply_changes_for_rel) as we + * won't know remote_final_lsn by that time. So, we don't start new apply + * background worker in this case. + */ + if (!AllTablesyncsReady()) + return false; + + return true; +} + +/* + * Try to start worker inside ApplyWorkersHash for requested xid. + */ +ApplyBgworkerState * +apply_bgworker_start(TransactionId xid) +{ + bool found; + ApplyBgworkerState *wstate; + ApplyBgworkerEntry *entry = NULL; + + if (!apply_bgworker_can_start(xid)) + return NULL; + + /* First time through, initialize apply workers hashtable */ + if (ApplyWorkersHash == NULL) + { + HASHCTL ctl; + + MemSet(&ctl, 0, sizeof(ctl)); + ctl.keysize = sizeof(TransactionId); + ctl.entrysize = sizeof(ApplyBgworkerEntry); + ctl.hcxt = ApplyContext; + + ApplyWorkersHash = hash_create("logical apply workers hash", 8, &ctl, + HASH_ELEM | HASH_BLOBS | HASH_CONTEXT); + } + + /* + * Now, we try to get a apply background worker. If there is at least one + * worker in the idle list, then take one. Otherwise, we try to start a + * new apply background worker. + */ + if (list_length(ApplyWorkersFreeList) > 0) + { + wstate = (ApplyBgworkerState *) llast(ApplyWorkersFreeList); + ApplyWorkersFreeList = list_delete_last(ApplyWorkersFreeList); + } + else + { + wstate = apply_bgworker_setup(); + + if (wstate == NULL) + return NULL; + } + + /* + * Create entry for requested transaction. + */ + entry = hash_search(ApplyWorkersHash, &xid, HASH_ENTER, &found); + if (found) + elog(ERROR, "hash table corrupted"); + + /* Fill up the hash entry */ + wstate->pstate->status = APPLY_BGWORKER_BUSY; + wstate->pstate->server_version = walrcv_server_version(LogRepWorkerWalRcvConn); + wstate->pstate->stream_xid = xid; + entry->wstate = wstate; + entry->xid = xid; + + return wstate; +} + +/* + * Try to look up worker inside ApplyWorkersHash for requested xid. + */ +ApplyBgworkerState * +apply_bgworker_find(TransactionId xid) +{ + bool found; + ApplyBgworkerEntry *entry = NULL; + + if (!TransactionIdIsValid(xid)) + return NULL; + + if (ApplyWorkersHash == NULL) + return NULL; + + /* + * Find entry for requested transaction. + */ + entry = hash_search(ApplyWorkersHash, &xid, HASH_FIND, &found); + if (found) + { + entry->wstate->pstate->status = APPLY_BGWORKER_BUSY; + return entry->wstate; + } + else + return NULL; +} + +/* + * Add the worker to the free list and remove the entry from hash table. + */ +void +apply_bgworker_free(ApplyBgworkerState *wstate) +{ + MemoryContext oldctx; + TransactionId xid = wstate->pstate->stream_xid; + + Assert(wstate->pstate->status == APPLY_BGWORKER_FINISHED); + + oldctx = MemoryContextSwitchTo(ApplyContext); + + hash_search(ApplyWorkersHash, &xid, HASH_REMOVE, NULL); + + elog(DEBUG1, "adding finished apply worker #%u for xid %u to the idle list", + wstate->pstate->n, wstate->pstate->stream_xid); + + ApplyWorkersFreeList = lappend(ApplyWorkersFreeList, wstate); + + MemoryContextSwitchTo(oldctx); +} + +/* Apply Background Worker main loop */ +static void +LogicalApplyBgwLoop(shm_mq_handle *mqh, volatile ApplyBgworkerShared *pst) +{ + shm_mq_result shmq_res; + PGPROC *registrant; + ErrorContextCallback errcallback; + + registrant = BackendPidGetProc(MyBgworkerEntry->bgw_notify_pid); + SetLatch(®istrant->procLatch); + + /* + * Push apply error context callback. Fields will be filled during + * applying a change. + */ + errcallback.callback = apply_error_callback; + errcallback.previous = error_context_stack; + error_context_stack = &errcallback; + + for (;;) + { + void *data; + Size len; + int c; + StringInfoData s; + MemoryContext oldctx; + + CHECK_FOR_INTERRUPTS(); + + /* Ensure we are reading the data into our memory context. */ + oldctx = MemoryContextSwitchTo(ApplyMessageContext); + + shmq_res = shm_mq_receive(mqh, &len, &data, false); + + if (shmq_res != SHM_MQ_SUCCESS) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("lost connection to the main apply worker"))); + + if (len == 0) + break; + + s.cursor = 0; + s.maxlen = -1; + s.data = (char *) data; + s.len = len; + + /* + * We use first byte of message for additional communication between + * main Logical replication worker and apply bgworkers, so if it + * differs from 'w', then process it first. + */ + c = pq_getmsgbyte(&s); + switch (c) + { + /* End message of streaming chunk */ + case LOGICAL_REP_MSG_STREAM_STOP: + elog(DEBUG1, "[Apply BGW #%u] ended processing streaming chunk," + "waiting on shm_mq_receive", pst->n); + + in_streamed_transaction = false; + pgstat_report_activity(STATE_IDLEINTRANSACTION, NULL); + continue; + case 'w': + break; + default: + elog(ERROR, "[Apply BGW #%u] unexpected message \"%c\"", + pst->n, c); + break; + } + + /* Ignore statistics fields that have been updated. */ + s.cursor += IGNORE_SIZE_IN_MESSAGE; + + apply_dispatch(&s); + + if (ConfigReloadPending) + { + ConfigReloadPending = false; + ProcessConfigFile(PGC_SIGHUP); + } + + MemoryContextSwitchTo(oldctx); + MemoryContextReset(ApplyMessageContext); + } + + MemoryContextSwitchTo(TopMemoryContext); + MemoryContextReset(ApplyContext); + + /* Pop the error context stack */ + error_context_stack = errcallback.previous; + + elog(DEBUG1, "[Apply BGW #%u] exiting", pst->n); + + /* Signal main process that we are done. */ + SetLatch(®istrant->procLatch); +} + +/* + * Set the exit status so that the main apply worker can realize we have + * shutdown. + */ +static void +ApplyBgwShutdown(int code, Datum arg) +{ + SpinLockAcquire(&MyParallelState->mutex); + MyParallelState->status = APPLY_BGWORKER_EXIT; + SpinLockRelease(&MyParallelState->mutex); + + dsm_detach((dsm_segment *) DatumGetPointer(arg)); +} + +/* + * Apply Background Worker entry point + */ +void +ApplyBgworkerMain(Datum main_arg) +{ + volatile ApplyBgworkerShared *pst; + + dsm_handle handle; + dsm_segment *seg; + shm_toc *toc; + shm_mq *mq; + shm_mq_handle *mqh; + MemoryContext oldcontext; + RepOriginId originid; + int worker_slot = DatumGetInt32(main_arg); + char originname[NAMEDATALEN]; + + MemoryContextSwitchTo(TopMemoryContext); + + /* Init the memory context for the apply background worker to work in. */ + ApplyContext = AllocSetContextCreate(TopMemoryContext, + "ApplyContext", + ALLOCSET_DEFAULT_SIZES); + + /* + * Init the ApplyMessageContext which we clean up after each replication + * protocol message. + */ + ApplyMessageContext = AllocSetContextCreate(ApplyContext, + "ApplyMessageContext", + ALLOCSET_DEFAULT_SIZES); + + /* Setup signal handling */ + pqsignal(SIGHUP, SignalHandlerForConfigReload); + pqsignal(SIGTERM, die); + BackgroundWorkerUnblockSignals(); + + /* + * Connect to the dynamic shared memory segment. + * + * The backend that registered this worker passed us the ID of a shared + * memory segment to which we must attach for further instructions. In + * order to attach to dynamic shared memory, we need a resource owner. + * Once we've mapped the segment in our address space, attach to the table + * of contents so we can locate the various data structures we'll need to + * find within the segment. + */ + CurrentResourceOwner = ResourceOwnerCreate(NULL, "Logical apply worker"); + memcpy(&handle, MyBgworkerEntry->bgw_extra, sizeof(dsm_handle)); + seg = dsm_attach(handle); + if (seg == NULL) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("unable to map dynamic shared memory segment"))); + toc = shm_toc_attach(PG_LOGICAL_APPLY_SHM_MAGIC, dsm_segment_address(seg)); + if (toc == NULL) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("bad magic number in dynamic shared memory segment"))); + + before_shmem_exit(ApplyBgwShutdown, PointerGetDatum(seg)); + + /* Look up the parallel state. */ + pst = shm_toc_lookup(toc, APPLY_BGWORKER_KEY_SHARED, false); + MyParallelState = pst; + + /* + * Attach to the message queue. + */ + mq = shm_toc_lookup(toc, APPLY_BGWORKER_KEY_MQ, false); + shm_mq_set_receiver(mq, MyProc); + mqh = shm_mq_attach(mq, seg, NULL); + + /* Run as replica session replication role. */ + SetConfigOption("session_replication_role", "replica", + PGC_SUSET, PGC_S_OVERRIDE); + + /* + * Now, we have initialized DSM. Attach to slot. + */ + logicalrep_worker_attach(worker_slot); + + /* Connect to our database. */ + BackgroundWorkerInitializeConnectionByOid(MyLogicalRepWorker->dbid, + MyLogicalRepWorker->userid, + 0); + + /* + * Set always-secure search path, so malicious users can't redirect user + * code (e.g. pg_index.indexprs). + */ + SetConfigOption("search_path", "", PGC_SUSET, PGC_S_OVERRIDE); + + /* + * Set the client encoding to the database encoding, since that is what + * the leader will expect. + */ + SetClientEncoding(GetDatabaseEncoding()); + + stream_xid = pst->stream_xid; + + StartTransactionCommand(); + oldcontext = MemoryContextSwitchTo(ApplyContext); + + MySubscription = GetSubscription(MyLogicalRepWorker->subid, true); + if (!MySubscription) + { + ereport(LOG, + (errmsg("logical replication apply worker for subscription %u will not " + "start because the subscription was removed during startup", + MyLogicalRepWorker->subid))); + proc_exit(0); + } + + MySubscriptionValid = true; + MemoryContextSwitchTo(oldcontext); + + /* Setup synchronous commit according to the user's wishes */ + SetConfigOption("synchronous_commit", MySubscription->synccommit, + PGC_BACKEND, PGC_S_OVERRIDE); + + /* Keep us informed about subscription changes. */ + CacheRegisterSyscacheCallback(SUBSCRIPTIONOID, + subscription_change_cb, + (Datum) 0); + + CommitTransactionCommand(); + + /* Setup replication origin tracking. */ + StartTransactionCommand(); + snprintf(originname, sizeof(originname), "pg_%u", MySubscription->oid); + originid = replorigin_by_name(originname, true); + if (!OidIsValid(originid)) + originid = replorigin_create(originname); + + /* + * The apply background worker doesn't need to monopolize this replication + * origin which was already acquired by its leader process. + */ + replorigin_session_setup(originid, false); + replorigin_session_origin = originid; + CommitTransactionCommand(); + + /* + * Allocate the origin name in long-lived context for error context + * message. + */ + apply_error_callback_arg.origin_name = MemoryContextStrdup(ApplyContext, + originname); + + elog(DEBUG1, "[Apply BGW #%u] started", pst->n); + + LogicalApplyBgwLoop(mqh, pst); + + /* + * We're done. Explicitly detach the shared memory segment so that we + * don't get a resource leak warning at commit time. This will fire any + * on_dsm_detach callbacks we've registered, as well. Once that's done, + * we can go ahead and exit. + */ + dsm_detach(seg); + proc_exit(0); +} + +/* + * Set up a dynamic shared memory segment. + * + * We set up a control region that contains a ApplyBgworkerShared, + * plus one region per message queue. There are as many message queues as + * the number of workers. + */ +static void +apply_bgworker_setup_dsm(ApplyBgworkerState *wstate) +{ + shm_toc_estimator e; + Size segsize; + dsm_segment *seg; + shm_toc *toc; + ApplyBgworkerShared *pst; + shm_mq *mq; + int64 queue_size = DSM_QUEUE_SIZE; + + /* + * Estimate how much shared memory we need. + * + * Because the TOC machinery may choose to insert padding of oddly-sized + * requests, we must estimate each chunk separately. + * + * We need one key to register the location of the header, and we need + * another key to track the location of the message queue. + */ + shm_toc_initialize_estimator(&e); + shm_toc_estimate_chunk(&e, sizeof(ApplyBgworkerShared)); + shm_toc_estimate_chunk(&e, (Size) queue_size); + + shm_toc_estimate_keys(&e, 1 + 1); + segsize = shm_toc_estimate(&e); + + /* Create the shared memory segment and establish a table of contents. */ + seg = dsm_create(shm_toc_estimate(&e), 0); + toc = shm_toc_create(PG_LOGICAL_APPLY_SHM_MAGIC, dsm_segment_address(seg), + segsize); + + /* Set up the header region. */ + pst = shm_toc_allocate(toc, sizeof(ApplyBgworkerShared)); + SpinLockInit(&pst->mutex); + pst->status = APPLY_BGWORKER_BUSY; + pst->server_version = walrcv_server_version(LogRepWorkerWalRcvConn); + pst->stream_xid = stream_xid; + pst->n = list_length(ApplyWorkersList) + 1; + + shm_toc_insert(toc, APPLY_BGWORKER_KEY_SHARED, pst); + + /* Set up message queue for the worker. */ + mq = shm_mq_create(shm_toc_allocate(toc, (Size) queue_size), + (Size) queue_size); + shm_toc_insert(toc, APPLY_BGWORKER_KEY_MQ, mq); + shm_mq_set_sender(mq, MyProc); + + /* Attach the queue. */ + wstate->mq_handle = shm_mq_attach(mq, seg, NULL); + + /* Return results to caller. */ + wstate->dsm_seg = seg; + wstate->pstate = pst; +} + +/* + * Start apply background worker process and allocate shared memory for it. + */ +static ApplyBgworkerState * +apply_bgworker_setup(void) +{ + MemoryContext oldcontext; + bool launched; + ApplyBgworkerState *wstate; + int napplyworkers; + + elog(DEBUG1, "setting up apply worker #%u", list_length(ApplyWorkersList) + 1); + + /* Check If there are free worker slot(s) */ + LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); + napplyworkers = logicalrep_apply_background_worker_count(MyLogicalRepWorker->subid); + LWLockRelease(LogicalRepWorkerLock); + if (napplyworkers >= max_apply_bgworkers_per_subscription) + return NULL; + + oldcontext = MemoryContextSwitchTo(ApplyContext); + + wstate = (ApplyBgworkerState *) palloc0(sizeof(ApplyBgworkerState)); + + /* Setup shared memory */ + apply_bgworker_setup_dsm(wstate); + + launched = logicalrep_worker_launch(MyLogicalRepWorker->dbid, + MySubscription->oid, + MySubscription->name, + MyLogicalRepWorker->userid, + InvalidOid, + dsm_segment_handle(wstate->dsm_seg)); + + if (launched) + ApplyWorkersList = lappend(ApplyWorkersList, wstate); + else + { + shm_mq_detach(wstate->mq_handle); + dsm_detach(wstate->dsm_seg); + pfree(wstate); + + wstate->mq_handle = NULL; + wstate->dsm_seg = NULL; + wstate = NULL; + } + + MemoryContextSwitchTo(oldcontext); + + return wstate; +} + +/* + * Send the data to the specified apply background worker via shared-memory queue. + */ +void +apply_bgworker_send_data(ApplyBgworkerState *wstate, Size nbytes, const void *data) +{ + shm_mq_result result; + + result = shm_mq_send(wstate->mq_handle, nbytes, data, false, true); + + if (result != SHM_MQ_SUCCESS) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("could not send tuples to shared-memory queue"))); +} + +/* + * Wait until the status of apply background worker reaches the + * 'wait_for_status' + */ +void +apply_bgworker_wait_for(ApplyBgworkerState *wstate, + ApplyBgworkerStatus wait_for_status) +{ + for (;;) + { + char status; + + SpinLockAcquire(&wstate->pstate->mutex); + status = wstate->pstate->status; + SpinLockRelease(&wstate->pstate->mutex); + + /* Done if already in correct status. */ + if (status == wait_for_status) + break; + + /* If any workers (or the postmaster) have died, we have failed. */ + if (status == APPLY_BGWORKER_EXIT) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("background worker %u failed to apply transaction %u", + wstate->pstate->n, wstate->pstate->stream_xid))); + + /* Wait to be signalled. */ + WaitLatch(MyLatch, WL_LATCH_SET | WL_EXIT_ON_PM_DEATH, 0, + WAIT_EVENT_LOGICAL_APPLY_WORKER_STATE_CHANGE); + + /* Reset the latch so we don't spin. */ + ResetLatch(MyLatch); + + /* An interrupt may have occurred while we were waiting. */ + CHECK_FOR_INTERRUPTS(); + } +} + +/* + * Check the status of workers and report an error if any apply background + * worker has exited unexpectedly. + * + * Exit if any relation is not in the READY state and if any worker is handling + * the streaming transaction at the same time. Because for streaming + * transactions that is being applied in apply background worker, we cannot + * decide whether to apply the change for a relation that is not in the READY + * state (see should_apply_changes_for_rel) as we won't know remote_final_lsn + * by that time. + */ +void +apply_bgworker_check_status(void) +{ + ListCell *lc; + + if (am_apply_bgworker() || MySubscription->stream != SUBSTREAM_APPLY) + return; + + foreach(lc, ApplyWorkersList) + { + ApplyBgworkerState *wstate = (ApplyBgworkerState *) lfirst(lc); + + /* + * We don't lock here as in the worst case we will just detect the + * failure of worker a bit later. + */ + if (wstate->pstate->status == APPLY_BGWORKER_EXIT) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("background worker %u exited unexpectedly", + wstate->pstate->n))); + } + + if (list_length(ApplyWorkersFreeList) != list_length(ApplyWorkersList) && + !AllTablesyncsReady()) + { + ereport(LOG, + (errmsg("logical replication apply workers for subscription \"%s\" will restart", + MySubscription->name), + errdetail("Cannot handle streamed replication transaction by apply " + "bgworkers until all tables are synchronized"))); + + proc_exit(0); + } +} + +/* Set the apply background worker status */ +void +apply_bgworker_set_status(ApplyBgworkerStatus status) +{ + if (!am_apply_bgworker()) + return; + + elog(DEBUG1, "[Apply BGW #%u] set status to %d", MyParallelState->n, status); + + SpinLockAcquire(&MyParallelState->mutex); + MyParallelState->status = status; + SpinLockRelease(&MyParallelState->mutex); +} + +/* + * Define a savepoint for a subxact in apply background worker if needed. + * + * Inside apply background worker we can figure out that new subtransaction was + * started if new change arrived with different xid. In that case we can define + * named savepoint, so that we were able to commit/rollback it separately + * later. + * Special case is if the first change comes from subtransaction, then + * we check that current_xid differs from stream_xid. + */ +void +apply_bgworker_subxact_info_add(TransactionId current_xid) +{ + if (current_xid != stream_xid && + !list_member_int(subxactlist, (int) current_xid)) + { + MemoryContext oldctx; + char spname[MAXPGPATH]; + + snprintf(spname, MAXPGPATH, "savepoint_for_xid_%u", current_xid); + + elog(DEBUG1, "[Apply BGW #%u] defining savepoint %s", + MyParallelState->n, spname); + + DefineSavepoint(spname); + CommitTransactionCommand(); + + oldctx = MemoryContextSwitchTo(ApplyContext); + subxactlist = lappend_int(subxactlist, (int) current_xid); + MemoryContextSwitchTo(oldctx); + } +} diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c index aa2427ba73..00fdde0830 100644 --- a/src/backend/replication/logical/decode.c +++ b/src/backend/replication/logical/decode.c @@ -651,9 +651,10 @@ DecodeCommit(LogicalDecodingContext *ctx, XLogRecordBuffer *buf, { for (i = 0; i < parsed->nsubxacts; i++) { - ReorderBufferForget(ctx->reorder, parsed->subxacts[i], buf->origptr); + ReorderBufferForget(ctx->reorder, parsed->subxacts[i], buf->origptr, + commit_time); } - ReorderBufferForget(ctx->reorder, xid, buf->origptr); + ReorderBufferForget(ctx->reorder, xid, buf->origptr, commit_time); return; } @@ -821,10 +822,11 @@ DecodeAbort(LogicalDecodingContext *ctx, XLogRecordBuffer *buf, for (i = 0; i < parsed->nsubxacts; i++) { ReorderBufferAbort(ctx->reorder, parsed->subxacts[i], - buf->record->EndRecPtr); + buf->record->EndRecPtr, abort_time); } - ReorderBufferAbort(ctx->reorder, xid, buf->record->EndRecPtr); + ReorderBufferAbort(ctx->reorder, xid, buf->record->EndRecPtr, + abort_time); } /* update the decoding stats */ diff --git a/src/backend/replication/logical/launcher.c b/src/backend/replication/logical/launcher.c index 2bdab53e19..42140b6ad5 100644 --- a/src/backend/replication/logical/launcher.c +++ b/src/backend/replication/logical/launcher.c @@ -54,6 +54,7 @@ int max_logical_replication_workers = 4; int max_sync_workers_per_subscription = 2; +int max_apply_bgworkers_per_subscription = 2; LogicalRepWorker *MyLogicalRepWorker = NULL; @@ -73,6 +74,7 @@ static void logicalrep_launcher_onexit(int code, Datum arg); static void logicalrep_worker_onexit(int code, Datum arg); static void logicalrep_worker_detach(void); static void logicalrep_worker_cleanup(LogicalRepWorker *worker); +static void logicalrep_worker_stop_internal(LogicalRepWorker *worker); static bool on_commit_launcher_wakeup = false; @@ -151,8 +153,10 @@ get_subscription_list(void) * * This is only needed for cleaning up the shared memory in case the worker * fails to attach. + * + * Returns false if the attach fails. Otherwise return true. */ -static void +static bool WaitForReplicationWorkerAttach(LogicalRepWorker *worker, uint16 generation, BackgroundWorkerHandle *handle) @@ -168,11 +172,11 @@ WaitForReplicationWorkerAttach(LogicalRepWorker *worker, LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); - /* Worker either died or has started; no need to do anything. */ + /* Worker either died or has started. Return false if died. */ if (!worker->in_use || worker->proc) { LWLockRelease(LogicalRepWorkerLock); - return; + return worker->in_use ? true : false; } LWLockRelease(LogicalRepWorkerLock); @@ -187,7 +191,7 @@ WaitForReplicationWorkerAttach(LogicalRepWorker *worker, if (generation == worker->generation) logicalrep_worker_cleanup(worker); LWLockRelease(LogicalRepWorkerLock); - return; + return false; } /* @@ -223,6 +227,10 @@ logicalrep_worker_find(Oid subid, Oid relid, bool only_running) { LogicalRepWorker *w = &LogicalRepCtx->workers[i]; + /* We only need main apply worker or table sync worker here */ + if (w->subworker) + continue; + if (w->in_use && w->subid == subid && w->relid == relid && (!only_running || w->proc)) { @@ -259,11 +267,11 @@ logicalrep_workers_find(Oid subid, bool only_running) } /* - * Start new apply background worker, if possible. + * Start new background worker, if possible. */ -void +bool logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, Oid userid, - Oid relid) + Oid relid, dsm_handle subworker_dsm) { BackgroundWorker bgw; BackgroundWorkerHandle *bgw_handle; @@ -273,6 +281,10 @@ logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, Oid userid, LogicalRepWorker *worker = NULL; int nsyncworkers; TimestampTz now; + bool is_subworker = (subworker_dsm != DSM_HANDLE_INVALID); + + /* We don't support table sync in subworker */ + Assert(!(is_subworker && OidIsValid(relid))); ereport(DEBUG1, (errmsg_internal("starting logical replication worker for subscription \"%s\"", @@ -350,7 +362,7 @@ retry: if (OidIsValid(relid) && nsyncworkers >= max_sync_workers_per_subscription) { LWLockRelease(LogicalRepWorkerLock); - return; + return false; } /* @@ -364,7 +376,7 @@ retry: (errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED), errmsg("out of logical replication worker slots"), errhint("You might need to increase max_logical_replication_workers."))); - return; + return false; } /* Prepare the worker slot. */ @@ -379,6 +391,7 @@ retry: worker->relstate = SUBREL_STATE_UNKNOWN; worker->relstate_lsn = InvalidXLogRecPtr; worker->stream_fileset = NULL; + worker->subworker = is_subworker; worker->last_lsn = InvalidXLogRecPtr; TIMESTAMP_NOBEGIN(worker->last_send_time); TIMESTAMP_NOBEGIN(worker->last_recv_time); @@ -396,19 +409,30 @@ retry: BGWORKER_BACKEND_DATABASE_CONNECTION; bgw.bgw_start_time = BgWorkerStart_RecoveryFinished; snprintf(bgw.bgw_library_name, BGW_MAXLEN, "postgres"); - snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyWorkerMain"); + + if (!is_subworker) + snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyWorkerMain"); + else + snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyBgworkerMain"); + if (OidIsValid(relid)) snprintf(bgw.bgw_name, BGW_MAXLEN, "logical replication worker for subscription %u sync %u", subid, relid); - else + else if (!is_subworker) snprintf(bgw.bgw_name, BGW_MAXLEN, "logical replication worker for subscription %u", subid); + else + snprintf(bgw.bgw_name, BGW_MAXLEN, + "logical replication background apply worker for subscription %u", subid); snprintf(bgw.bgw_type, BGW_MAXLEN, "logical replication worker"); bgw.bgw_restart_time = BGW_NEVER_RESTART; bgw.bgw_notify_pid = MyProcPid; bgw.bgw_main_arg = Int32GetDatum(slot); + if (is_subworker) + memcpy(bgw.bgw_extra, &subworker_dsm, sizeof(dsm_handle)); + if (!RegisterDynamicBackgroundWorker(&bgw, &bgw_handle)) { /* Failed to start worker, so clean up the worker slot. */ @@ -421,11 +445,11 @@ retry: (errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED), errmsg("out of background worker slots"), errhint("You might need to increase max_worker_processes."))); - return; + return false; } /* Now wait until it attaches. */ - WaitForReplicationWorkerAttach(worker, generation, bgw_handle); + return WaitForReplicationWorkerAttach(worker, generation, bgw_handle); } /* @@ -436,7 +460,6 @@ void logicalrep_worker_stop(Oid subid, Oid relid) { LogicalRepWorker *worker; - uint16 generation; LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); @@ -449,6 +472,22 @@ logicalrep_worker_stop(Oid subid, Oid relid) return; } + logicalrep_worker_stop_internal(worker); + + LWLockRelease(LogicalRepWorkerLock); +} + +/* + * Workhorse for logicalrep_worker_stop() and logicalrep_worker_detach(). Stop + * the worker and wait for it to die. + */ +static void +logicalrep_worker_stop_internal(LogicalRepWorker *worker) +{ + uint16 generation; + + Assert(LWLockHeldByMe(LogicalRepWorkerLock)); + /* * Remember which generation was our worker so we can check if what we see * is still the same one. @@ -485,10 +524,7 @@ logicalrep_worker_stop(Oid subid, Oid relid) * different, meaning that a different worker has taken the slot. */ if (!worker->in_use || worker->generation != generation) - { - LWLockRelease(LogicalRepWorkerLock); return; - } /* Worker has assigned proc, so it has started. */ if (worker->proc) @@ -522,8 +558,6 @@ logicalrep_worker_stop(Oid subid, Oid relid) LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); } - - LWLockRelease(LogicalRepWorkerLock); } /* @@ -599,6 +633,29 @@ logicalrep_worker_attach(int slot) static void logicalrep_worker_detach(void) { + /* + * If we are the main apply worker, stop all the apply background workers + * we started before. + */ + if (!MyLogicalRepWorker->subworker) + { + List *workers; + ListCell *lc; + + LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); + + workers = logicalrep_workers_find(MyLogicalRepWorker->subid, true); + foreach(lc, workers) + { + LogicalRepWorker *w = (LogicalRepWorker *) lfirst(lc); + + if (w->subworker) + logicalrep_worker_stop_internal(w); + } + + LWLockRelease(LogicalRepWorkerLock); + } + /* Block concurrent access. */ LWLockAcquire(LogicalRepWorkerLock, LW_EXCLUSIVE); @@ -621,6 +678,7 @@ logicalrep_worker_cleanup(LogicalRepWorker *worker) worker->userid = InvalidOid; worker->subid = InvalidOid; worker->relid = InvalidOid; + worker->subworker = false; } /* @@ -679,6 +737,30 @@ logicalrep_sync_worker_count(Oid subid) return res; } +/* + * Count the number of registered (not necessarily running) apply background + * workers for a subscription. + */ +int +logicalrep_apply_background_worker_count(Oid subid) +{ + int i; + int res = 0; + + Assert(LWLockHeldByMe(LogicalRepWorkerLock)); + + /* Search for attached worker for a given subscription id. */ + for (i = 0; i < max_logical_replication_workers; i++) + { + LogicalRepWorker *w = &LogicalRepCtx->workers[i]; + + if (w->subid == subid && w->subworker) + res++; + } + + return res; +} + /* * ApplyLauncherShmemSize * Compute space needed for replication launcher shared memory @@ -868,7 +950,7 @@ ApplyLauncherMain(Datum main_arg) wait_time = wal_retrieve_retry_interval; logicalrep_worker_launch(sub->dbid, sub->oid, sub->name, - sub->owner, InvalidOid); + sub->owner, InvalidOid, DSM_HANDLE_INVALID); } } diff --git a/src/backend/replication/logical/origin.c b/src/backend/replication/logical/origin.c index 21937ab2d3..9273011b0a 100644 --- a/src/backend/replication/logical/origin.c +++ b/src/backend/replication/logical/origin.c @@ -1063,12 +1063,21 @@ ReplicationOriginExitCleanup(int code, Datum arg) * array doesn't have to be searched when calling * replorigin_session_advance(). * - * Obviously only one such cached origin can exist per process and the current + * Normally only one such cached origin can exist per process and the current * cached value can only be set again after the previous value is torn down * with replorigin_session_reset(). + * + * However, If must_acquire is false, we allow process to get the slot which is + * already acquired by other process. It's safe because 1) The only caller + * (apply background workers) will maintain the commit order by allowing only + * one process to commit at a time, so no two workers will be operating on the + * same origin at the same time (see comments in logical/worker.c). 2) Even + * though we try to advance the session's origin concurrently, it's safe to do + * so as we change/advance the session_origin LSNs under replicate_state + * LWLock. */ void -replorigin_session_setup(RepOriginId node) +replorigin_session_setup(RepOriginId node, bool must_acquire) { static bool registered_cleanup; int i; @@ -1110,7 +1119,7 @@ replorigin_session_setup(RepOriginId node) if (curstate->roident != node) continue; - else if (curstate->acquired_by != 0) + else if (curstate->acquired_by != 0 && must_acquire) { ereport(ERROR, (errcode(ERRCODE_OBJECT_IN_USE), @@ -1141,7 +1150,14 @@ replorigin_session_setup(RepOriginId node) Assert(session_replication_state->roident != InvalidRepOriginId); - session_replication_state->acquired_by = MyProcPid; + if (must_acquire) + session_replication_state->acquired_by = MyProcPid; + else if (session_replication_state->acquired_by == 0) + ereport(ERROR, + (errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED), + errmsg("could not find correct replication state slot for replication origin with OID %u for apply background worker", + node), + errhint("There is no replication state slot set by its main apply worker."))); LWLockRelease(ReplicationOriginLock); @@ -1321,7 +1337,7 @@ pg_replication_origin_session_setup(PG_FUNCTION_ARGS) name = text_to_cstring((text *) DatumGetPointer(PG_GETARG_DATUM(0))); origin = replorigin_by_name(name, false); - replorigin_session_setup(origin); + replorigin_session_setup(origin, true); replorigin_session_origin = origin; diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c index ff8513e2d2..a888038eb4 100644 --- a/src/backend/replication/logical/proto.c +++ b/src/backend/replication/logical/proto.c @@ -1166,28 +1166,47 @@ logicalrep_read_stream_commit(StringInfo in, LogicalRepCommitData *commit_data) */ void logicalrep_write_stream_abort(StringInfo out, TransactionId xid, - TransactionId subxid) + ReorderBufferTXN *txn, XLogRecPtr abort_lsn) { pq_sendbyte(out, LOGICAL_REP_MSG_STREAM_ABORT); - Assert(TransactionIdIsValid(xid) && TransactionIdIsValid(subxid)); + Assert(TransactionIdIsValid(xid) && TransactionIdIsValid(txn->xid)); /* transaction ID */ pq_sendint32(out, xid); - pq_sendint32(out, subxid); + pq_sendint32(out, txn->xid); + pq_sendint64(out, abort_lsn); + pq_sendint64(out, txn->xact_time.abort_time); } /* * Read STREAM ABORT from the output stream. */ void -logicalrep_read_stream_abort(StringInfo in, TransactionId *xid, - TransactionId *subxid) +logicalrep_read_stream_abort(StringInfo in, + LogicalRepStreamAbortData *abort_data, + bool include_abort_lsn) { - Assert(xid && subxid); + Assert(abort_data); + + abort_data->xid = pq_getmsgint(in, 4); + abort_data->subxid = pq_getmsgint(in, 4); - *xid = pq_getmsgint(in, 4); - *subxid = pq_getmsgint(in, 4); + /* + * If the version of the publisher is lower than the version of the + * subscriber, it may not support sending these two fields, so only take + * these fields when include_abort_lsn is true. + */ + if (include_abort_lsn) + { + abort_data->abort_lsn = pq_getmsgint64(in); + abort_data->abort_time = pq_getmsgint64(in); + } + else + { + abort_data->abort_lsn = InvalidXLogRecPtr; + abort_data->abort_time = 0; + } } /* diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c index 8da5f9089c..3d2bbdb38d 100644 --- a/src/backend/replication/logical/reorderbuffer.c +++ b/src/backend/replication/logical/reorderbuffer.c @@ -2826,7 +2826,8 @@ ReorderBufferFinishPrepared(ReorderBuffer *rb, TransactionId xid, * disk. */ void -ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn) +ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn, + TimestampTz abort_time) { ReorderBufferTXN *txn; @@ -2837,6 +2838,8 @@ ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn) if (txn == NULL) return; + txn->xact_time.abort_time = abort_time; + /* For streamed transactions notify the remote node about the abort. */ if (rbtxn_is_streamed(txn)) { @@ -2911,7 +2914,8 @@ ReorderBufferAbortOld(ReorderBuffer *rb, TransactionId oldestRunningXid) * to this xid might re-create the transaction incompletely. */ void -ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn) +ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn, + TimestampTz abort_time) { ReorderBufferTXN *txn; @@ -2922,6 +2926,8 @@ ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn) if (txn == NULL) return; + txn->xact_time.abort_time = abort_time; + /* For streamed transactions notify the remote node about the abort. */ if (rbtxn_is_streamed(txn)) rb->stream_abort(rb, txn, lsn); diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c index 670c6fcada..8ffba7e2e5 100644 --- a/src/backend/replication/logical/tablesync.c +++ b/src/backend/replication/logical/tablesync.c @@ -568,7 +568,8 @@ process_syncing_tables_for_apply(XLogRecPtr current_lsn) MySubscription->oid, MySubscription->name, MyLogicalRepWorker->userid, - rstate->relid); + rstate->relid, + DSM_HANDLE_INVALID); hentry->last_start_time = now; } } @@ -589,6 +590,9 @@ process_syncing_tables_for_apply(XLogRecPtr current_lsn) void process_syncing_tables(XLogRecPtr current_lsn) { + if (MyLogicalRepWorker->subworker) + return; + if (am_tablesync_worker()) process_syncing_tables_for_sync(current_lsn); else @@ -1273,7 +1277,7 @@ LogicalRepSyncTableStart(XLogRecPtr *origin_startpos) * time this tablesync was launched. */ originid = replorigin_by_name(originname, false); - replorigin_session_setup(originid); + replorigin_session_setup(originid, true); replorigin_session_origin = originid; *origin_startpos = replorigin_session_get_progress(false); @@ -1384,7 +1388,7 @@ LogicalRepSyncTableStart(XLogRecPtr *origin_startpos) true /* go backward */ , true /* WAL log */ ); UnlockRelationOid(ReplicationOriginRelationId, RowExclusiveLock); - replorigin_session_setup(originid); + replorigin_session_setup(originid, true); replorigin_session_origin = originid; } else diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c index 38e3b1c1b3..cc31e016e4 100644 --- a/src/backend/replication/logical/worker.c +++ b/src/backend/replication/logical/worker.c @@ -22,8 +22,28 @@ * STREAMED TRANSACTIONS * --------------------- * Streamed transactions (large transactions exceeding a memory limit on the - * upstream) are not applied immediately, but instead, the data is written - * to temporary files and then applied at once when the final commit arrives. + * upstream) are applied via one of two approaches. + * + * 1) Separate background workers + * + * Assign a new apply background worker (if available) as soon as the xact's + * first stream is received and the main apply worker will send changes to this + * new worker via shared memory. We keep this worker assigned till the + * transaction commit is received and also wait for the worker to finish at + * commit. This preserves commit ordering and avoids writing to and reading + * from file in most cases. We still need to spill if there is no worker + * available. It is important to maintain commit order to avoid failures + * due to (a) transaction dependencies, say if we insert a row in the first + * transaction and update it in the second transaction then allowing to apply + * both in parallel can lead to failure in the update. (b) deadlocks, allowing + * transactions that update the same set of rows/tables in opposite order to be + * applied in parallel can lead to deadlocks. + * + * 2) Write to temporary files and apply when the final commit arrives + * + * If no worker is available to handle streamed transaction, the data is + * written to temporary files and then applied at once when the final commit + * arrives. * * Unlike the regular (non-streamed) case, handling streamed transactions has * to handle aborts of both the toplevel transaction and subtransactions. This @@ -219,20 +239,7 @@ typedef struct ApplyExecutionData PartitionTupleRouting *proute; /* partition routing info */ } ApplyExecutionData; -/* Struct for saving and restoring apply errcontext information */ -typedef struct ApplyErrorCallbackArg -{ - LogicalRepMsgType command; /* 0 if invalid */ - LogicalRepRelMapEntry *rel; - - /* Remote node information */ - int remote_attnum; /* -1 if invalid */ - TransactionId remote_xid; - XLogRecPtr finish_lsn; - char *origin_name; -} ApplyErrorCallbackArg; - -static ApplyErrorCallbackArg apply_error_callback_arg = +ApplyErrorCallbackArg apply_error_callback_arg = { .command = 0, .rel = NULL, @@ -242,7 +249,7 @@ static ApplyErrorCallbackArg apply_error_callback_arg = .origin_name = NULL, }; -static MemoryContext ApplyMessageContext = NULL; +MemoryContext ApplyMessageContext = NULL; MemoryContext ApplyContext = NULL; /* per stream context for streaming transactions */ @@ -251,27 +258,38 @@ static MemoryContext LogicalStreamingContext = NULL; WalReceiverConn *LogRepWorkerWalRcvConn = NULL; Subscription *MySubscription = NULL; -static bool MySubscriptionValid = false; +bool MySubscriptionValid = false; bool in_remote_transaction = false; static XLogRecPtr remote_final_lsn = InvalidXLogRecPtr; /* fields valid only when processing streamed transaction */ -static bool in_streamed_transaction = false; +bool in_streamed_transaction = false; + +TransactionId stream_xid = InvalidTransactionId; +static ApplyBgworkerState *stream_apply_worker = NULL; + +/* check if we are applying the transaction in apply background worker */ +#define apply_bgworker_active() (in_streamed_transaction && stream_apply_worker != NULL) -static TransactionId stream_xid = InvalidTransactionId; +/* + * The number of changes during one streaming block (only for apply background + * workers) + */ +static uint32 nchanges = 0; /* * We enable skipping all data modification changes (INSERT, UPDATE, etc.) for * the subscription if the remote transaction's finish LSN matches the subskiplsn. * Once we start skipping changes, we don't stop it until we skip all changes of * the transaction even if pg_subscription is updated and MySubscription->skiplsn - * gets changed or reset during that. Also, in streaming transaction cases, we - * don't skip receiving and spooling the changes since we decide whether or not + * gets changed or reset during that. Also, in streaming transaction cases (streaming = on), + * we don't skip receiving and spooling the changes since we decide whether or not * to skip applying the changes when starting to apply changes. The subskiplsn is * cleared after successfully skipping the transaction or applying non-empty * transaction. The latter prevents the mistakenly specified subskiplsn from - * being left. + * being left. Note that we cannot skip the streaming transaction in apply mode, + * because we cannot get the finish LSN before applying the changes. */ static XLogRecPtr skip_xact_finish_lsn = InvalidXLogRecPtr; #define is_skipping_changes() (unlikely(!XLogRecPtrIsInvalid(skip_xact_finish_lsn))) @@ -324,9 +342,6 @@ static void maybe_reread_subscription(void); static void DisableSubscriptionAndExit(void); -/* prototype needed because of stream_commit */ -static void apply_dispatch(StringInfo s); - static void apply_handle_commit_internal(LogicalRepCommitData *commit_data); static void apply_handle_insert_internal(ApplyExecutionData *edata, ResultRelInfo *relinfo, @@ -359,7 +374,6 @@ static void stop_skipping_changes(void); static void clear_subscription_skip_lsn(XLogRecPtr finish_lsn); /* Functions for apply error callback */ -static void apply_error_callback(void *arg); static inline void set_apply_error_context_xact(TransactionId xid, XLogRecPtr lsn); static inline void reset_apply_error_context_info(void); @@ -426,41 +440,76 @@ end_replication_step(void) } /* - * Handle streamed transactions. + * Handle streamed transactions for both main apply worker and apply background + * worker. * - * If in streaming mode (receiving a block of streamed transaction), we - * simply redirect it to a file for the proper toplevel transaction. + * In streaming case (receiving a block of streamed transaction), for + * SUBSTREAM_ON mode, we simply redirect it to a file for the proper toplevel + * transaction, and for SUBSTREAM_APPLY mode, we send the changes to background + * apply worker (LOGICAL_REP_MSG_RELATION or LOGICAL_REP_MSG_TYPE changes will + * also be applied in main apply worker). * - * Returns true for streamed transactions, false otherwise (regular mode). + * For non-streamed transactions, returns false; + * For streamed transactions, returns true if in main apply worker (except we + * apply streamed transaction in "apply" mode and address + * LOGICAL_REP_MSG_RELATION or LOGICAL_REP_MSG_TYPE changes), false otherwise. */ static bool handle_streamed_transaction(LogicalRepMsgType action, StringInfo s) { - TransactionId xid; + TransactionId current_xid = InvalidTransactionId; /* not in streaming mode */ - if (!in_streamed_transaction) + if (!(in_streamed_transaction || am_apply_bgworker())) return false; - Assert(stream_fd != NULL); Assert(TransactionIdIsValid(stream_xid)); /* * We should have received XID of the subxact as the first part of the * message, so extract it. */ - xid = pq_getmsgint(s, 4); + current_xid = pq_getmsgint(s, 4); - if (!TransactionIdIsValid(xid)) + if (!TransactionIdIsValid(current_xid)) ereport(ERROR, (errcode(ERRCODE_PROTOCOL_VIOLATION), errmsg_internal("invalid transaction ID in streamed replication transaction"))); - /* Add the new subxact to the array (unless already there). */ - subxact_info_add(xid); + if (am_apply_bgworker()) + { + /* define a savepoint for a subxact if needed. */ + apply_bgworker_subxact_info_add(current_xid); + + return false; + } + else if (apply_bgworker_active()) + { + /* + * If we decided to apply the changes of this transaction in an apply + * background worker, pass the data to the worker. + */ + apply_bgworker_send_data(stream_apply_worker, s->len, s->data); + nchanges += 1; + + /* + * XXX The publisher side doesn't always send relation/type update + * messages after the streaming transaction, so also update the + * relation/type in main apply worker here. See function + * cleanup_rel_sync_cache. + */ + if (action == LOGICAL_REP_MSG_RELATION || + action == LOGICAL_REP_MSG_TYPE) + return false; + } + else + { + /* Add the new subxact to the array (unless already there). */ + subxact_info_add(current_xid); - /* write the change to the current file */ - stream_write_change(action, s); + /* write the change to the current file */ + stream_write_change(action, s); + } return true; } @@ -844,6 +893,9 @@ apply_handle_commit(StringInfo s) apply_handle_commit_internal(&commit_data); + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); + /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(commit_data.end_lsn); @@ -898,7 +950,9 @@ apply_handle_prepare_internal(LogicalRepPreparedTxnData *prepare_data) * BeginTransactionBlock is necessary to balance the EndTransactionBlock * called within the PrepareTransactionBlock below. */ - BeginTransactionBlock(); + if (!IsTransactionBlock()) + BeginTransactionBlock(); + CommitTransactionCommand(); /* Completes the preceding Begin command. */ /* @@ -950,6 +1004,9 @@ apply_handle_prepare(StringInfo s) in_remote_transaction = false; + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); + /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(prepare_data.end_lsn); @@ -1064,10 +1121,6 @@ apply_handle_rollback_prepared(StringInfo s) /* * Handle STREAM PREPARE. - * - * Logic is in two parts: - * 1. Replay all the spooled operations - * 2. Mark the transaction as prepared */ static void apply_handle_stream_prepare(StringInfo s) @@ -1088,24 +1141,70 @@ apply_handle_stream_prepare(StringInfo s) logicalrep_read_stream_prepare(s, &prepare_data); set_apply_error_context_xact(prepare_data.xid, prepare_data.prepare_lsn); - elog(DEBUG1, "received prepare for streamed transaction %u", prepare_data.xid); + if (am_apply_bgworker()) + { + /* Mark the transaction as prepared. */ + apply_handle_prepare_internal(&prepare_data); - /* Replay all the spooled operations. */ - apply_spooled_messages(prepare_data.xid, prepare_data.prepare_lsn); + CommitTransactionCommand(); - /* Mark the transaction as prepared. */ - apply_handle_prepare_internal(&prepare_data); + pgstat_report_stat(false); - CommitTransactionCommand(); + list_free(subxactlist); + subxactlist = NIL; - pgstat_report_stat(false); + apply_bgworker_set_status(APPLY_BGWORKER_FINISHED); + } + else + { + ApplyBgworkerState *wstate = apply_bgworker_find(prepare_data.xid); - store_flush_position(prepare_data.end_lsn); + elog(DEBUG1, "received prepare for streamed transaction %u", + prepare_data.xid); + + /* + * This is the main apply worker. Check if we are processing this + * transaction in a apply background worker. + */ + if (wstate) + { + apply_bgworker_send_data(wstate, s->len, s->data); + + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_FINISHED); + apply_bgworker_free(wstate); + + pgstat_report_stat(false); + store_flush_position(prepare_data.end_lsn); + } + else + { + /* + * This is the main apply worker and the transaction has been + * serialized to file, replay all the spooled operations. + */ + apply_spooled_messages(prepare_data.xid, prepare_data.prepare_lsn); + + /* Mark the transaction as prepared. */ + apply_handle_prepare_internal(&prepare_data); + + CommitTransactionCommand(); + + pgstat_report_stat(false); + + store_flush_position(prepare_data.end_lsn); + + in_remote_transaction = false; + + /* unlink the files with serialized changes and subxact info. */ + stream_cleanup_files(MyLogicalRepWorker->subid, prepare_data.xid); + } + } in_remote_transaction = false; + stream_apply_worker = NULL; - /* unlink the files with serialized changes and subxact info. */ - stream_cleanup_files(MyLogicalRepWorker->subid, prepare_data.xid); + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(prepare_data.end_lsn); @@ -1155,15 +1254,6 @@ apply_handle_stream_start(StringInfo s) (errcode(ERRCODE_PROTOCOL_VIOLATION), errmsg_internal("duplicate STREAM START message"))); - /* - * Start a transaction on stream start, this transaction will be committed - * on the stream stop unless it is a tablesync worker in which case it - * will be committed after processing all the messages. We need the - * transaction for handling the buffile, used for serializing the - * streaming data and subxact info. - */ - begin_replication_step(); - /* notify handle methods we're processing a remote transaction */ in_streamed_transaction = true; @@ -1177,36 +1267,95 @@ apply_handle_stream_start(StringInfo s) set_apply_error_context_xact(stream_xid, InvalidXLogRecPtr); - /* - * Initialize the worker's stream_fileset if we haven't yet. This will be - * used for the entire duration of the worker so create it in a permanent - * context. We create this on the very first streaming message from any - * transaction and then use it for this and other streaming transactions. - * Now, we could create a fileset at the start of the worker as well but - * then we won't be sure that it will ever be used. - */ - if (MyLogicalRepWorker->stream_fileset == NULL) + if (am_apply_bgworker()) { - MemoryContext oldctx; - - oldctx = MemoryContextSwitchTo(ApplyContext); + /* + * Make sure the handle apply_dispatch methods are aware we're in a + * remote transaction. + */ + in_remote_transaction = true; - MyLogicalRepWorker->stream_fileset = palloc(sizeof(FileSet)); - FileSetInit(MyLogicalRepWorker->stream_fileset); + /* If we are in a apply background worker, begin the transaction */ + AcceptInvalidationMessages(); + maybe_reread_subscription(); - MemoryContextSwitchTo(oldctx); + StartTransactionCommand(); + BeginTransactionBlock(); + CommitTransactionCommand(); } + else + { + /* + * This is the main apply worker. Check if there is any free apply + * background worker we can use to process this transaction. + */ + if (first_segment) + stream_apply_worker = apply_bgworker_start(stream_xid); + else + stream_apply_worker = apply_bgworker_find(stream_xid); - /* open the spool file for this transaction */ - stream_open_file(MyLogicalRepWorker->subid, stream_xid, first_segment); + if (stream_apply_worker) + { + /* + * If we have found a free worker or if we are already applying this + * transaction in an apply background worker, then we pass the data to + * that worker. + */ + if (first_segment) + apply_bgworker_send_data(stream_apply_worker, s->len, s->data); - /* if this is not the first segment, open existing subxact file */ - if (!first_segment) - subxact_info_read(MyLogicalRepWorker->subid, stream_xid); + nchanges = 0; + elog(DEBUG1, "starting streaming of xid %u", stream_xid); + } - pgstat_report_activity(STATE_RUNNING, NULL); + /* + * If no worker is available for the first stream start, we start to + * serialize all the changes of the transaction. + */ + else + { + /* + * Start a transaction on stream start, this transaction will be + * committed on the stream stop unless it is a tablesync worker in + * which case it will be committed after processing all the messages. + * We need the transaction for handling the buffile, used for + * serializing the streaming data and subxact info. + */ + begin_replication_step(); - end_replication_step(); + /* + * Initialize the worker's stream_fileset if we haven't yet. This will + * be used for the entire duration of the worker so create it in a + * permanent context. We create this on the very first streaming + * message from any transaction and then use it for this and other + * streaming transactions. Now, we could create a fileset at the start + * of the worker as well but then we won't be sure that it will ever + * be used. + */ + if (MyLogicalRepWorker->stream_fileset == NULL) + { + MemoryContext oldctx; + + oldctx = MemoryContextSwitchTo(ApplyContext); + + MyLogicalRepWorker->stream_fileset = palloc(sizeof(FileSet)); + FileSetInit(MyLogicalRepWorker->stream_fileset); + + MemoryContextSwitchTo(oldctx); + } + + /* open the spool file for this transaction */ + stream_open_file(MyLogicalRepWorker->subid, stream_xid, first_segment); + + /* if this is not the first segment, open existing subxact file */ + if (!first_segment) + subxact_info_read(MyLogicalRepWorker->subid, stream_xid); + + end_replication_step(); + } + } + + pgstat_report_activity(STATE_RUNNING, NULL); } /* @@ -1220,53 +1369,52 @@ apply_handle_stream_stop(StringInfo s) (errcode(ERRCODE_PROTOCOL_VIOLATION), errmsg_internal("STREAM STOP message without STREAM START"))); - /* - * Close the file with serialized changes, and serialize information about - * subxacts for the toplevel transaction. - */ - subxact_info_write(MyLogicalRepWorker->subid, stream_xid); - stream_close_file(); + if (apply_bgworker_active()) + { + char action = LOGICAL_REP_MSG_STREAM_STOP; - /* We must be in a valid transaction state */ - Assert(IsTransactionState()); + apply_bgworker_send_data(stream_apply_worker, 1, &action); - /* Commit the per-stream transaction */ - CommitTransactionCommand(); + elog(DEBUG1, "stopped streaming of xid %u, %u changes streamed", stream_xid, nchanges); + } + else + { + /* + * Close the file with serialized changes, and serialize information + * about subxacts for the toplevel transaction. + */ + subxact_info_write(MyLogicalRepWorker->subid, stream_xid); + stream_close_file(); - in_streamed_transaction = false; + /* We must be in a valid transaction state */ + Assert(IsTransactionState()); + + /* Commit the per-stream transaction */ + CommitTransactionCommand(); - /* Reset per-stream context */ - MemoryContextReset(LogicalStreamingContext); + /* Reset per-stream context */ + MemoryContextReset(LogicalStreamingContext); + } + + in_streamed_transaction = false; + stream_apply_worker = NULL; pgstat_report_activity(STATE_IDLE, NULL); reset_apply_error_context_info(); } /* - * Handle STREAM abort message. + * Handle STREAM ABORT message when the transaction was spilled to disk. */ static void -apply_handle_stream_abort(StringInfo s) +serialize_stream_abort(TransactionId xid, TransactionId subxid) { - TransactionId xid; - TransactionId subxid; - - if (in_streamed_transaction) - ereport(ERROR, - (errcode(ERRCODE_PROTOCOL_VIOLATION), - errmsg_internal("STREAM ABORT message without STREAM STOP"))); - - logicalrep_read_stream_abort(s, &xid, &subxid); - /* * If the two XIDs are the same, it's in fact abort of toplevel xact, so * just delete the files with serialized info. */ if (xid == subxid) - { - set_apply_error_context_xact(xid, InvalidXLogRecPtr); stream_cleanup_files(MyLogicalRepWorker->subid, xid); - } else { /* @@ -1290,8 +1438,6 @@ apply_handle_stream_abort(StringInfo s) bool found = false; char path[MAXPGPATH]; - set_apply_error_context_xact(subxid, InvalidXLogRecPtr); - subidx = -1; begin_replication_step(); subxact_info_read(MyLogicalRepWorker->subid, xid); @@ -1316,7 +1462,6 @@ apply_handle_stream_abort(StringInfo s) cleanup_subxact_info(); end_replication_step(); CommitTransactionCommand(); - reset_apply_error_context_info(); return; } @@ -1339,6 +1484,133 @@ apply_handle_stream_abort(StringInfo s) end_replication_step(); CommitTransactionCommand(); } +} + +/* + * Handle STREAM ABORT message. + */ +static void +apply_handle_stream_abort(StringInfo s) +{ + TransactionId xid; + TransactionId subxid; + LogicalRepStreamAbortData abort_data; + bool include_abort_lsn = false; + + if (in_streamed_transaction) + ereport(ERROR, + (errcode(ERRCODE_PROTOCOL_VIOLATION), + errmsg_internal("STREAM ABORT message without STREAM STOP"))); + + /* Check whether the publisher sends abort_lsn and abort_time. */ + if (am_apply_bgworker()) + include_abort_lsn = MyParallelState->server_version >= 150000; + + logicalrep_read_stream_abort(s, &abort_data, include_abort_lsn); + + xid = abort_data.xid; + subxid = abort_data.subxid; + + set_apply_error_context_xact(subxid, abort_data.abort_lsn); + + if (am_apply_bgworker()) + { + elog(DEBUG1, "[Apply BGW #%u] aborting current transaction xid=%u, subxid=%u", + MyParallelState->n, GetCurrentTransactionIdIfAny(), + GetCurrentSubTransactionId()); + + /* + * Update origin state so we can restart streaming from correct + * position in case of crash. + */ + if (include_abort_lsn) + { + replorigin_session_origin_lsn = abort_data.abort_lsn; + replorigin_session_origin_timestamp = abort_data.abort_time; + } + + /* + * If the two XIDs are the same, it's in fact abort of toplevel xact, + * so just free the subxactlist. + */ + if (subxid == xid) + { + AbortCurrentTransaction(); + + EndTransactionBlock(false); + CommitTransactionCommand(); + + in_remote_transaction = false; + pgstat_report_activity(STATE_IDLE, NULL); + + list_free(subxactlist); + subxactlist = NIL; + + apply_bgworker_set_status(APPLY_BGWORKER_FINISHED); + } + else + { + /* + * OK, so it's a subxact. Rollback to the savepoint. + * + * We also need to read the subxactlist, determine the offset + * tracked for the subxact, and truncate the list. + */ + int i; + bool found = false; + char spname[MAXPGPATH]; + + snprintf(spname, MAXPGPATH, "savepoint_for_xid_%u", subxid); + + elog(DEBUG1, "[Apply BGW #%u] rolling back to savepoint %s", + MyParallelState->n, spname); + + for (i = list_length(subxactlist) - 1; i >= 0; i--) + { + xid = (TransactionId) list_nth_int(subxactlist, i); + if (xid == subxid) + { + found = true; + break; + } + } + + if (found) + { + RollbackToSavepoint(spname); + CommitTransactionCommand(); + subxactlist = list_truncate(subxactlist, i + 1); + } + + pgstat_report_activity(STATE_IDLEINTRANSACTION, NULL); + } + } + else + { + ApplyBgworkerState *wstate = apply_bgworker_find(xid); + + /* + * This is the main apply worker. Check if we are processing this + * transaction in a apply background worker. + */ + if (wstate) + { + apply_bgworker_send_data(wstate, s->len, s->data); + + if (subxid == xid) + { + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_FINISHED); + apply_bgworker_free(wstate); + } + } + + /* + * We are in main apply worker and the transaction has been serialized + * to file. + */ + else + serialize_stream_abort(xid, subxid); + } reset_apply_error_context_info(); } @@ -1462,40 +1734,6 @@ apply_spooled_messages(TransactionId xid, XLogRecPtr lsn) return; } -/* - * Handle STREAM COMMIT message. - */ -static void -apply_handle_stream_commit(StringInfo s) -{ - TransactionId xid; - LogicalRepCommitData commit_data; - - if (in_streamed_transaction) - ereport(ERROR, - (errcode(ERRCODE_PROTOCOL_VIOLATION), - errmsg_internal("STREAM COMMIT message without STREAM STOP"))); - - xid = logicalrep_read_stream_commit(s, &commit_data); - set_apply_error_context_xact(xid, commit_data.commit_lsn); - - elog(DEBUG1, "received commit for streamed transaction %u", xid); - - apply_spooled_messages(xid, commit_data.commit_lsn); - - apply_handle_commit_internal(&commit_data); - - /* unlink the files with serialized changes and subxact info */ - stream_cleanup_files(MyLogicalRepWorker->subid, xid); - - /* Process any tables that are being synchronized in parallel. */ - process_syncing_tables(commit_data.end_lsn); - - pgstat_report_activity(STATE_IDLE, NULL); - - reset_apply_error_context_info(); -} - /* * Helper function for apply_handle_commit and apply_handle_stream_commit. */ @@ -2463,11 +2701,109 @@ apply_handle_truncate(StringInfo s) end_replication_step(); } +/* + * Handle STREAM COMMIT message. + */ +static void +apply_handle_stream_commit(StringInfo s) +{ + LogicalRepCommitData commit_data; + TransactionId xid; + + if (in_streamed_transaction) + ereport(ERROR, + (errcode(ERRCODE_PROTOCOL_VIOLATION), + errmsg_internal("STREAM COMMIT message without STREAM STOP"))); + + xid = logicalrep_read_stream_commit(s, &commit_data); + set_apply_error_context_xact(xid, commit_data.commit_lsn); + + if (am_apply_bgworker()) + { + /* + * Update origin state so we can restart streaming from correct + * position in case of crash. + */ + replorigin_session_origin_lsn = commit_data.end_lsn; + replorigin_session_origin_timestamp = commit_data.committime; + + CommitTransactionCommand(); + EndTransactionBlock(false); + CommitTransactionCommand(); + + in_remote_transaction = false; + + pgstat_report_stat(false); + + list_free(subxactlist); + subxactlist = NIL; + + apply_bgworker_set_status(APPLY_BGWORKER_FINISHED); + } + else + { + /* + * This is the main apply worker. Check if we are processing this + * transaction in an apply background worker. + */ + ApplyBgworkerState *wstate = apply_bgworker_find(xid); + + elog(DEBUG1, "received commit for streamed transaction %u", xid); + + if (wstate) + { + /* Send commit message */ + apply_bgworker_send_data(wstate, s->len, s->data); + + /* + * Wait for apply background worker to finish. This is required to + * maintain commit order which avoids failures due to transaction + * dependencies and deadlocks. + */ + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_FINISHED); + + pgstat_report_stat(false); + store_flush_position(commit_data.end_lsn); + stop_skipping_changes(); + + apply_bgworker_free(wstate); + + /* + * The transaction is either non-empty or skipped, so we clear the + * subskiplsn. + */ + clear_subscription_skip_lsn(commit_data.commit_lsn); + } + else + { + /* + * This is the main apply worker and the transaction has been + * serialized to file, replay all the spooled operations. + */ + apply_spooled_messages(xid, commit_data.commit_lsn); + + apply_handle_commit_internal(&commit_data); + + /* unlink the files with serialized changes and subxact info */ + stream_cleanup_files(MyLogicalRepWorker->subid, xid); + } + } + + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); + + /* Process any tables that are being synchronized in parallel. */ + process_syncing_tables(commit_data.end_lsn); + + pgstat_report_activity(STATE_IDLE, NULL); + + reset_apply_error_context_info(); +} /* * Logical replication protocol message dispatcher. */ -static void +void apply_dispatch(StringInfo s) { LogicalRepMsgType action = pq_getmsgbyte(s); @@ -2636,6 +2972,10 @@ store_flush_position(XLogRecPtr remote_lsn) { FlushPosition *flushpos; + /* We only need to collect the LSN in main apply worker */ + if (am_apply_bgworker()) + return; + /* Need to do this in permanent context */ MemoryContextSwitchTo(ApplyContext); @@ -2650,7 +2990,7 @@ store_flush_position(XLogRecPtr remote_lsn) /* Update statistics of the worker. */ -static void +void UpdateWorkerStats(XLogRecPtr last_lsn, TimestampTz send_time, bool reply) { MyLogicalRepWorker->last_lsn = last_lsn; @@ -2812,6 +3152,9 @@ LogicalRepApplyLoop(XLogRecPtr last_received) AcceptInvalidationMessages(); maybe_reread_subscription(); + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); + /* Process any table synchronization changes. */ process_syncing_tables(last_received); } @@ -3113,7 +3456,7 @@ maybe_reread_subscription(void) /* * Callback from subscription syscache invalidation. */ -static void +void subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue) { MySubscriptionValid = false; @@ -3709,7 +4052,7 @@ ApplyWorkerMain(Datum main_arg) originid = replorigin_by_name(originname, true); if (!OidIsValid(originid)) originid = replorigin_create(originname); - replorigin_session_setup(originid); + replorigin_session_setup(originid, true); replorigin_session_origin = originid; origin_startpos = replorigin_session_get_progress(false); CommitTransactionCommand(); @@ -3756,7 +4099,7 @@ ApplyWorkerMain(Datum main_arg) options.proto.logical.publication_names = MySubscription->publications; options.proto.logical.binary = MySubscription->binary; - options.proto.logical.streaming = MySubscription->stream; + options.proto.logical.streaming = (MySubscription->stream != SUBSTREAM_OFF); options.proto.logical.twophase = false; if (!am_tablesync_worker()) @@ -3914,7 +4257,8 @@ clear_subscription_skip_lsn(XLogRecPtr finish_lsn) XLogRecPtr myskiplsn = MySubscription->skiplsn; bool started_tx = false; - if (likely(XLogRecPtrIsInvalid(myskiplsn))) + if (likely(XLogRecPtrIsInvalid(myskiplsn)) || + am_apply_bgworker()) return; if (!IsTransactionState()) @@ -3986,7 +4330,7 @@ clear_subscription_skip_lsn(XLogRecPtr finish_lsn) } /* Error callback to give more context info about the change being applied */ -static void +void apply_error_callback(void *arg) { ApplyErrorCallbackArg *errarg = &apply_error_callback_arg; diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c index 8deae57143..6ebfa24baf 100644 --- a/src/backend/replication/pgoutput/pgoutput.c +++ b/src/backend/replication/pgoutput/pgoutput.c @@ -1833,7 +1833,7 @@ pgoutput_stream_abort(struct LogicalDecodingContext *ctx, Assert(rbtxn_is_streamed(toptxn)); OutputPluginPrepareWrite(ctx, true); - logicalrep_write_stream_abort(ctx->out, toptxn->xid, txn->xid); + logicalrep_write_stream_abort(ctx->out, toptxn->xid, txn, abort_lsn); OutputPluginWrite(ctx, true); cleanup_rel_sync_cache(toptxn->xid, false); diff --git a/src/backend/utils/activity/wait_event.c b/src/backend/utils/activity/wait_event.c index 87c15b9c6f..ba781e6f08 100644 --- a/src/backend/utils/activity/wait_event.c +++ b/src/backend/utils/activity/wait_event.c @@ -388,6 +388,9 @@ pgstat_get_wait_ipc(WaitEventIPC w) case WAIT_EVENT_HASH_GROW_BUCKETS_REINSERT: event_name = "HashGrowBucketsReinsert"; break; + case WAIT_EVENT_LOGICAL_APPLY_WORKER_STATE_CHANGE: + event_name = "LogicalApplyWorkerStateChange"; + break; case WAIT_EVENT_LOGICAL_SYNC_DATA: event_name = "LogicalSyncData"; break; diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c index a7cc49898b..d05e1d2de0 100644 --- a/src/backend/utils/misc/guc.c +++ b/src/backend/utils/misc/guc.c @@ -3220,6 +3220,18 @@ static struct config_int ConfigureNamesInt[] = NULL, NULL, NULL }, + { + {"max_apply_bgworkers_per_subscription", + PGC_SIGHUP, + REPLICATION_SUBSCRIBERS, + gettext_noop("Maximum number of apply backgrand workers per subscription."), + NULL, + }, + &max_apply_bgworkers_per_subscription, + 2, 0, MAX_BACKENDS, + NULL, NULL, NULL + }, + { {"log_rotation_age", PGC_SIGHUP, LOGGING_WHERE, gettext_noop("Sets the amount of time to wait before forcing " diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample index 48ad80cf2e..292808dd83 100644 --- a/src/backend/utils/misc/postgresql.conf.sample +++ b/src/backend/utils/misc/postgresql.conf.sample @@ -360,6 +360,7 @@ #max_logical_replication_workers = 4 # taken from max_worker_processes # (change requires restart) #max_sync_workers_per_subscription = 2 # taken from max_logical_replication_workers +#max_apply_bgworkers_per_subscription = 2 # taken from max_logical_replication_workers #------------------------------------------------------------------------------ diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 7cc9c72e49..6f8b30abb0 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -4450,7 +4450,7 @@ getSubscriptions(Archive *fout) if (fout->remoteVersion >= 140000) appendPQExpBufferStr(query, " s.substream,\n"); else - appendPQExpBufferStr(query, " false AS substream,\n"); + appendPQExpBufferStr(query, " 'f' AS substream,\n"); if (fout->remoteVersion >= 150000) appendPQExpBufferStr(query, @@ -4580,8 +4580,10 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo) if (strcmp(subinfo->subbinary, "t") == 0) appendPQExpBufferStr(query, ", binary = true"); - if (strcmp(subinfo->substream, "f") != 0) + if (strcmp(subinfo->substream, "t") == 0) appendPQExpBufferStr(query, ", streaming = on"); + else if (strcmp(subinfo->substream, "a") == 0) + appendPQExpBufferStr(query, ", streaming = apply"); if (strcmp(subinfo->subtwophasestate, two_phase_disabled) != 0) appendPQExpBufferStr(query, ", two_phase = on"); diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h index d1260f590c..9b394a45fe 100644 --- a/src/include/catalog/pg_subscription.h +++ b/src/include/catalog/pg_subscription.h @@ -68,7 +68,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW bool subbinary; /* True if the subscription wants the * publisher to send data in binary */ - bool substream; /* Stream in-progress transactions. */ + char substream; /* Stream in-progress transactions. + * See SUBSTREAM_xxx constants. */ char subtwophasestate; /* Stream two-phase transactions */ @@ -109,7 +110,7 @@ typedef struct Subscription bool enabled; /* Indicates if the subscription is enabled */ bool binary; /* Indicates if the subscription wants data in * binary format */ - bool stream; /* Allow streaming in-progress transactions. */ + char stream; /* Allow streaming in-progress transactions. */ char twophasestate; /* Allow streaming two-phase transactions */ bool disableonerr; /* Indicates if the subscription should be * automatically disabled if a worker error @@ -120,6 +121,18 @@ typedef struct Subscription List *publications; /* List of publication names to subscribe to */ } Subscription; +/* Disallow streaming in-progress transactions */ +#define SUBSTREAM_OFF 'f' + +/* + * Streaming transactions are written to a temporary file and applied only + * after the transaction is committed on upstream. + */ +#define SUBSTREAM_ON 't' + +/* Streaming transactions are applied immediately via a background worker */ +#define SUBSTREAM_APPLY 'a' + extern Subscription *GetSubscription(Oid subid, bool missing_ok); extern void FreeSubscription(Subscription *sub); extern void DisableSubscription(Oid subid); diff --git a/src/include/replication/logicallauncher.h b/src/include/replication/logicallauncher.h index f1e2821e25..ac8ef94381 100644 --- a/src/include/replication/logicallauncher.h +++ b/src/include/replication/logicallauncher.h @@ -14,6 +14,7 @@ extern PGDLLIMPORT int max_logical_replication_workers; extern PGDLLIMPORT int max_sync_workers_per_subscription; +extern PGDLLIMPORT int max_apply_bgworkers_per_subscription; extern void ApplyLauncherRegister(void); extern void ApplyLauncherMain(Datum main_arg); diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h index a771ab8ff3..7bba77c9e7 100644 --- a/src/include/replication/logicalproto.h +++ b/src/include/replication/logicalproto.h @@ -175,6 +175,17 @@ typedef struct LogicalRepRollbackPreparedTxnData char gid[GIDSIZE]; } LogicalRepRollbackPreparedTxnData; +/* + * Transaction protocol information for stream abort. + */ +typedef struct LogicalRepStreamAbortData +{ + TransactionId xid; + TransactionId subxid; + XLogRecPtr abort_lsn; + TimestampTz abort_time; +} LogicalRepStreamAbortData; + extern void logicalrep_write_begin(StringInfo out, ReorderBufferTXN *txn); extern void logicalrep_read_begin(StringInfo in, LogicalRepBeginData *begin_data); @@ -246,9 +257,11 @@ extern void logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn extern TransactionId logicalrep_read_stream_commit(StringInfo out, LogicalRepCommitData *commit_data); extern void logicalrep_write_stream_abort(StringInfo out, TransactionId xid, - TransactionId subxid); -extern void logicalrep_read_stream_abort(StringInfo in, TransactionId *xid, - TransactionId *subxid); + ReorderBufferTXN *txn, + XLogRecPtr abort_lsn); +extern void logicalrep_read_stream_abort(StringInfo in, + LogicalRepStreamAbortData *abort_data, + bool include_abort_lsn); extern char *logicalrep_message_type(LogicalRepMsgType action); #endif /* LOGICAL_PROTO_H */ diff --git a/src/include/replication/logicalworker.h b/src/include/replication/logicalworker.h index cd1b6e8afc..6a1af7f13c 100644 --- a/src/include/replication/logicalworker.h +++ b/src/include/replication/logicalworker.h @@ -13,6 +13,7 @@ #define LOGICALWORKER_H extern void ApplyWorkerMain(Datum main_arg); +extern void ApplyBgworkerMain(Datum main_arg); extern bool IsLogicalWorker(void); diff --git a/src/include/replication/origin.h b/src/include/replication/origin.h index 2d1b5e5c28..c7389b40a7 100644 --- a/src/include/replication/origin.h +++ b/src/include/replication/origin.h @@ -53,7 +53,7 @@ extern XLogRecPtr replorigin_get_progress(RepOriginId node, bool flush); extern void replorigin_session_advance(XLogRecPtr remote_commit, XLogRecPtr local_commit); -extern void replorigin_session_setup(RepOriginId node); +extern void replorigin_session_setup(RepOriginId node, bool must_acquire); extern void replorigin_session_reset(void); extern XLogRecPtr replorigin_session_get_progress(bool flush); diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h index 4a01f877e5..aba1640f4d 100644 --- a/src/include/replication/reorderbuffer.h +++ b/src/include/replication/reorderbuffer.h @@ -301,6 +301,7 @@ typedef struct ReorderBufferTXN { TimestampTz commit_time; TimestampTz prepare_time; + TimestampTz abort_time; } xact_time; /* @@ -647,9 +648,11 @@ extern void ReorderBufferFinishPrepared(ReorderBuffer *rb, TransactionId xid, extern void ReorderBufferAssignChild(ReorderBuffer *, TransactionId, TransactionId, XLogRecPtr commit_lsn); extern void ReorderBufferCommitChild(ReorderBuffer *, TransactionId, TransactionId, XLogRecPtr commit_lsn, XLogRecPtr end_lsn); -extern void ReorderBufferAbort(ReorderBuffer *, TransactionId, XLogRecPtr lsn); +extern void ReorderBufferAbort(ReorderBuffer *, TransactionId, XLogRecPtr lsn, + TimestampTz abort_time); extern void ReorderBufferAbortOld(ReorderBuffer *, TransactionId xid); -extern void ReorderBufferForget(ReorderBuffer *, TransactionId, XLogRecPtr lsn); +extern void ReorderBufferForget(ReorderBuffer *, TransactionId, XLogRecPtr lsn, + TimestampTz abort_time); extern void ReorderBufferInvalidate(ReorderBuffer *, TransactionId, XLogRecPtr lsn); extern void ReorderBufferSetBaseSnapshot(ReorderBuffer *, TransactionId, XLogRecPtr lsn, struct SnapshotData *snap); diff --git a/src/include/replication/worker_internal.h b/src/include/replication/worker_internal.h index 901845abc2..73f728dc27 100644 --- a/src/include/replication/worker_internal.h +++ b/src/include/replication/worker_internal.h @@ -17,8 +17,11 @@ #include "access/xlogdefs.h" #include "catalog/pg_subscription.h" #include "datatype/timestamp.h" +#include "replication/logicalrelation.h" #include "storage/fileset.h" #include "storage/lock.h" +#include "storage/shm_mq.h" +#include "storage/shm_toc.h" #include "storage/spin.h" @@ -60,6 +63,9 @@ typedef struct LogicalRepWorker */ FileSet *stream_fileset; + /* Indicates if this slot is used for an apply background worker. */ + bool subworker; + /* Stats. */ XLogRecPtr last_lsn; TimestampTz last_send_time; @@ -68,9 +74,68 @@ typedef struct LogicalRepWorker TimestampTz reply_time; } LogicalRepWorker; +/* Struct for saving and restoring apply errcontext information */ +typedef struct ApplyErrorCallbackArg +{ + LogicalRepMsgType command; /* 0 if invalid */ + LogicalRepRelMapEntry *rel; + + /* Remote node information */ + int remote_attnum; /* -1 if invalid */ + TransactionId remote_xid; + XLogRecPtr finish_lsn; + char *origin_name; +} ApplyErrorCallbackArg; + +/* + * Status for apply background worker. + */ +typedef enum ApplyBgworkerStatus +{ + APPLY_BGWORKER_BUSY = 0, /* assigned to a transaction */ + APPLY_BGWORKER_FINISHED, /* transaction is completed */ + APPLY_BGWORKER_EXIT /* exit */ +} ApplyBgworkerStatus; + +/* + * Shared information among apply workers. + */ +typedef struct ApplyBgworkerShared +{ + slock_t mutex; + + /* Status for apply background worker. */ + ApplyBgworkerStatus status; + + /* server version of publisher. */ + int server_version; + + TransactionId stream_xid; + uint32 n; /* id of apply background worker */ +} ApplyBgworkerShared; + +/* + * Struct for maintaining an apply background worker. + */ +typedef struct ApplyBgworkerState +{ + shm_mq_handle *mq_handle; + dsm_segment *dsm_seg; + ApplyBgworkerShared volatile *pstate; +} ApplyBgworkerState; + /* Main memory context for apply worker. Permanent during worker lifetime. */ extern PGDLLIMPORT MemoryContext ApplyContext; +extern PGDLLIMPORT MemoryContext ApplyMessageContext; + +extern PGDLLIMPORT ApplyErrorCallbackArg apply_error_callback_arg; + +extern PGDLLIMPORT bool MySubscriptionValid; + +extern PGDLLIMPORT volatile ApplyBgworkerShared *MyParallelState; +extern PGDLLIMPORT List *subxactlist; + /* libpqreceiver connection */ extern PGDLLIMPORT struct WalReceiverConn *LogRepWorkerWalRcvConn; @@ -79,18 +144,22 @@ extern PGDLLIMPORT Subscription *MySubscription; extern PGDLLIMPORT LogicalRepWorker *MyLogicalRepWorker; extern PGDLLIMPORT bool in_remote_transaction; +extern PGDLLIMPORT bool in_streamed_transaction; +extern PGDLLIMPORT TransactionId stream_xid; extern void logicalrep_worker_attach(int slot); extern LogicalRepWorker *logicalrep_worker_find(Oid subid, Oid relid, bool only_running); extern List *logicalrep_workers_find(Oid subid, bool only_running); -extern void logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, - Oid userid, Oid relid); +extern bool logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, + Oid userid, Oid relid, + dsm_handle subworker_dsm); extern void logicalrep_worker_stop(Oid subid, Oid relid); extern void logicalrep_worker_wakeup(Oid subid, Oid relid); extern void logicalrep_worker_wakeup_ptr(LogicalRepWorker *worker); extern int logicalrep_sync_worker_count(Oid subid); +extern int logicalrep_apply_background_worker_count(Oid subid); extern void ReplicationOriginNameForTablesync(Oid suboid, Oid relid, char *originname, int szorgname); @@ -103,10 +172,39 @@ extern void process_syncing_tables(XLogRecPtr current_lsn); extern void invalidate_syncing_table_states(Datum arg, int cacheid, uint32 hashvalue); +extern void UpdateWorkerStats(XLogRecPtr last_lsn, TimestampTz send_time, + bool reply); + +/* prototype needed because of stream_commit */ +extern void apply_dispatch(StringInfo s); + +/* Function for apply error callback */ +extern void apply_error_callback(void *arg); + +extern void subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue); + +/* apply background worker setup and interactions */ +extern ApplyBgworkerState *apply_bgworker_start(TransactionId xid); +extern ApplyBgworkerState *apply_bgworker_find(TransactionId xid); +extern void apply_bgworker_wait_for(ApplyBgworkerState *wstate, + ApplyBgworkerStatus wait_for_status); +extern void apply_bgworker_send_data(ApplyBgworkerState *wstate, Size nbytes, + const void *data); +extern void apply_bgworker_free(ApplyBgworkerState *wstate); +extern void apply_bgworker_check_status(void); +extern void apply_bgworker_set_status(ApplyBgworkerStatus status); +extern void apply_bgworker_subxact_info_add(TransactionId current_xid); + static inline bool am_tablesync_worker(void) { return OidIsValid(MyLogicalRepWorker->relid); } +static inline bool +am_apply_bgworker(void) +{ + return MyLogicalRepWorker->subworker; +} + #endif /* WORKER_INTERNAL_H */ diff --git a/src/include/utils/wait_event.h b/src/include/utils/wait_event.h index b578e2ec75..c2d2a114d7 100644 --- a/src/include/utils/wait_event.h +++ b/src/include/utils/wait_event.h @@ -105,6 +105,7 @@ typedef enum WAIT_EVENT_HASH_GROW_BUCKETS_ALLOCATE, WAIT_EVENT_HASH_GROW_BUCKETS_ELECT, WAIT_EVENT_HASH_GROW_BUCKETS_REINSERT, + WAIT_EVENT_LOGICAL_APPLY_WORKER_STATE_CHANGE, WAIT_EVENT_LOGICAL_SYNC_DATA, WAIT_EVENT_LOGICAL_SYNC_STATE_CHANGE, WAIT_EVENT_MQ_INTERNAL, diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out index 5db7146e06..b3c34ce10f 100644 --- a/src/test/regress/expected/subscription.out +++ b/src/test/regress/expected/subscription.out @@ -197,7 +197,7 @@ ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); DROP SUBSCRIPTION regress_testsub; -- fail - streaming must be boolean CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = foo); -ERROR: streaming requires a Boolean value +ERROR: streaming requires a Boolean value or "apply" -- now it works CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = true); WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 4fb746930a..664d4b8b1a 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -116,6 +116,10 @@ Append AppendPath AppendRelInfo AppendState +ApplyBgworkerEntry +ApplyBgworkerShared +ApplyBgworkerState +ApplyBgworkerStatus ApplyErrorCallbackArg ApplyExecutionData ApplySubXactData @@ -1483,6 +1487,7 @@ LogicalRepRelId LogicalRepRelMapEntry LogicalRepRelation LogicalRepRollbackPreparedTxnData +LogicalRepStreamAbortData LogicalRepTupleData LogicalRepTyp LogicalRepWorker -- 2.23.0.windows.1 [application/octet-stream] v13-0002-Test-streaming-apply-option-in-tap-test.patch (68.9K, ../../OS3PR01MB6275F61CC74C5F3E47C7DD229EB59@OS3PR01MB6275.jpnprd01.prod.outlook.com/3-v13-0002-Test-streaming-apply-option-in-tap-test.patch) download | inline diff: From ffdce8aa319fa66fd126414adca697c2e32cb74b Mon Sep 17 00:00:00 2001 From: "shiy.fnst" <[email protected]> Date: Fri, 13 May 2022 14:50:30 +0800 Subject: [PATCH v13 2/4] Test streaming apply option in tap test Change all TAP tests using the SUBSCRIPTION "streaming" option, so they now test both 'on' and 'apply' values. --- src/test/subscription/t/015_stream.pl | 199 ++++--- src/test/subscription/t/016_stream_subxact.pl | 119 +++-- src/test/subscription/t/017_stream_ddl.pl | 188 ++++--- .../t/018_stream_subxact_abort.pl | 195 ++++--- .../t/019_stream_subxact_ddl_abort.pl | 110 +++- .../subscription/t/022_twophase_cascade.pl | 363 +++++++------ .../subscription/t/023_twophase_stream.pl | 498 ++++++++++-------- 7 files changed, 1035 insertions(+), 637 deletions(-) diff --git a/src/test/subscription/t/015_stream.pl b/src/test/subscription/t/015_stream.pl index 6561b189de..b3d84b1c7c 100644 --- a/src/test/subscription/t/015_stream.pl +++ b/src/test/subscription/t/015_stream.pl @@ -8,6 +8,116 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" option +# so the same code can be run both for the streaming=on and streaming=apply +# cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_apply) = @_; + + # Interleave a pair of transactions, each exceeding the 64kB limit. + my $in = ''; + my $out = ''; + + my $timer = IPC::Run::timeout($PostgreSQL::Test::Utils::timeout_default); + + my $h = $node_publisher->background_psql('postgres', \$in, \$out, $timer, + on_error_stop => 0); + + # Check that a background worker starts if "streaming" option is + # specified as "apply". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_apply) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + $in .= q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + }; + $h->pump_nb; + + if ($is_apply) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001, 9999) s(i); + DELETE FROM test_tab WHERE a > 5000; + COMMIT; + }); + + $in .= q{ + COMMIT; + \q + }; + $h->finish; # errors make the next test fail, so ignore them here + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), + 'check extra columns contain local defaults'); + + # Test the streaming in binary mode + $node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET (binary = on)"); + + # Insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001, 10000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + COMMIT; + }); + + $node_publisher->wait_for_catchup($appname); + + $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(6667|6667|6667), + 'check extra columns contain local defaults'); + + # Change the local values of the extra columns on the subscriber, + # update publisher, and check that subscriber retains the expected + # values. This is to ensure that non-streaming transactions behave + # properly after a streaming transaction. + $node_subscriber->safe_psql('postgres', + "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'" + ); + $node_publisher->safe_psql('postgres', + "UPDATE test_tab SET b = md5(a::text)"); + + $node_publisher->wait_for_catchup($appname); + + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab" + ); + is($result, qq(6667|6667|6667), + 'check extra columns contain locally changed data'); + + # Cleanup the test data + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE (a > 2)"); + $node_publisher->wait_for_catchup($appname); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -37,6 +147,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -54,82 +168,25 @@ my $result = "SELECT count(*), count(c), count(d = 999) FROM test_tab"); is($result, qq(2|2|2), 'check initial data was copied to subscriber'); -# Interleave a pair of transactions, each exceeding the 64kB limit. -my $in = ''; -my $out = ''; - -my $timer = IPC::Run::timeout($PostgreSQL::Test::Utils::timeout_default); - -my $h = $node_publisher->background_psql('postgres', \$in, \$out, $timer, - on_error_stop => 0); - -$in .= q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -}; -$h->pump_nb; - -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001, 9999) s(i); -DELETE FROM test_tab WHERE a > 5000; -COMMIT; -}); - -$in .= q{ -COMMIT; -\q -}; -$h->finish; # errors make the next test fail, so ignore them here - -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), 'check extra columns contain local defaults'); - -# Test the streaming in binary mode -$node_subscriber->safe_psql('postgres', - "ALTER SUBSCRIPTION tap_sub SET (binary = on)"); - -# Insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001, 10000) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -COMMIT; -}); - -$node_publisher->wait_for_catchup($appname); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(6667|6667|6667), 'check extra columns contain local defaults'); +################################### +# Test using streaming mode 'apply' +################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" +); -# Change the local values of the extra columns on the subscriber, -# update publisher, and check that subscriber retains the expected -# values. This is to ensure that non-streaming transactions behave -# properly after a streaming transaction. $node_subscriber->safe_psql('postgres', - "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'" -); -$node_publisher->safe_psql('postgres', - "UPDATE test_tab SET b = md5(a::text)"); + "ALTER SUBSCRIPTION tap_sub SET(streaming = apply, binary = off)"); -$node_publisher->wait_for_catchup($appname); +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab" -); -is($result, qq(6667|6667|6667), - 'check extra columns contain locally changed data'); +test_streaming($node_publisher, $node_subscriber, $appname, 1); $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/016_stream_subxact.pl b/src/test/subscription/t/016_stream_subxact.pl index f27f1694f2..b5b6dc379f 100644 --- a/src/test/subscription/t/016_stream_subxact.pl +++ b/src/test/subscription/t/016_stream_subxact.pl @@ -8,6 +8,72 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" option +# so the same code can be run both for the streaming=on and streaming=apply +# cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_apply) = @_; + + # Check that a background worker starts if "streaming" option is + # specified as "apply". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_apply) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + # Insert, update and delete enough rows to exceed 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series( 3, 500) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501, 1000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001, 1500) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + SAVEPOINT s3; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501, 2000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + SAVEPOINT s4; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001, 2500) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + COMMIT; + }); + + if ($is_apply) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(1667|1667|1667), + 'check data was copied to subscriber in streaming mode and extra columns contain local defaults' + ); + + # Cleanup the test data + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE (a > 2)"); + $node_publisher->wait_for_catchup($appname); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -37,6 +103,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -54,41 +124,26 @@ my $result = "SELECT count(*), count(c), count(d = 999) FROM test_tab"); is($result, qq(2|2|2), 'check initial data was copied to subscriber'); -# Insert, update and delete enough rows to exceed 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series( 3, 500) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501, 1000) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001, 1500) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -SAVEPOINT s3; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501, 2000) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -SAVEPOINT s4; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001, 2500) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -COMMIT; -}); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(1667|1667|1667), - 'check data was copied to subscriber in streaming mode and extra columns contain local defaults' +################################### +# Test using streaming mode 'apply' +################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" ); +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = apply)"); + +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; + +test_streaming($node_publisher, $node_subscriber, $appname, 1); + $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/017_stream_ddl.pl b/src/test/subscription/t/017_stream_ddl.pl index 0bce63b716..7e5dc18cd2 100644 --- a/src/test/subscription/t/017_stream_ddl.pl +++ b/src/test/subscription/t/017_stream_ddl.pl @@ -8,6 +8,111 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" option +# so the same code can be run both for the streaming=on and streaming=apply +# cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_apply) = @_; + + # a small (non-streamed) transaction with DDL and DML + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab VALUES (3, md5(3::text)); + ALTER TABLE test_tab ADD COLUMN c INT; + SAVEPOINT s1; + INSERT INTO test_tab VALUES (4, md5(4::text), -4); + COMMIT; + }); + + # Check that a background worker starts if "streaming" option is + # specified as "apply". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_apply) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + # large (streamed) transaction with DDL and DML + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(5, 1000) s(i); + ALTER TABLE test_tab ADD COLUMN d INT; + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001, 2000) s(i); + COMMIT; + }); + + if ($is_apply) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + # a small (non-streamed) transaction with DDL and DML + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab VALUES (2001, md5(2001::text), -2001, 2*2001); + ALTER TABLE test_tab ADD COLUMN e INT; + SAVEPOINT s1; + INSERT INTO test_tab VALUES (2002, md5(2002::text), -2002, 2*2002, -3*2002); + COMMIT; + }); + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d), count(e) FROM test_tab"); + is($result, qq(2002|1999|1002|1), + 'check data was copied to subscriber in streaming mode and extra columns contain local defaults' + ); + + # A large (streamed) transaction with DDL and DML. One of the DDL is performed + # after DML to ensure that we invalidate the schema sent for test_tab so that + # the next transaction has to send the schema again. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(2003,5000) s(i); + ALTER TABLE test_tab ADD COLUMN f INT; + COMMIT; + }); + + # A small transaction that won't get streamed. This is just to ensure that we + # send the schema again to reflect the last column added in the previous test. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i, 4*i FROM generate_series(5001,5005) s(i); + COMMIT; + }); + + $node_publisher->wait_for_catchup($appname); + + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d), count(e), count(f) FROM test_tab" + ); + is($result, qq(5005|5002|4005|3004|5), + 'check data was copied to subscriber for both streaming and non-streaming transactions' + ); + + # Cleanup the test data + $node_publisher->safe_psql( + 'postgres', q{ + DELETE FROM test_tab WHERE (a > 2); + ALTER TABLE test_tab DROP COLUMN c, DROP COLUMN d, DROP COLUMN e, DROP COLUMN f; + }); + $node_publisher->wait_for_catchup($appname); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -37,6 +142,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -54,76 +163,25 @@ my $result = "SELECT count(*), count(c), count(d = 999) FROM test_tab"); is($result, qq(2|0|0), 'check initial data was copied to subscriber'); -# a small (non-streamed) transaction with DDL and DML -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab VALUES (3, md5(3::text)); -ALTER TABLE test_tab ADD COLUMN c INT; -SAVEPOINT s1; -INSERT INTO test_tab VALUES (4, md5(4::text), -4); -COMMIT; -}); - -# large (streamed) transaction with DDL and DML -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(5, 1000) s(i); -ALTER TABLE test_tab ADD COLUMN d INT; -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001, 2000) s(i); -COMMIT; -}); - -# a small (non-streamed) transaction with DDL and DML -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab VALUES (2001, md5(2001::text), -2001, 2*2001); -ALTER TABLE test_tab ADD COLUMN e INT; -SAVEPOINT s1; -INSERT INTO test_tab VALUES (2002, md5(2002::text), -2002, 2*2002, -3*2002); -COMMIT; -}); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d), count(e) FROM test_tab"); -is($result, qq(2002|1999|1002|1), - 'check data was copied to subscriber in streaming mode and extra columns contain local defaults' +################################### +# Test using streaming mode 'apply' +################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" ); -# A large (streamed) transaction with DDL and DML. One of the DDL is performed -# after DML to ensure that we invalidate the schema sent for test_tab so that -# the next transaction has to send the schema again. -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(2003,5000) s(i); -ALTER TABLE test_tab ADD COLUMN f INT; -COMMIT; -}); - -# A small transaction that won't get streamed. This is just to ensure that we -# send the schema again to reflect the last column added in the previous test. -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i, 4*i FROM generate_series(5001,5005) s(i); -COMMIT; -}); +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = apply)"); -$node_publisher->wait_for_catchup($appname); +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d), count(e), count(f) FROM test_tab"); -is($result, qq(5005|5002|4005|3004|5), - 'check data was copied to subscriber for both streaming and non-streaming transactions' -); +test_streaming($node_publisher, $node_subscriber, $appname, 1); $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/018_stream_subxact_abort.pl b/src/test/subscription/t/018_stream_subxact_abort.pl index 7155442e76..71f8c1dd05 100644 --- a/src/test/subscription/t/018_stream_subxact_abort.pl +++ b/src/test/subscription/t/018_stream_subxact_abort.pl @@ -8,6 +8,113 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" option +# so the same code can be run both for the streaming=on and streaming=apply +# cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_apply) = @_; + + # Check that a background worker starts if "streaming" option is + # specified as "apply". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_apply) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + # large (streamed) transaction with DDL, DML and ROLLBACKs + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i); + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501,1000) s(i); + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001,1500) s(i); + SAVEPOINT s3; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501,2000) s(i); + ROLLBACK TO s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001,2500) s(i); + ROLLBACK TO s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2501,3000) s(i); + SAVEPOINT s4; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3001,3500) s(i); + SAVEPOINT s5; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3501,4000) s(i); + COMMIT; + }); + + if ($is_apply) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c) FROM test_tab"); + is($result, qq(2000|0), + 'check rollback to savepoint was reflected on subscriber and extra columns contain local defaults' + ); + + # large (streamed) transaction with subscriber receiving out of order + # subtransaction ROLLBACKs + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(4001,4500) s(i); + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001,5500) s(i); + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(6001,6500) s(i); + SAVEPOINT s3; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(7001,7500) s(i); + RELEASE s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(8001,8500) s(i); + ROLLBACK TO s1; + COMMIT; + }); + + $node_publisher->wait_for_catchup($appname); + + $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c) FROM test_tab"); + is($result, qq(2500|0), + 'check rollback to savepoint was reflected on subscriber'); + + # large (streamed) transaction with subscriber receiving rollback + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(8501,9000) s(i); + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(9001,9500) s(i); + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(9501,10000) s(i); + ROLLBACK; + }); + + $node_publisher->wait_for_catchup($appname); + + $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c) FROM test_tab"); + is($result, qq(2500|0), 'check rollback was reflected on subscriber'); + + # Cleanup the test data + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE (a > 2)"); + $node_publisher->wait_for_catchup($appname); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -36,6 +143,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -53,81 +164,25 @@ my $result = "SELECT count(*), count(c) FROM test_tab"); is($result, qq(2|0), 'check initial data was copied to subscriber'); -# large (streamed) transaction with DDL, DML and ROLLBACKs -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i); -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501,1000) s(i); -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001,1500) s(i); -SAVEPOINT s3; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501,2000) s(i); -ROLLBACK TO s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001,2500) s(i); -ROLLBACK TO s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2501,3000) s(i); -SAVEPOINT s4; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3001,3500) s(i); -SAVEPOINT s5; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3501,4000) s(i); -COMMIT; -}); - -$node_publisher->wait_for_catchup($appname); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c) FROM test_tab"); -is($result, qq(2000|0), - 'check rollback to savepoint was reflected on subscriber and extra columns contain local defaults' +################################### +# Test using streaming mode 'apply' +################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" ); -# large (streamed) transaction with subscriber receiving out of order -# subtransaction ROLLBACKs -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(4001,4500) s(i); -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001,5500) s(i); -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(6001,6500) s(i); -SAVEPOINT s3; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(7001,7500) s(i); -RELEASE s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(8001,8500) s(i); -ROLLBACK TO s1; -COMMIT; -}); - -$node_publisher->wait_for_catchup($appname); +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = apply)"); -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c) FROM test_tab"); -is($result, qq(2500|0), - 'check rollback to savepoint was reflected on subscriber'); - -# large (streamed) transaction with subscriber receiving rollback -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(8501,9000) s(i); -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(9001,9500) s(i); -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(9501,10000) s(i); -ROLLBACK; -}); +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c) FROM test_tab"); -is($result, qq(2500|0), 'check rollback was reflected on subscriber'); +test_streaming($node_publisher, $node_subscriber, $appname, 1); $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/019_stream_subxact_ddl_abort.pl b/src/test/subscription/t/019_stream_subxact_ddl_abort.pl index dbd0fca4d1..d0cd869430 100644 --- a/src/test/subscription/t/019_stream_subxact_ddl_abort.pl +++ b/src/test/subscription/t/019_stream_subxact_ddl_abort.pl @@ -9,6 +9,69 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" option +# so the same code can be run both for the streaming=on and streaming=apply +# cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_apply) = @_; + + # Check that a background worker starts if "streaming" option is + # specified as "apply". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_apply) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + # large (streamed) transaction with DDL, DML and ROLLBACKs + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i); + ALTER TABLE test_tab ADD COLUMN c INT; + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(501,1000) s(i); + ALTER TABLE test_tab ADD COLUMN d INT; + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001,1500) s(i); + ALTER TABLE test_tab ADD COLUMN e INT; + SAVEPOINT s3; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(1501,2000) s(i); + ALTER TABLE test_tab DROP COLUMN c; + ROLLBACK TO s1; + INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(501,1000) s(i); + COMMIT; + }); + + if ($is_apply) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c) FROM test_tab"); + is($result, qq(1000|500), + 'check rollback to savepoint was reflected on subscriber and extra columns contain local defaults' + ); + + # Cleanup the test data + $node_publisher->safe_psql( + 'postgres', q{ + DELETE FROM test_tab WHERE (a > 2); + ALTER TABLE test_tab DROP COLUMN c; + }); + $node_publisher->wait_for_catchup($appname); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -37,6 +100,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -54,35 +121,26 @@ my $result = "SELECT count(*), count(c) FROM test_tab"); is($result, qq(2|0), 'check initial data was copied to subscriber'); -# large (streamed) transaction with DDL, DML and ROLLBACKs -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i); -ALTER TABLE test_tab ADD COLUMN c INT; -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(501,1000) s(i); -ALTER TABLE test_tab ADD COLUMN d INT; -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001,1500) s(i); -ALTER TABLE test_tab ADD COLUMN e INT; -SAVEPOINT s3; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(1501,2000) s(i); -ALTER TABLE test_tab DROP COLUMN c; -ROLLBACK TO s1; -INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(501,1000) s(i); -COMMIT; -}); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c) FROM test_tab"); -is($result, qq(1000|500), - 'check rollback to savepoint was reflected on subscriber and extra columns contain local defaults' +################################### +# Test using streaming mode 'apply' +################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" ); +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = apply)"); + +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; + +test_streaming($node_publisher, $node_subscriber, $appname, 1); + $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/022_twophase_cascade.pl b/src/test/subscription/t/022_twophase_cascade.pl index 7a797f37ba..b52af74e35 100644 --- a/src/test/subscription/t/022_twophase_cascade.pl +++ b/src/test/subscription/t/022_twophase_cascade.pl @@ -11,6 +11,208 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" option +# so the same code can be run both for the streaming=on and streaming=apply +# cases. +sub test_streaming +{ + my ($node_A, $node_B, $node_C, $appname_B, $appname_C, $streaming_mode) = + @_; + + my $oldpid_B = $node_A->safe_psql( + 'postgres', " + SELECT pid FROM pg_stat_replication + WHERE application_name = '$appname_B' AND state = 'streaming';"); + my $oldpid_C = $node_B->safe_psql( + 'postgres', " + SELECT pid FROM pg_stat_replication + WHERE application_name = '$appname_C' AND state = 'streaming';"); + + # Setup logical replication streaming mode + + $node_B->safe_psql( + 'postgres', " + ALTER SUBSCRIPTION tap_sub_B + SET (streaming = $streaming_mode);"); + $node_C->safe_psql( + 'postgres', " + ALTER SUBSCRIPTION tap_sub_C + SET (streaming = $streaming_mode)"); + + # Wait for subscribers to finish initialization + + $node_A->poll_query_until( + 'postgres', " + SELECT pid != $oldpid_B FROM pg_stat_replication + WHERE application_name = '$appname_B' AND state = 'streaming';" + ) or die "Timed out while waiting for apply to restart"; + $node_B->poll_query_until( + 'postgres', " + SELECT pid != $oldpid_C FROM pg_stat_replication + WHERE application_name = '$appname_C' AND state = 'streaming';" + ) or die "Timed out while waiting for apply to restart"; + + ############################### + # Test 2PC PREPARE / COMMIT PREPARED. + # 1. Data is streamed as a 2PC transaction. + # 2. Then do commit prepared. + # + # Expect all data is replicated on subscriber(s) after the commit. + ############################### + + # Check that a background worker starts if "streaming" option is + # specified as "apply". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($streaming_mode eq 'apply') + { + $node_B->append_conf('postgresql.conf', "log_min_messages = debug1"); + $node_B->reload; + + $node_C->append_conf('postgresql.conf', "log_min_messages = debug1"); + $node_C->reload; + } + + # Insert, update and delete enough rows to exceed the 64kB limit. + # Then 2PC PREPARE + $node_A->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + if ($streaming_mode eq 'apply') + { + $node_B->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_B->append_conf('postgresql.conf', "log_min_messages = warning"); + $node_B->reload; + + $node_C->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_C->append_conf('postgresql.conf', "log_min_messages = warning"); + $node_C->reload; + } + + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); + + # check the transaction state is prepared on subscriber(s) + my $result = + $node_B->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber B'); + $result = + $node_C->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber C'); + + # 2PC COMMIT + $node_A->safe_psql('postgres', "COMMIT PREPARED 'test_prepared_tab';"); + + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); + + # check that transaction was committed on subscriber(s) + $result = $node_B->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), + 'Rows inserted by 2PC have committed on subscriber B, and extra columns have local defaults' + ); + $result = $node_C->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), + 'Rows inserted by 2PC have committed on subscriber C, and extra columns have local defaults' + ); + + # check the transaction state is ended on subscriber(s) + $result = + $node_B->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is committed on subscriber B'); + $result = + $node_C->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is committed on subscriber C'); + + ############################### + # Test 2PC PREPARE with a nested ROLLBACK TO SAVEPOINT. + # 0. Cleanup from previous test leaving only 2 rows. + # 1. Insert one more row. + # 2. Record a SAVEPOINT. + # 3. Data is streamed using 2PC. + # 4. Do rollback to SAVEPOINT prior to the streamed inserts. + # 5. Then COMMIT PREPARED. + # + # Expect data after the SAVEPOINT is aborted leaving only 3 rows (= 2 original + 1 from step 1). + ############################### + + # First, delete the data except for 2 rows (delete will be replicated) + $node_A->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); + + # 2PC PREPARE with a nested ROLLBACK TO SAVEPOINT + $node_A->safe_psql( + 'postgres', " + BEGIN; + INSERT INTO test_tab VALUES (9999, 'foobar'); + SAVEPOINT sp_inner; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + ROLLBACK TO SAVEPOINT sp_inner; + PREPARE TRANSACTION 'outer'; + "); + + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); + + # check the transaction state prepared on subscriber(s) + $result = + $node_B->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber B'); + $result = + $node_C->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber C'); + + # 2PC COMMIT + $node_A->safe_psql('postgres', "COMMIT PREPARED 'outer';"); + + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); + + # check the transaction state is ended on subscriber + $result = + $node_B->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is ended on subscriber B'); + $result = + $node_C->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is ended on subscriber C'); + + # check inserts are visible at subscriber(s). + # All the streamed data (prior to the SAVEPOINT) should be rolled back. + # (9999, 'foobar') should be committed. + $result = $node_B->safe_psql('postgres', + "SELECT count(*) FROM test_tab where b = 'foobar';"); + is($result, qq(1), 'Rows committed are present on subscriber B'); + $result = + $node_B->safe_psql('postgres', "SELECT count(*) FROM test_tab;"); + is($result, qq(3), 'Rows committed are present on subscriber B'); + $result = $node_C->safe_psql('postgres', + "SELECT count(*) FROM test_tab where b = 'foobar';"); + is($result, qq(1), 'Rows committed are present on subscriber C'); + $result = + $node_C->safe_psql('postgres', "SELECT count(*) FROM test_tab;"); + is($result, qq(3), 'Rows committed are present on subscriber C'); + + # Cleanup the test data + $node_A->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); +} + ############################### # Setup a cascade of pub/sub nodes. # node_A -> node_B -> node_C @@ -260,160 +462,15 @@ is($result, qq(21), 'Rows committed are present on subscriber C'); # 2PC + STREAMING TESTS # --------------------- -my $oldpid_B = $node_A->safe_psql( - 'postgres', " - SELECT pid FROM pg_stat_replication - WHERE application_name = '$appname_B' AND state = 'streaming';"); -my $oldpid_C = $node_B->safe_psql( - 'postgres', " - SELECT pid FROM pg_stat_replication - WHERE application_name = '$appname_C' AND state = 'streaming';"); - -# Setup logical replication (streaming = on) - -$node_B->safe_psql( - 'postgres', " - ALTER SUBSCRIPTION tap_sub_B - SET (streaming = on);"); -$node_C->safe_psql( - 'postgres', " - ALTER SUBSCRIPTION tap_sub_C - SET (streaming = on)"); - -# Wait for subscribers to finish initialization - -$node_A->poll_query_until( - 'postgres', " - SELECT pid != $oldpid_B FROM pg_stat_replication - WHERE application_name = '$appname_B' AND state = 'streaming';" -) or die "Timed out while waiting for apply to restart"; -$node_B->poll_query_until( - 'postgres', " - SELECT pid != $oldpid_C FROM pg_stat_replication - WHERE application_name = '$appname_C' AND state = 'streaming';" -) or die "Timed out while waiting for apply to restart"; - -############################### -# Test 2PC PREPARE / COMMIT PREPARED. -# 1. Data is streamed as a 2PC transaction. -# 2. Then do commit prepared. -# -# Expect all data is replicated on subscriber(s) after the commit. -############################### - -# Insert, update and delete enough rows to exceed the 64kB limit. -# Then 2PC PREPARE -$node_A->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_A->wait_for_catchup($appname_B); -$node_B->wait_for_catchup($appname_C); +################################ +# Test using streaming mode 'on' +################################ +test_streaming($node_A, $node_B, $node_C, $appname_B, $appname_C, 'on'); -# check the transaction state is prepared on subscriber(s) -$result = - $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber B'); -$result = - $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber C'); - -# 2PC COMMIT -$node_A->safe_psql('postgres', "COMMIT PREPARED 'test_prepared_tab';"); - -$node_A->wait_for_catchup($appname_B); -$node_B->wait_for_catchup($appname_C); - -# check that transaction was committed on subscriber(s) -$result = $node_B->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), - 'Rows inserted by 2PC have committed on subscriber B, and extra columns have local defaults' -); -$result = $node_C->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), - 'Rows inserted by 2PC have committed on subscriber C, and extra columns have local defaults' -); - -# check the transaction state is ended on subscriber(s) -$result = - $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is committed on subscriber B'); -$result = - $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is committed on subscriber C'); - -############################### -# Test 2PC PREPARE with a nested ROLLBACK TO SAVEPOINT. -# 0. Cleanup from previous test leaving only 2 rows. -# 1. Insert one more row. -# 2. Record a SAVEPOINT. -# 3. Data is streamed using 2PC. -# 4. Do rollback to SAVEPOINT prior to the streamed inserts. -# 5. Then COMMIT PREPARED. -# -# Expect data after the SAVEPOINT is aborted leaving only 3 rows (= 2 original + 1 from step 1). -############################### - -# First, delete the data except for 2 rows (delete will be replicated) -$node_A->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); - -# 2PC PREPARE with a nested ROLLBACK TO SAVEPOINT -$node_A->safe_psql( - 'postgres', " - BEGIN; - INSERT INTO test_tab VALUES (9999, 'foobar'); - SAVEPOINT sp_inner; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - ROLLBACK TO SAVEPOINT sp_inner; - PREPARE TRANSACTION 'outer'; - "); - -$node_A->wait_for_catchup($appname_B); -$node_B->wait_for_catchup($appname_C); - -# check the transaction state prepared on subscriber(s) -$result = - $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber B'); -$result = - $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber C'); - -# 2PC COMMIT -$node_A->safe_psql('postgres', "COMMIT PREPARED 'outer';"); - -$node_A->wait_for_catchup($appname_B); -$node_B->wait_for_catchup($appname_C); - -# check the transaction state is ended on subscriber -$result = - $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is ended on subscriber B'); -$result = - $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is ended on subscriber C'); - -# check inserts are visible at subscriber(s). -# All the streamed data (prior to the SAVEPOINT) should be rolled back. -# (9999, 'foobar') should be committed. -$result = $node_B->safe_psql('postgres', - "SELECT count(*) FROM test_tab where b = 'foobar';"); -is($result, qq(1), 'Rows committed are present on subscriber B'); -$result = $node_B->safe_psql('postgres', "SELECT count(*) FROM test_tab;"); -is($result, qq(3), 'Rows committed are present on subscriber B'); -$result = $node_C->safe_psql('postgres', - "SELECT count(*) FROM test_tab where b = 'foobar';"); -is($result, qq(1), 'Rows committed are present on subscriber C'); -$result = $node_C->safe_psql('postgres', "SELECT count(*) FROM test_tab;"); -is($result, qq(3), 'Rows committed are present on subscriber C'); +################################### +# Test using streaming mode 'apply' +################################### +test_streaming($node_A, $node_B, $node_C, $appname_B, $appname_C, 'apply'); ############################### # check all the cleanup diff --git a/src/test/subscription/t/023_twophase_stream.pl b/src/test/subscription/t/023_twophase_stream.pl index d8475d25a4..f55c5c95e7 100644 --- a/src/test/subscription/t/023_twophase_stream.pl +++ b/src/test/subscription/t/023_twophase_stream.pl @@ -8,6 +8,266 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" option +# so the same code can be run both for the streaming=on and streaming=apply +# cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_apply) = @_; + + ############################### + # Test 2PC PREPARE / COMMIT PREPARED. + # 1. Data is streamed as a 2PC transaction. + # 2. Then do commit prepared. + # + # Expect all data is replicated on subscriber side after the commit. + ############################### + + # Check that a background worker starts if "streaming" option is + # specified as "apply". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_apply) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + # check that 2PC gets replicated to subscriber + # Insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + if ($is_apply) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is in prepared state on subscriber + my $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber'); + + # 2PC transaction gets committed + $node_publisher->safe_psql('postgres', + "COMMIT PREPARED 'test_prepared_tab';"); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is committed on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), + 'Rows inserted by 2PC have committed on subscriber, and extra columns contain local defaults' + ); + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is committed on subscriber'); + + ############################### + # Test 2PC PREPARE / ROLLBACK PREPARED. + # 1. Table is deleted back to 2 rows which are replicated on subscriber. + # 2. Data is streamed using 2PC. + # 3. Do rollback prepared. + # + # Expect data rolls back leaving only the original 2 rows. + ############################### + + # First, delete the data except for 2 rows (will be replicated) + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE a > 2;"); + + # Then insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is in prepared state on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber'); + + # 2PC transaction gets aborted + $node_publisher->safe_psql('postgres', + "ROLLBACK PREPARED 'test_prepared_tab';"); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is aborted on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(2|2|2), + 'Rows inserted by 2PC are rolled back, leaving only the original 2 rows' + ); + + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is aborted on subscriber'); + + ############################### + # Check that 2PC COMMIT PREPARED is decoded properly on crash restart. + # 1. insert, update and delete enough rows to exceed the 64kB limit. + # 2. Then server crashes before the 2PC transaction is committed. + # 3. After servers are restarted the pending transaction is committed. + # + # Expect all data is replicated on subscriber side after the commit. + # Note: both publisher and subscriber do crash/restart. + ############################### + + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + $node_subscriber->stop('immediate'); + $node_publisher->stop('immediate'); + + $node_publisher->start; + $node_subscriber->start; + + # commit post the restart + $node_publisher->safe_psql('postgres', + "COMMIT PREPARED 'test_prepared_tab';"); + $node_publisher->wait_for_catchup($appname); + + # check inserts are visible + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), + 'Rows inserted by 2PC have committed on subscriber, and extra columns contain local defaults' + ); + + ############################### + # Do INSERT after the PREPARE but before ROLLBACK PREPARED. + # 1. Table is deleted back to 2 rows which are replicated on subscriber. + # 2. Data is streamed using 2PC. + # 3. A single row INSERT is done which is after the PREPARE. + # 4. Then do a ROLLBACK PREPARED. + # + # Expect the 2PC data rolls back leaving only 3 rows on the subscriber + # (the original 2 + inserted 1). + ############################### + + # First, delete the data except for 2 rows (will be replicated) + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE a > 2;"); + + # Then insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is in prepared state on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber'); + + # Insert a different record (now we are outside of the 2PC transaction) + # Note: the 2PC transaction still holds row locks so make sure this insert is for a separate primary key + $node_publisher->safe_psql('postgres', + "INSERT INTO test_tab VALUES (99999, 'foobar')"); + + # 2PC transaction gets aborted + $node_publisher->safe_psql('postgres', + "ROLLBACK PREPARED 'test_prepared_tab';"); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is aborted on subscriber, + # but the extra INSERT outside of the 2PC still was replicated + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3|3|3), + 'check the outside insert was copied to subscriber'); + + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is aborted on subscriber'); + + ############################### + # Do INSERT after the PREPARE but before COMMIT PREPARED. + # 1. Table is deleted back to 2 rows which are replicated on subscriber. + # 2. Data is streamed using 2PC. + # 3. A single row INSERT is done which is after the PREPARE. + # 4. Then do a COMMIT PREPARED. + # + # Expect 2PC data + the extra row are on the subscriber + # (the 3334 + inserted 1 = 3335). + ############################### + + # First, delete the data except for 2 rows (will be replicated) + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE a > 2;"); + + # Then insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is in prepared state on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber'); + + # Insert a different record (now we are outside of the 2PC transaction) + # Note: the 2PC transaction still holds row locks so make sure this insert is for a separare primary key + $node_publisher->safe_psql('postgres', + "INSERT INTO test_tab VALUES (99999, 'foobar')"); + + # 2PC transaction gets committed + $node_publisher->safe_psql('postgres', + "COMMIT PREPARED 'test_prepared_tab';"); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is committed on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3335|3335|3335), + 'Rows inserted by 2PC (as well as outside insert) have committed on subscriber, and extra columns contain local defaults' + ); + + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is committed on subscriber'); + + # Cleanup the test data + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE a > 2;"); + $node_publisher->wait_for_catchup($appname); +} + ############################### # Setup ############################### @@ -48,6 +308,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql( 'postgres', " CREATE SUBSCRIPTION tap_sub @@ -70,236 +334,30 @@ my $twophase_query = $node_subscriber->poll_query_until('postgres', $twophase_query) or die "Timed out while waiting for subscriber to enable twophase"; -############################### # Check initial data was copied to subscriber -############################### my $result = $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab"); is($result, qq(2|2|2), 'check initial data was copied to subscriber'); -############################### -# Test 2PC PREPARE / COMMIT PREPARED. -# 1. Data is streamed as a 2PC transaction. -# 2. Then do commit prepared. -# -# Expect all data is replicated on subscriber side after the commit. -############################### - -# check that 2PC gets replicated to subscriber -# Insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is in prepared state on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber'); - -# 2PC transaction gets committed -$node_publisher->safe_psql('postgres', - "COMMIT PREPARED 'test_prepared_tab';"); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is committed on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), - 'Rows inserted by 2PC have committed on subscriber, and extra columns contain local defaults' -); -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is committed on subscriber'); - -############################### -# Test 2PC PREPARE / ROLLBACK PREPARED. -# 1. Table is deleted back to 2 rows which are replicated on subscriber. -# 2. Data is streamed using 2PC. -# 3. Do rollback prepared. -# -# Expect data rolls back leaving only the original 2 rows. -############################### - -# First, delete the data except for 2 rows (will be replicated) -$node_publisher->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); - -# Then insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is in prepared state on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber'); - -# 2PC transaction gets aborted -$node_publisher->safe_psql('postgres', - "ROLLBACK PREPARED 'test_prepared_tab';"); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is aborted on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(2|2|2), - 'Rows inserted by 2PC are rolled back, leaving only the original 2 rows'); - -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is aborted on subscriber'); - -############################### -# Check that 2PC COMMIT PREPARED is decoded properly on crash restart. -# 1. insert, update and delete enough rows to exceed the 64kB limit. -# 2. Then server crashes before the 2PC transaction is committed. -# 3. After servers are restarted the pending transaction is committed. -# -# Expect all data is replicated on subscriber side after the commit. -# Note: both publisher and subscriber do crash/restart. -############################### - -$node_publisher->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_subscriber->stop('immediate'); -$node_publisher->stop('immediate'); - -$node_publisher->start; -$node_subscriber->start; - -# commit post the restart -$node_publisher->safe_psql('postgres', - "COMMIT PREPARED 'test_prepared_tab';"); -$node_publisher->wait_for_catchup($appname); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -# check inserts are visible -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), - 'Rows inserted by 2PC have committed on subscriber, and extra columns contain local defaults' +################################### +# Test using streaming mode 'apply' +################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" ); -############################### -# Do INSERT after the PREPARE but before ROLLBACK PREPARED. -# 1. Table is deleted back to 2 rows which are replicated on subscriber. -# 2. Data is streamed using 2PC. -# 3. A single row INSERT is done which is after the PREPARE. -# 4. Then do a ROLLBACK PREPARED. -# -# Expect the 2PC data rolls back leaving only 3 rows on the subscriber -# (the original 2 + inserted 1). -############################### - -# First, delete the data except for 2 rows (will be replicated) -$node_publisher->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); - -# Then insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is in prepared state on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber'); - -# Insert a different record (now we are outside of the 2PC transaction) -# Note: the 2PC transaction still holds row locks so make sure this insert is for a separate primary key -$node_publisher->safe_psql('postgres', - "INSERT INTO test_tab VALUES (99999, 'foobar')"); - -# 2PC transaction gets aborted -$node_publisher->safe_psql('postgres', - "ROLLBACK PREPARED 'test_prepared_tab';"); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is aborted on subscriber, -# but the extra INSERT outside of the 2PC still was replicated -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3|3|3), 'check the outside insert was copied to subscriber'); - -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is aborted on subscriber'); - -############################### -# Do INSERT after the PREPARE but before COMMIT PREPARED. -# 1. Table is deleted back to 2 rows which are replicated on subscriber. -# 2. Data is streamed using 2PC. -# 3. A single row INSERT is done which is after the PREPARE. -# 4. Then do a COMMIT PREPARED. -# -# Expect 2PC data + the extra row are on the subscriber -# (the 3334 + inserted 1 = 3335). -############################### - -# First, delete the data except for 2 rows (will be replicated) -$node_publisher->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); - -# Then insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is in prepared state on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber'); - -# Insert a different record (now we are outside of the 2PC transaction) -# Note: the 2PC transaction still holds row locks so make sure this insert is for a separare primary key -$node_publisher->safe_psql('postgres', - "INSERT INTO test_tab VALUES (99999, 'foobar')"); - -# 2PC transaction gets committed -$node_publisher->safe_psql('postgres', - "COMMIT PREPARED 'test_prepared_tab';"); - -$node_publisher->wait_for_catchup($appname); +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = apply)"); -# check that transaction is committed on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3335|3335|3335), - 'Rows inserted by 2PC (as well as outside insert) have committed on subscriber, and extra columns contain local defaults' -); +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is committed on subscriber'); +test_streaming($node_publisher, $node_subscriber, $appname, 1); ############################### # check all the cleanup -- 2.23.0.windows.1 [application/octet-stream] v13-0003-Add-some-checks-before-using-apply-background-wo.patch (35.7K, ../../OS3PR01MB6275F61CC74C5F3E47C7DD229EB59@OS3PR01MB6275.jpnprd01.prod.outlook.com/4-v13-0003-Add-some-checks-before-using-apply-background-wo.patch) download | inline diff: From d2e1b80a5dfd8cf5e011c789fc400568093638fe Mon Sep 17 00:00:00 2001 From: wangw <[email protected]> Date: Tue, 14 Jun 2022 11:23:52 +0800 Subject: [PATCH v13 3/4] Add some checks before using apply background worker to apply changes. If any of the following checks are violated, an error will be reported. 1. The unique columns between publisher and subscriber are difference. 2. There is any non-immutable function present in expression in subscriber's relation. Check from the following 4 items: a. The function in triggers; b. Column default value expressions and domain constraints; c. Constraint expressions. d. The foreign keys. --- doc/src/sgml/ref/create_subscription.sgml | 4 + .../replication/logical/applybgwroker.c | 44 ++ src/backend/replication/logical/proto.c | 62 ++- src/backend/replication/logical/relation.c | 190 +++++++++ src/backend/replication/logical/tablesync.c | 1 + src/backend/replication/logical/worker.c | 23 +- src/backend/utils/cache/typcache.c | 17 + src/include/replication/logicalproto.h | 1 + src/include/replication/logicalrelation.h | 16 + src/include/replication/worker_internal.h | 1 + src/include/utils/typcache.h | 2 + .../subscription/t/022_twophase_cascade.pl | 7 + .../subscription/t/032_streaming_apply.pl | 391 ++++++++++++++++++ 13 files changed, 751 insertions(+), 8 deletions(-) create mode 100644 src/test/subscription/t/032_streaming_apply.pl diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml index d4caae0222..8de1a23ce4 100644 --- a/doc/src/sgml/ref/create_subscription.sgml +++ b/doc/src/sgml/ref/create_subscription.sgml @@ -240,6 +240,10 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl the transaction is committed. Note that if an error happens when applying changes in a background worker, it might not report the finish LSN of the remote transaction in the server log. + To run in this mode, there are following two requirements. The first + is that the unique column should be the same between publisher and + subscriber; the second is that there should not be any non-immutable + function in subscriber-side replicated table. </para> </listitem> </varlistentry> diff --git a/src/backend/replication/logical/applybgwroker.c b/src/backend/replication/logical/applybgwroker.c index 3e2f615f20..e004132544 100644 --- a/src/backend/replication/logical/applybgwroker.c +++ b/src/backend/replication/logical/applybgwroker.c @@ -775,3 +775,47 @@ apply_bgworker_subxact_info_add(TransactionId current_xid) MemoryContextSwitchTo(oldctx); } } + +/* + * Check if changes on this logical replication relation can be applied by + * apply background worker. + * + * Although we maintains the commit order by allowing only one process to + * commit at a time, our access order to the relation has changed. + * This could cause unexpected problems if the unique column on the replicated + * table is inconsistent with the publisher-side or contains non-immutable + * functions when applying transactions in the apply background worker. + */ +void +apply_bgworker_relation_check(LogicalRepRelMapEntry *rel) +{ + /* Check only we are in apply bgworker. */ + if (!am_apply_bgworker()) + return; + + /* + * If it is a partitioned table, we do not check it, we will check its + * partition later. + */ + if (rel->localrel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + return; + + /* + * If any unique index exist, check that they are same as remoterel. + */ + if (!rel->sameunique) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot replicate relation with different unique index"), + errhint("Please change the streaming option to 'on' instead of 'apply'."))); + + /* + * Check if there is any non-immutable function present in expression in + * this relation. + */ + if (rel->volatility == FUNCTION_NONIMMUTABLE) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot replicate relation. There is at least one non-immutable function"), + errhint("Please change the streaming option to 'on' instead of 'apply'."))); +} diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c index a888038eb4..6af3302270 100644 --- a/src/backend/replication/logical/proto.c +++ b/src/backend/replication/logical/proto.c @@ -23,7 +23,8 @@ /* * Protocol message flags. */ -#define LOGICALREP_IS_REPLICA_IDENTITY 1 +#define LOGICALREP_IS_REPLICA_IDENTITY 0x0001 +#define LOGICALREP_IS_UNIQUE 0x0002 #define MESSAGE_TRANSACTIONAL (1<<0) #define TRUNCATE_CASCADE (1<<0) @@ -933,11 +934,55 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns) TupleDesc desc; int i; uint16 nliveatts = 0; - Bitmapset *idattrs = NULL; + Bitmapset *idattrs = NULL, + *attunique = NULL; bool replidentfull; desc = RelationGetDescr(rel); + if (rel->rd_rel->relhasindex) + { + List *indexoidlist = RelationGetIndexList(rel); + ListCell *indexoidscan; + + foreach(indexoidscan, indexoidlist) + { + Oid indexoid = lfirst_oid(indexoidscan); + Relation indexRel; + + /* Look up the description for index */ + indexRel = RelationIdGetRelation(indexoid); + + if (!RelationIsValid(indexRel)) + elog(ERROR, "could not open relation with OID %u", indexoid); + + if (indexRel->rd_index->indisunique) + { + int i; + + /* Add referenced attributes to idindexattrs */ + for (i = 0; i < indexRel->rd_index->indnatts; i++) + { + int attrnum = indexRel->rd_index->indkey.values[i]; + + /* + * We don't include non-key columns into idindexattrs + * bitmaps. See RelationGetIndexAttrBitmap. + */ + if (attrnum != 0) + { + if (i < indexRel->rd_index->indnkeyatts && + !bms_is_member(attrnum - FirstLowInvalidHeapAttributeNumber, attunique)) + attunique = bms_add_member(attunique, + attrnum - FirstLowInvalidHeapAttributeNumber); + } + } + } + RelationClose(indexRel); + } + list_free(indexoidlist); + } + /* send number of live attributes */ for (i = 0; i < desc->natts; i++) { @@ -976,6 +1021,10 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns) idattrs)) flags |= LOGICALREP_IS_REPLICA_IDENTITY; + if (bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber, + attunique)) + flags |= LOGICALREP_IS_UNIQUE; + pq_sendbyte(out, flags); /* attribute name */ @@ -1001,7 +1050,8 @@ logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel) int natts; char **attnames; Oid *atttyps; - Bitmapset *attkeys = NULL; + Bitmapset *attkeys = NULL, + *attunique = NULL; natts = pq_getmsgint(in, 2); attnames = palloc(natts * sizeof(char *)); @@ -1012,11 +1062,14 @@ logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel) { uint8 flags; - /* Check for replica identity column */ + /* Check for replica identity and unique column */ flags = pq_getmsgbyte(in); if (flags & LOGICALREP_IS_REPLICA_IDENTITY) attkeys = bms_add_member(attkeys, i); + if (flags & LOGICALREP_IS_UNIQUE) + attunique = bms_add_member(attunique, i); + /* attribute name */ attnames[i] = pstrdup(pq_getmsgstring(in)); @@ -1030,6 +1083,7 @@ logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel) rel->attnames = attnames; rel->atttyps = atttyps; rel->attkeys = attkeys; + rel->attunique = attunique; rel->natts = natts; } diff --git a/src/backend/replication/logical/relation.c b/src/backend/replication/logical/relation.c index e989047681..8fb34640da 100644 --- a/src/backend/replication/logical/relation.c +++ b/src/backend/replication/logical/relation.c @@ -19,12 +19,19 @@ #include "access/table.h" #include "catalog/namespace.h" +#include "catalog/pg_proc.h" #include "catalog/pg_subscription_rel.h" +#include "commands/trigger.h" #include "executor/executor.h" #include "nodes/makefuncs.h" +#include "optimizer/optimizer.h" #include "replication/logicalrelation.h" #include "replication/worker_internal.h" +#include "rewrite/rewriteHandler.h" #include "utils/inval.h" +#include "utils/lsyscache.h" +#include "utils/syscache.h" +#include "utils/typcache.h" static MemoryContext LogicalRepRelMapContext = NULL; @@ -91,6 +98,23 @@ logicalrep_relmap_invalidate_cb(Datum arg, Oid reloid) } } +/* + * Reset the flag volatility of all existing entry in the relation map cache. + */ +static void +logicalrep_relmap_reset_volatility_cb(Datum arg, int cacheid, uint32 hashvalue) +{ + HASH_SEQ_STATUS hash_seq; + LogicalRepRelMapEntry *entry; + + if (LogicalRepRelMap == NULL) + return; + + hash_seq_init(&hash_seq, LogicalRepRelMap); + while ((entry = hash_seq_search(&hash_seq)) != NULL) + entry->volatility = FUNCTION_UNKNOWN; +} + /* * Initialize the relation map cache. */ @@ -116,6 +140,9 @@ logicalrep_relmap_init(void) /* Watch for invalidation events. */ CacheRegisterRelcacheCallback(logicalrep_relmap_invalidate_cb, (Datum) 0); + CacheRegisterSyscacheCallback(PROCOID, + logicalrep_relmap_reset_volatility_cb, + (Datum) 0); } /* @@ -142,6 +169,7 @@ logicalrep_relmap_free_entry(LogicalRepRelMapEntry *entry) pfree(remoterel->atttyps); } bms_free(remoterel->attkeys); + bms_free(remoterel->attunique); if (entry->attrmap) free_attrmap(entry->attrmap); @@ -190,6 +218,7 @@ logicalrep_relmap_update(LogicalRepRelation *remoterel) } entry->remoterel.replident = remoterel->replident; entry->remoterel.attkeys = bms_copy(remoterel->attkeys); + entry->remoterel.attunique = bms_copy(remoterel->attunique); MemoryContextSwitchTo(oldctx); } @@ -310,6 +339,160 @@ logicalrep_rel_mark_updatable(LogicalRepRelMapEntry *entry) } } +/* + * Check if unique index/constraint matches and mark sameunique and volatility + * flag. + * + * Don't throw any error here just mark the relation entry as not sameunique or + * FUNCTION_NONIMMUTABLE as we only check these in apply background worker. + */ +static void +logicalrep_rel_mark_safe_in_apply_bgworker(LogicalRepRelMapEntry *entry) +{ + Bitmapset *ukey; + int i; + TupleDesc tupdesc; + int attnum; + List *fkeys = NIL; + + /* + * Check that the unique column in the relation on the subscriber-side is + * also the unique column on the publisher-side. + */ + entry->sameunique = true; + ukey = RelationGetIndexAttrBitmap(entry->localrel, + INDEX_ATTR_BITMAP_KEY); + + if (ukey) + { + i = -1; + while ((i = bms_next_member(ukey, i)) >= 0) + { + attnum = AttrNumberGetAttrOffset(i + FirstLowInvalidHeapAttributeNumber); + + if (entry->attrmap->attnums[attnum] < 0 || + !bms_is_member(entry->attrmap->attnums[attnum], entry->remoterel.attunique)) + { + entry->sameunique = false; + break; + } + } + } + + /* + * Check whether there is any non-immutable function in the local table. + * + * a. The function in triggers; + * b. Column default value expressions and domain constraints; + * c. Constraint expressions; + * d. Foreign keys. + */ + if (entry->volatility != FUNCTION_UNKNOWN) + return; + + /* Initialize the flag. */ + entry->volatility = FUNCTION_IMMUTABLE; + + /* Check the trigger functions. */ + if (entry->localrel->trigdesc != NULL) + { + for (i = 0; i < entry->localrel->trigdesc->numtriggers; i++) + { + Trigger *trig = entry->localrel->trigdesc->triggers + i; + + if (trig->tgenabled != TRIGGER_FIRES_ALWAYS && + trig->tgenabled != TRIGGER_FIRES_ON_REPLICA) + continue; + + if (func_volatile(trig->tgfoid) != PROVOLATILE_IMMUTABLE) + { + entry->volatility = FUNCTION_NONIMMUTABLE; + return; + } + } + } + + /* Check the columns. */ + tupdesc = RelationGetDescr(entry->localrel); + for (attnum = 0; attnum < tupdesc->natts; attnum++) + { + Form_pg_attribute att = TupleDescAttr(tupdesc, attnum); + + /* We don't need info for dropped or generated attributes */ + if (att->attisdropped || att->attgenerated) + continue; + + /* + * We don't need to check columns that only exist on the + * subscriber + */ + if (entry->attrmap->attnums[attnum] < 0) + continue; + + if (att->atthasdef) + { + Node *defaultexpr; + + defaultexpr = build_column_default(entry->localrel, attnum + 1); + if (contain_mutable_functions(defaultexpr)) + { + entry->volatility = FUNCTION_NONIMMUTABLE; + return; + } + } + + /* + * If the column is of a DOMAIN type, determine whether + * that domain has any CHECK expressions that are not + * immutable. + */ + if (get_typtype(att->atttypid) == TYPTYPE_DOMAIN) + { + List *domain_constraints; + ListCell *lc; + + domain_constraints = GetDomainConstraints(att->atttypid); + + foreach(lc, domain_constraints) + { + DomainConstraintState *con = (DomainConstraintState *) lfirst(lc); + + if (con->check_expr && contain_mutable_functions((Node *) con->check_expr)) + { + entry->volatility = FUNCTION_NONIMMUTABLE; + return; + } + } + } + } + + /* Check the constraints. */ + if (tupdesc->constr) + { + ConstrCheck *check = tupdesc->constr->check; + + /* + * Determine if there are any CHECK constraints which + * contains non-immutable function. + */ + for (i = 0; i < tupdesc->constr->num_check; i++) + { + Expr *check_expr = stringToNode(check[i].ccbin); + + if (contain_mutable_functions((Node *) check_expr)) + { + entry->volatility = FUNCTION_NONIMMUTABLE; + return; + } + } + } + + /* Check the foreign keys. */ + fkeys = RelationGetFKeyList(entry->localrel); + if (fkeys) + entry->volatility = FUNCTION_NONIMMUTABLE; +} + /* * Open the local relation associated with the remote one. * @@ -438,6 +621,9 @@ logicalrep_rel_open(LogicalRepRelId remoteid, LOCKMODE lockmode) */ logicalrep_rel_mark_updatable(entry); + /* Set if changes could be applied in the apply background worker. */ + logicalrep_rel_mark_safe_in_apply_bgworker(entry); + entry->localrelvalid = true; } @@ -653,6 +839,7 @@ logicalrep_partition_open(LogicalRepRelMapEntry *root, } entry->remoterel.replident = remoterel->replident; entry->remoterel.attkeys = bms_copy(remoterel->attkeys); + entry->remoterel.attunique = bms_copy(remoterel->attunique); } entry->localrel = partrel; @@ -696,6 +883,9 @@ logicalrep_partition_open(LogicalRepRelMapEntry *root, /* Set if the table's replica identity is enough to apply update/delete. */ logicalrep_rel_mark_updatable(entry); + /* Set if changes could be applied in the apply background worker. */ + logicalrep_rel_mark_safe_in_apply_bgworker(entry); + entry->localrelvalid = true; /* state and statelsn are left set to 0. */ diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c index 8ffba7e2e5..3cdbf8b457 100644 --- a/src/backend/replication/logical/tablesync.c +++ b/src/backend/replication/logical/tablesync.c @@ -884,6 +884,7 @@ fetch_remote_table_info(char *nspname, char *relname, lrel->attnames = palloc0(MaxTupleAttributeNumber * sizeof(char *)); lrel->atttyps = palloc0(MaxTupleAttributeNumber * sizeof(Oid)); lrel->attkeys = NULL; + lrel->attunique = NULL; /* * Store the columns as a list of names. Ignore those that are not diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c index cc31e016e4..7dfdbbc0ab 100644 --- a/src/backend/replication/logical/worker.c +++ b/src/backend/replication/logical/worker.c @@ -1373,6 +1373,14 @@ apply_handle_stream_stop(StringInfo s) { char action = LOGICAL_REP_MSG_STREAM_STOP; + /* + * Unlike stream_commit, we don't need to wait here for stream_stop to + * finish. Allowing the other transaction to be applied before stream_stop + * is finished can only lead to failures if the unique index/constraint is + * different between publisher and subscriber. But for such cases, we don't + * allow streamed transactions to be applied in parallel. See + * apply_bgworker_relation_check. + */ apply_bgworker_send_data(stream_apply_worker, 1, &action); elog(DEBUG1, "stopped streaming of xid %u, %u changes streamed", stream_xid, nchanges); @@ -1915,6 +1923,8 @@ apply_handle_insert(StringInfo s) /* Set relation for error callback */ apply_error_callback_arg.rel = rel; + apply_bgworker_relation_check(rel); + /* Initialize the executor state. */ edata = create_edata_for_relation(rel); estate = edata->estate; @@ -2058,6 +2068,8 @@ apply_handle_update(StringInfo s) /* Check if we can do the update. */ check_relation_updatable(rel); + apply_bgworker_relation_check(rel); + /* Initialize the executor state. */ edata = create_edata_for_relation(rel); estate = edata->estate; @@ -2226,6 +2238,8 @@ apply_handle_delete(StringInfo s) /* Check if we can do the delete. */ check_relation_updatable(rel); + apply_bgworker_relation_check(rel); + /* Initialize the executor state. */ edata = create_edata_for_relation(rel); estate = edata->estate; @@ -2411,13 +2425,14 @@ apply_handle_tuple_routing(ApplyExecutionData *edata, } MemoryContextSwitchTo(oldctx); + part_entry = logicalrep_partition_open(relmapentry, partrel, + attrmap); + + apply_bgworker_relation_check(part_entry); + /* Check if we can do the update or delete on the leaf partition. */ if (operation == CMD_UPDATE || operation == CMD_DELETE) - { - part_entry = logicalrep_partition_open(relmapentry, partrel, - attrmap); check_relation_updatable(part_entry); - } switch (operation) { diff --git a/src/backend/utils/cache/typcache.c b/src/backend/utils/cache/typcache.c index 808f9ebd0d..b248899d82 100644 --- a/src/backend/utils/cache/typcache.c +++ b/src/backend/utils/cache/typcache.c @@ -2540,6 +2540,23 @@ compare_values_of_enum(TypeCacheEntry *tcache, Oid arg1, Oid arg2) return 0; } +/* + * GetDomainConstraints --- get DomainConstraintState list of specified domain type + */ +List * +GetDomainConstraints(Oid type_id) +{ + TypeCacheEntry *typentry; + List *constraints = NIL; + + typentry = lookup_type_cache(type_id, TYPECACHE_DOMAIN_CONSTR_INFO); + + if(typentry->domainData != NULL) + constraints = typentry->domainData->constraints; + + return constraints; +} + /* * Load (or re-load) the enumData member of the typcache entry. */ diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h index 7bba77c9e7..a503eb62c2 100644 --- a/src/include/replication/logicalproto.h +++ b/src/include/replication/logicalproto.h @@ -108,6 +108,7 @@ typedef struct LogicalRepRelation char replident; /* replica identity */ char relkind; /* remote relation kind */ Bitmapset *attkeys; /* Bitmap of key columns */ + Bitmapset *attunique; /* Bitmap of unique columns */ } LogicalRepRelation; /* Type mapping info */ diff --git a/src/include/replication/logicalrelation.h b/src/include/replication/logicalrelation.h index 78cd7e77f5..4476cf7cec 100644 --- a/src/include/replication/logicalrelation.h +++ b/src/include/replication/logicalrelation.h @@ -15,6 +15,17 @@ #include "access/attmap.h" #include "replication/logicalproto.h" +/* + * States to determine volatility of the function in expressions in one + * relation. + */ +typedef enum RelFuncVolatility +{ + FUNCTION_UNKNOWN = 0, /* initializing */ + FUNCTION_IMMUTABLE, /* all functions are immutable function */ + FUNCTION_NONIMMUTABLE /* at least one non-immutable function */ +} RelFuncVolatility; + typedef struct LogicalRepRelMapEntry { LogicalRepRelation remoterel; /* key is remoterel.remoteid */ @@ -31,6 +42,11 @@ typedef struct LogicalRepRelMapEntry Relation localrel; /* relcache entry (NULL when closed) */ AttrMap *attrmap; /* map of local attributes to remote ones */ bool updatable; /* Can apply updates/deletes? */ + bool sameunique; /* Are all unique columns of the local + relation contained by the unique columns in + remote? */ + RelFuncVolatility volatility; /* all functions in localrel are + immutable function? */ /* Sync state. */ char state; diff --git a/src/include/replication/worker_internal.h b/src/include/replication/worker_internal.h index 73f728dc27..196e795dae 100644 --- a/src/include/replication/worker_internal.h +++ b/src/include/replication/worker_internal.h @@ -194,6 +194,7 @@ extern void apply_bgworker_free(ApplyBgworkerState *wstate); extern void apply_bgworker_check_status(void); extern void apply_bgworker_set_status(ApplyBgworkerStatus status); extern void apply_bgworker_subxact_info_add(TransactionId current_xid); +extern void apply_bgworker_relation_check(LogicalRepRelMapEntry *rel); static inline bool am_tablesync_worker(void) diff --git a/src/include/utils/typcache.h b/src/include/utils/typcache.h index 431ad7f1b3..ed7c2e7f48 100644 --- a/src/include/utils/typcache.h +++ b/src/include/utils/typcache.h @@ -199,6 +199,8 @@ extern uint64 assign_record_type_identifier(Oid type_id, int32 typmod); extern int compare_values_of_enum(TypeCacheEntry *tcache, Oid arg1, Oid arg2); +extern List *GetDomainConstraints(Oid type_id); + extern size_t SharedRecordTypmodRegistryEstimate(void); extern void SharedRecordTypmodRegistryInit(SharedRecordTypmodRegistry *, diff --git a/src/test/subscription/t/022_twophase_cascade.pl b/src/test/subscription/t/022_twophase_cascade.pl index b52af74e35..81791c8d3a 100644 --- a/src/test/subscription/t/022_twophase_cascade.pl +++ b/src/test/subscription/t/022_twophase_cascade.pl @@ -39,6 +39,13 @@ sub test_streaming ALTER SUBSCRIPTION tap_sub_C SET (streaming = $streaming_mode)"); + if ($streaming_mode eq 'apply') + { + $node_C->safe_psql( + 'postgres', " + ALTER TABLE test_tab ALTER c DROP DEFAULT"); + } + # Wait for subscribers to finish initialization $node_A->poll_query_until( diff --git a/src/test/subscription/t/032_streaming_apply.pl b/src/test/subscription/t/032_streaming_apply.pl new file mode 100644 index 0000000000..84a6900b33 --- /dev/null +++ b/src/test/subscription/t/032_streaming_apply.pl @@ -0,0 +1,391 @@ +# Copyright (c) 2022, PostgreSQL Global Development Group + +# Test the restrictions of streaming mode "apply" in logical replication + +use strict; +use warnings; +use PostgreSQL::Test::Cluster; +use PostgreSQL::Test::Utils; +use Test::More; + +my $offset = 0; + +# Create publisher node +my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); +$node_publisher->init(allows_streaming => 'logical'); +$node_publisher->append_conf('postgresql.conf', + 'logical_decoding_work_mem = 64kB'); +$node_publisher->start; + +# Create subscriber node +my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber'); +$node_subscriber->init; +$node_subscriber->start; + +# Setup structure on publisher +$node_publisher->safe_psql('postgres', + "CREATE TABLE test_tab (a int primary key, b varchar)"); +$node_publisher->safe_psql('postgres', + "CREATE TABLE test_tab_partitioned (a int primary key, b varchar)"); + +# Setup structure on subscriber +# We need to test normal table and partition table. +$node_subscriber->safe_psql('postgres', + "CREATE TABLE test_tab (a int primary key, b varchar)"); +$node_subscriber->safe_psql('postgres', + "CREATE TABLE test_tab_partitioned (a int primary key, b varchar) PARTITION BY RANGE(a)"); +$node_subscriber->safe_psql('postgres', + "CREATE TABLE test_tab_partition (LIKE test_tab_partitioned)"); +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partitioned ATTACH PARTITION test_tab_partition DEFAULT" +); + +# Setup logical replication +my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres'; +$node_publisher->safe_psql('postgres', + "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); +$node_publisher->safe_psql('postgres', + "CREATE PUBLICATION tap_pub_partitioned FOR TABLE test_tab_partitioned"); + +my $appname = 'tap_sub'; +$node_subscriber->safe_psql( + 'postgres', " + CREATE SUBSCRIPTION tap_sub + CONNECTION '$publisher_connstr application_name=$appname' + PUBLICATION tap_pub, tap_pub_partitioned + WITH (streaming = apply, copy_data = false)"); + +$node_publisher->wait_for_catchup($appname); + +# It is not allowed that the unique index on the publisher and the subscriber +# is different. Check the error reported by background worker in this case. +# First we check the unique index on normal table. +$node_subscriber->safe_psql('postgres', + "CREATE UNIQUE INDEX test_tab_b_idx ON test_tab (b)"); + +# Check that a background worker starts if "streaming" option is +# specified as "apply". We have to look for the DEBUG1 log messages +# about that, so temporarily bump up the log verbosity. +$node_subscriber->append_conf('postgresql.conf', "log_min_messages = debug1"); +$node_subscriber->reload; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); +$node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); +$node_subscriber->reload; + +$node_subscriber->wait_for_log( + qr/ERROR: cannot replicate relation with different unique index/, + $offset); + +# Drop the unique index on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', "DROP INDEX test_tab_b_idx"); + +$node_publisher->wait_for_catchup($appname); + +my $result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); +is($result, qq(5000), 'data replicated to subscriber after dropping index'); + +# Then we check the unique index on partition table. +$node_subscriber->safe_psql('postgres', + "CREATE UNIQUE INDEX test_tab_b_partition_idx ON test_tab_partition (b)"); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab_partitioned SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log( + qr/ERROR: cannot replicate relation with different unique index/, + $offset); + +# Drop the unique index on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', "DROP INDEX test_tab_b_partition_idx"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); +is($result, qq(5000), 'data replicated to subscriber after dropping index'); + +# Triggers which execute non-immutable function are not allowed on the +# subscriber side. Check the error reported by background worker in this case. +# First we check the trigger function on normal table. +$node_subscriber->safe_psql( + 'postgres', qq{ +CREATE FUNCTION trigger_func() RETURNS TRIGGER AS \$\$ + BEGIN + RETURN NULL; + END +\$\$ language plpgsql; +CREATE TRIGGER insert_trig +BEFORE INSERT ON test_tab +FOR EACH ROW EXECUTE PROCEDURE trigger_func(); +ALTER TABLE test_tab ENABLE REPLICA TRIGGER insert_trig; +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', "DELETE FROM test_tab"); + +$node_subscriber->wait_for_log( + qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, + $offset); + +# Drop the trigger on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "DROP TRIGGER insert_trig ON test_tab"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); +is($result, qq(0), 'data replicated to subscriber after dropping trigger'); + +# Then we check the unique index on partition table. +$node_subscriber->safe_psql( + 'postgres', qq{ +CREATE TRIGGER insert_trig +BEFORE INSERT ON test_tab_partition +FOR EACH ROW EXECUTE PROCEDURE trigger_func(); +ALTER TABLE test_tab_partition ENABLE REPLICA TRIGGER insert_trig; +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', "DELETE FROM test_tab_partitioned"); + +$node_subscriber->wait_for_log( + qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, + $offset); + +# Drop the trigger on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "DROP TRIGGER insert_trig ON test_tab_partition"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); +is($result, qq(0), 'data replicated to subscriber after dropping trigger'); + +# It is not allowed that column default value expression contains a +# non-immutable function on the subscriber side. Check the error reported by +# background worker in this case. +# First we check the column default value expression on normal table. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab ALTER COLUMN b SET DEFAULT random()"); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log( + qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, + $offset); + +# Drop default value on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab ALTER COLUMN b DROP DEFAULT"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); +is($result, qq(5000), + 'data replicated to subscriber after dropping default value expression'); + +# Then we check the column default value expression on partition table. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partition ALTER COLUMN b SET DEFAULT random()"); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab_partitioned SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log( + qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, + $offset); + +# Drop default value on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partition ALTER COLUMN b DROP DEFAULT"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); +is($result, qq(5000), + 'data replicated to subscriber after dropping default value expression'); + +# It is not allowed that domain constraint expression contains a non-immutable +# function on the subscriber side. Check the error reported by background +# worker in this case. +# Because the column type of the partition table must be the same as its parent +# table, only test normal table here. +$node_subscriber->safe_psql( + 'postgres', qq{ +CREATE DOMAIN test_domain AS int CHECK(VALUE > random()); +ALTER TABLE test_tab ALTER COLUMN a TYPE test_domain; +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', "DELETE FROM test_tab"); + +$node_subscriber->wait_for_log( + qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, + $offset); + +# Drop domain constraint expression on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab ALTER COLUMN a TYPE int"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); +is($result, qq(0), + 'data replicated to subscriber after dropping domain constraint expression' +); + +# It is not allowed that constraint expression contains a non-immutable function +# on the subscriber side. Check the error reported by background worker in this +# case. +# First we check the constraint expression on normal table. +$node_subscriber->safe_psql( + 'postgres', qq{ +ALTER TABLE test_tab ADD CONSTRAINT test_tab_con check (a > random()); +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log( + qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, + $offset); + +# Drop constraint on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab DROP CONSTRAINT test_tab_con"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); +is($result, qq(5000), + 'data replicated to subscriber after dropping constraint expression'); + +# Then we check the constraint expression on partition table. +$node_subscriber->safe_psql( + 'postgres', qq{ +ALTER TABLE test_tab_partition ADD CONSTRAINT test_tab_con check (a > random()); +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', "DELETE FROM test_tab_partitioned"); + +$node_subscriber->wait_for_log( + qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, + $offset); + +# Drop constraint on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partition DROP CONSTRAINT test_tab_con"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); +is($result, qq(0), + 'data replicated to subscriber after dropping constraint expression'); + +# It is not allowed that foreign key on the subscriber side. Check the error +# reported by background worker in this case. +# First we check the foreign key on normal table. +$node_publisher->safe_psql('postgres', "DELETE FROM test_tab"); +$node_publisher->wait_for_catchup($appname); +$node_subscriber->safe_psql( + 'postgres', qq{ +CREATE TABLE test_tab_f (a int primary key); +ALTER TABLE test_tab ADD CONSTRAINT test_tabfk FOREIGN KEY(a) REFERENCES test_tab_f(a); +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log( + qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, + $offset); + +# Drop the foreign key constraint on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab DROP CONSTRAINT test_tabfk"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); +is($result, qq(5000), + 'data replicated to subscriber after dropping the foreign key'); + +# Then we check the foreign key on partition table. +$node_publisher->wait_for_catchup($appname); +$node_subscriber->safe_psql( + 'postgres', qq{ +CREATE TABLE test_tab_partition_f (a int primary key); +ALTER TABLE test_tab_partition ADD CONSTRAINT test_tab_patition_fk FOREIGN KEY(a) REFERENCES test_tab_partition_f(a); +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab_partitioned SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log( + qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, + $offset); + +# Drop the foreign key constraint on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partition DROP CONSTRAINT test_tab_patition_fk"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); +is($result, qq(5000), + 'data replicated to subscriber after dropping the foreign key'); + +$node_subscriber->stop; +$node_publisher->stop; + +done_testing(); -- 2.23.0.windows.1 [application/octet-stream] v13-0004-Retry-to-apply-streaming-xact-only-in-apply-work.patch (25.5K, ../../OS3PR01MB6275F61CC74C5F3E47C7DD229EB59@OS3PR01MB6275.jpnprd01.prod.outlook.com/5-v13-0004-Retry-to-apply-streaming-xact-only-in-apply-work.patch) download | inline diff: From 53b9701d0f0175ac5fdd73fca44c52942201f1b7 Mon Sep 17 00:00:00 2001 From: wangw <[email protected]> Date: Wed, 15 Jun 2022 11:02:08 +0800 Subject: [PATCH v13 4/4] Retry to apply streaming xact only in apply worker. --- doc/src/sgml/catalogs.sgml | 11 ++ doc/src/sgml/ref/create_subscription.sgml | 4 + src/backend/catalog/pg_subscription.c | 1 + src/backend/catalog/system_views.sql | 4 +- src/backend/commands/subscriptioncmds.c | 1 + .../replication/logical/applybgwroker.c | 19 ++- src/backend/replication/logical/worker.c | 89 +++++++++++ src/bin/pg_dump/pg_dump.c | 5 +- src/include/catalog/pg_subscription.h | 4 + .../subscription/t/032_streaming_apply.pl | 139 ++++++++++-------- 10 files changed, 209 insertions(+), 68 deletions(-) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index 5cb39b18bd..928ad02ace 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -7907,6 +7907,17 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>subretry</structfield> <type>bool</type> + </para> + <para> + If true, the subscription will not try to apply streaming transaction + in <literal>apply</literal> mode. See + <xref linkend="sql-createsubscription"/> for more information. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>subconninfo</structfield> <type>text</type> diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml index 8de1a23ce4..70fbf81668 100644 --- a/doc/src/sgml/ref/create_subscription.sgml +++ b/doc/src/sgml/ref/create_subscription.sgml @@ -244,6 +244,10 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl is that the unique column should be the same between publisher and subscriber; the second is that there should not be any non-immutable function in subscriber-side replicated table. + When applying a streaming transaction, if either requirement is not + met, the background worker will exit with an error. And when retrying + later, we will try to apply this transaction in <literal>on</literal> + mode. </para> </listitem> </varlistentry> diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c index add51caadf..1824390d7d 100644 --- a/src/backend/catalog/pg_subscription.c +++ b/src/backend/catalog/pg_subscription.c @@ -71,6 +71,7 @@ GetSubscription(Oid subid, bool missing_ok) sub->stream = subform->substream; sub->twophasestate = subform->subtwophasestate; sub->disableonerr = subform->subdisableonerr; + sub->retry = subform->subretry; /* Get conninfo */ datum = SysCacheGetAttr(SUBSCRIPTIONOID, diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql index fedaed533b..10f4dd6785 100644 --- a/src/backend/catalog/system_views.sql +++ b/src/backend/catalog/system_views.sql @@ -1298,8 +1298,8 @@ REVOKE ALL ON pg_replication_origin_status FROM public; -- All columns of pg_subscription except subconninfo are publicly readable. REVOKE ALL ON pg_subscription FROM public; GRANT SELECT (oid, subdbid, subskiplsn, subname, subowner, subenabled, - subbinary, substream, subtwophasestate, subdisableonerr, subslotname, - subsynccommit, subpublications) + subbinary, substream, subtwophasestate, subdisableonerr, + subretry, subslotname, subsynccommit, subpublications) ON pg_subscription TO public; CREATE VIEW pg_stat_subscription_stats AS diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c index 4080bba987..38697184d9 100644 --- a/src/backend/commands/subscriptioncmds.c +++ b/src/backend/commands/subscriptioncmds.c @@ -663,6 +663,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt, LOGICALREP_TWOPHASE_STATE_PENDING : LOGICALREP_TWOPHASE_STATE_DISABLED); values[Anum_pg_subscription_subdisableonerr - 1] = BoolGetDatum(opts.disableonerr); + values[Anum_pg_subscription_subretry - 1] = BoolGetDatum(false); values[Anum_pg_subscription_subconninfo - 1] = CStringGetTextDatum(conninfo); if (opts.slot_name) diff --git a/src/backend/replication/logical/applybgwroker.c b/src/backend/replication/logical/applybgwroker.c index e004132544..a716b43082 100644 --- a/src/backend/replication/logical/applybgwroker.c +++ b/src/backend/replication/logical/applybgwroker.c @@ -109,6 +109,19 @@ apply_bgworker_can_start(TransactionId xid) if (!XLogRecPtrIsInvalid(MySubscription->skiplsn)) return false; + /* + * We don't start new background worker if retry was set as it's possible + * that the last time we tried to apply a transaction in background worker + * and the check failed (see function apply_bgworker_relation_check). So + * we will try to apply this transaction in apply worker. + */ + if (MySubscription->retry) + { + elog(DEBUG1, "retry to apply an streaming transaction in apply " + "background worker"); + return false; + } + /* * For streaming transactions that are being applied in apply background * worker, we cannot decide whether to apply the change for a relation @@ -806,8 +819,7 @@ apply_bgworker_relation_check(LogicalRepRelMapEntry *rel) if (!rel->sameunique) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("cannot replicate relation with different unique index"), - errhint("Please change the streaming option to 'on' instead of 'apply'."))); + errmsg("cannot replicate relation with different unique index"))); /* * Check if there is any non-immutable function present in expression in @@ -816,6 +828,5 @@ apply_bgworker_relation_check(LogicalRepRelMapEntry *rel) if (rel->volatility == FUNCTION_NONIMMUTABLE) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("cannot replicate relation. There is at least one non-immutable function"), - errhint("Please change the streaming option to 'on' instead of 'apply'."))); + errmsg("cannot replicate relation. There is at least one non-immutable function"))); } diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c index 7dfdbbc0ab..8c517794c3 100644 --- a/src/backend/replication/logical/worker.c +++ b/src/backend/replication/logical/worker.c @@ -377,6 +377,8 @@ static void clear_subscription_skip_lsn(XLogRecPtr finish_lsn); static inline void set_apply_error_context_xact(TransactionId xid, XLogRecPtr lsn); static inline void reset_apply_error_context_info(void); +static void set_subscription_retry(bool retry); + /* * Should this worker apply changes for given relation. * @@ -893,6 +895,9 @@ apply_handle_commit(StringInfo s) apply_handle_commit_internal(&commit_data); + /* Set the flag that we will not retry later. */ + set_subscription_retry(false); + /* Check the status of apply background worker if any. */ apply_bgworker_check_status(); @@ -1004,6 +1009,9 @@ apply_handle_prepare(StringInfo s) in_remote_transaction = false; + /* Set the flag that we will not retry later. */ + set_subscription_retry(false); + /* Check the status of apply background worker if any. */ apply_bgworker_check_status(); @@ -1057,6 +1065,9 @@ apply_handle_commit_prepared(StringInfo s) store_flush_position(prepare_data.end_lsn); in_remote_transaction = false; + /* Set the flag that we will not retry later. */ + set_subscription_retry(false); + /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(prepare_data.end_lsn); @@ -1112,6 +1123,9 @@ apply_handle_rollback_prepared(StringInfo s) store_flush_position(rollback_data.rollback_end_lsn); in_remote_transaction = false; + /* Set the flag that we will not retry later. */ + set_subscription_retry(false); + /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(rollback_data.rollback_end_lsn); @@ -1198,6 +1212,9 @@ apply_handle_stream_prepare(StringInfo s) /* unlink the files with serialized changes and subxact info. */ stream_cleanup_files(MyLogicalRepWorker->subid, prepare_data.xid); } + + /* Set the flag that we will not retry later. */ + set_subscription_retry(false); } in_remote_transaction = false; @@ -1618,6 +1635,9 @@ apply_handle_stream_abort(StringInfo s) */ else serialize_stream_abort(xid, subxid); + + /* Set the flag that we will not retry later. */ + set_subscription_retry(false); } reset_apply_error_context_info(); @@ -2802,6 +2822,9 @@ apply_handle_stream_commit(StringInfo s) /* unlink the files with serialized changes and subxact info */ stream_cleanup_files(MyLogicalRepWorker->subid, xid); } + + /* Set the flag that we will not retry later. */ + set_subscription_retry(false); } /* Check the status of apply background worker if any. */ @@ -3870,6 +3893,9 @@ start_table_sync(XLogRecPtr *origin_startpos, char **myslotname) } PG_CATCH(); { + /* Set the flag that we will retry later. */ + set_subscription_retry(true); + if (MySubscription->disableonerr) DisableSubscriptionAndExit(); else @@ -3908,6 +3934,9 @@ start_apply(XLogRecPtr origin_startpos) } PG_CATCH(); { + /* Set the flag that we will retry later. */ + set_subscription_retry(true); + if (MySubscription->disableonerr) DisableSubscriptionAndExit(); else @@ -4409,3 +4438,63 @@ reset_apply_error_context_info(void) apply_error_callback_arg.remote_attnum = -1; set_apply_error_context_xact(InvalidTransactionId, InvalidXLogRecPtr); } + +/* + * Set subretry of pg_subscription catalog. + * + * If retry is true, subscriber is about to exit with an error. Otherwise, it + * means that the changes was applied successfully. + */ +static void +set_subscription_retry(bool retry) +{ + Relation rel; + HeapTuple tup; + bool started_tx = false; + bool nulls[Natts_pg_subscription]; + bool replaces[Natts_pg_subscription]; + Datum values[Natts_pg_subscription]; + + if (MySubscription->retry == retry || + am_apply_bgworker()) + return; + + if (!IsTransactionState()) + { + StartTransactionCommand(); + started_tx = true; + } + + /* Look up the subscription in the catalog */ + rel = table_open(SubscriptionRelationId, RowExclusiveLock); + tup = SearchSysCacheCopy1(SUBSCRIPTIONOID, + ObjectIdGetDatum(MySubscription->oid)); + + if (!HeapTupleIsValid(tup)) + elog(ERROR, "subscription \"%s\" does not exist", MySubscription->name); + + LockSharedObject(SubscriptionRelationId, MySubscription->oid, 0, + AccessShareLock); + + /* Form a new tuple. */ + memset(values, 0, sizeof(values)); + memset(nulls, false, sizeof(nulls)); + memset(replaces, false, sizeof(replaces)); + + /* reset subretry */ + values[Anum_pg_subscription_subretry - 1] = BoolGetDatum(retry); + replaces[Anum_pg_subscription_subretry - 1] = true; + + tup = heap_modify_tuple(tup, RelationGetDescr(rel), values, nulls, + replaces); + + /* Update the catalog. */ + CatalogTupleUpdate(rel, &tup->t_self, tup); + + /* Cleanup. */ + heap_freetuple(tup); + table_close(rel, NoLock); + + if (started_tx) + CommitTransactionCommand(); +} diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 6f8b30abb0..9b3ffe0888 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -4472,8 +4472,9 @@ getSubscriptions(Archive *fout) ntups = PQntuples(res); /* - * Get subscription fields. We don't include subskiplsn in the dump as - * after restoring the dump this value may no longer be relevant. + * Get subscription fields. We don't include subskiplsn and subretry in + * the dump as after restoring the dump this value may no longer be + * relevant. */ i_tableoid = PQfnumber(res, "tableoid"); i_oid = PQfnumber(res, "oid"); diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h index 9b394a45fe..dc7597bf80 100644 --- a/src/include/catalog/pg_subscription.h +++ b/src/include/catalog/pg_subscription.h @@ -76,6 +76,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW bool subdisableonerr; /* True if a worker error should cause the * subscription to be disabled */ + bool subretry; /* True if the previous apply change failed. */ + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* Connection string to the publisher */ text subconninfo BKI_FORCE_NOT_NULL; @@ -115,6 +117,8 @@ typedef struct Subscription bool disableonerr; /* Indicates if the subscription should be * automatically disabled if a worker error * occurs */ + bool retry; /* Indicates if the previous apply change + * failed. */ char *conninfo; /* Connection string to the publisher */ char *slotname; /* Name of the replication slot */ char *synccommit; /* Synchronous commit setting for worker */ diff --git a/src/test/subscription/t/032_streaming_apply.pl b/src/test/subscription/t/032_streaming_apply.pl index 84a6900b33..8f2e254b39 100644 --- a/src/test/subscription/t/032_streaming_apply.pl +++ b/src/test/subscription/t/032_streaming_apply.pl @@ -58,7 +58,9 @@ $node_subscriber->safe_psql( $node_publisher->wait_for_catchup($appname); # It is not allowed that the unique index on the publisher and the subscriber -# is different. Check the error reported by background worker in this case. +# is different. Check the error reported by background worker in this case. And +# after retrying in apply worker, we check if the data is replicated +# successfully. # First we check the unique index on normal table. $node_subscriber->safe_psql('postgres', "CREATE UNIQUE INDEX test_tab_b_idx ON test_tab (b)"); @@ -82,14 +84,15 @@ $node_subscriber->wait_for_log( qr/ERROR: cannot replicate relation with different unique index/, $offset); -# Drop the unique index on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', "DROP INDEX test_tab_b_idx"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); my $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); -is($result, qq(5000), 'data replicated to subscriber after dropping index'); +is($result, qq(5000), 'data replicated to subscribers after retrying because of unique index'); + +# Drop the unique index on the subscriber. +$node_subscriber->safe_psql('postgres', "DROP INDEX test_tab_b_idx"); # Then we check the unique index on partition table. $node_subscriber->safe_psql('postgres', @@ -106,17 +109,20 @@ $node_subscriber->wait_for_log( qr/ERROR: cannot replicate relation with different unique index/, $offset); -# Drop the unique index on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', "DROP INDEX test_tab_b_partition_idx"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); -is($result, qq(5000), 'data replicated to subscriber after dropping index'); +is($result, qq(5000), 'data replicated to subscribers after retrying because of unique index'); + +# Drop the unique index on the subscriber. +$node_subscriber->safe_psql('postgres', "DROP INDEX test_tab_b_partition_idx"); # Triggers which execute non-immutable function are not allowed on the # subscriber side. Check the error reported by background worker in this case. +# And after retrying in apply worker, we check if the data is replicated +# successfully. # First we check the trigger function on normal table. $node_subscriber->safe_psql( 'postgres', qq{ @@ -140,15 +146,16 @@ $node_subscriber->wait_for_log( qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, $offset); -# Drop the trigger on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "DROP TRIGGER insert_trig ON test_tab"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); -is($result, qq(0), 'data replicated to subscriber after dropping trigger'); +is($result, qq(0), 'data replicated to subscribers after retrying because of trigger'); + +# Drop the trigger on the subscriber. +$node_subscriber->safe_psql('postgres', + "DROP TRIGGER insert_trig ON test_tab"); # Then we check the unique index on partition table. $node_subscriber->safe_psql( @@ -168,19 +175,21 @@ $node_subscriber->wait_for_log( qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, $offset); -# Drop the trigger on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "DROP TRIGGER insert_trig ON test_tab_partition"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); -is($result, qq(0), 'data replicated to subscriber after dropping trigger'); +is($result, qq(0), 'data replicated to subscribers after retrying because of trigger'); + +# Drop the trigger on the subscriber. +$node_subscriber->safe_psql('postgres', + "DROP TRIGGER insert_trig ON test_tab_partition"); # It is not allowed that column default value expression contains a # non-immutable function on the subscriber side. Check the error reported by -# background worker in this case. +# background worker in this case. And after retrying in apply worker, we check +# if the data is replicated successfully. # First we check the column default value expression on normal table. $node_subscriber->safe_psql('postgres', "ALTER TABLE test_tab ALTER COLUMN b SET DEFAULT random()"); @@ -196,16 +205,17 @@ $node_subscriber->wait_for_log( qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, $offset); -# Drop default value on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "ALTER TABLE test_tab ALTER COLUMN b DROP DEFAULT"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); is($result, qq(5000), - 'data replicated to subscriber after dropping default value expression'); + 'data replicated to subscribers after retrying because of column default value'); + +# Drop default value on the subscriber. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab ALTER COLUMN b DROP DEFAULT"); # Then we check the column default value expression on partition table. $node_subscriber->safe_psql('postgres', @@ -222,20 +232,22 @@ $node_subscriber->wait_for_log( qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, $offset); -# Drop default value on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "ALTER TABLE test_tab_partition ALTER COLUMN b DROP DEFAULT"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); is($result, qq(5000), - 'data replicated to subscriber after dropping default value expression'); + 'data replicated to subscribers after retrying because of column default value'); + +# Drop default value on the subscriber. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partition ALTER COLUMN b DROP DEFAULT"); # It is not allowed that domain constraint expression contains a non-immutable # function on the subscriber side. Check the error reported by background -# worker in this case. +# worker in this case. And after retrying in apply worker, we check if the data +# is replicated successfully. # Because the column type of the partition table must be the same as its parent # table, only test normal table here. $node_subscriber->safe_psql( @@ -253,21 +265,23 @@ $node_subscriber->wait_for_log( qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, $offset); -# Drop domain constraint expression on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "ALTER TABLE test_tab ALTER COLUMN a TYPE int"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); is($result, qq(0), - 'data replicated to subscriber after dropping domain constraint expression' + 'data replicated to subscribers after retrying because of domain' ); -# It is not allowed that constraint expression contains a non-immutable function -# on the subscriber side. Check the error reported by background worker in this -# case. +# Drop domain constraint expression on the subscriber. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab ALTER COLUMN a TYPE int"); + +# It is not allowed that constraint expression contains a non-immutable +# function on the subscriber side. Check the error reported by background +# worker in this case. And after retrying in apply worker, we check if the data +# is replicated successfully. # First we check the constraint expression on normal table. $node_subscriber->safe_psql( 'postgres', qq{ @@ -285,16 +299,17 @@ $node_subscriber->wait_for_log( qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, $offset); -# Drop constraint on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "ALTER TABLE test_tab DROP CONSTRAINT test_tab_con"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); is($result, qq(5000), - 'data replicated to subscriber after dropping constraint expression'); + 'data replicated to subscribers after retrying because of constraint'); + +# Drop constraint on the subscriber. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab DROP CONSTRAINT test_tab_con"); # Then we check the constraint expression on partition table. $node_subscriber->safe_psql( @@ -311,19 +326,21 @@ $node_subscriber->wait_for_log( qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, $offset); -# Drop constraint on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "ALTER TABLE test_tab_partition DROP CONSTRAINT test_tab_con"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); is($result, qq(0), - 'data replicated to subscriber after dropping constraint expression'); + 'data replicated to subscribers after retrying because of constraint'); + +# Drop constraint on the subscriber. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partition DROP CONSTRAINT test_tab_con"); # It is not allowed that foreign key on the subscriber side. Check the error -# reported by background worker in this case. +# reported by background worker in this case. And after retrying in apply +# worker, we check if the data is replicated successfully. # First we check the foreign key on normal table. $node_publisher->safe_psql('postgres', "DELETE FROM test_tab"); $node_publisher->wait_for_catchup($appname); @@ -344,16 +361,17 @@ $node_subscriber->wait_for_log( qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, $offset); -# Drop the foreign key constraint on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "ALTER TABLE test_tab DROP CONSTRAINT test_tabfk"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); is($result, qq(5000), - 'data replicated to subscriber after dropping the foreign key'); + 'data replicated to subscribers after retrying because of foreign key'); + +# Drop the foreign key constraint on the subscriber. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab DROP CONSTRAINT test_tabfk"); # Then we check the foreign key on partition table. $node_publisher->wait_for_catchup($appname); @@ -374,16 +392,17 @@ $node_subscriber->wait_for_log( qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, $offset); -# Drop the foreign key constraint on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "ALTER TABLE test_tab_partition DROP CONSTRAINT test_tab_patition_fk"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); is($result, qq(5000), - 'data replicated to subscriber after dropping the foreign key'); + 'data replicated to subscribers after retrying because of foreign key'); + +# Drop the foreign key constraint on the subscriber. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partition DROP CONSTRAINT test_tab_patition_fk"); $node_subscriber->stop; $node_publisher->stop; -- 2.23.0.windows.1 ^ permalink raw reply [nested|flat] 66+ messages in thread
* Re: Perform streaming logical transactions by background workers and parallel apply @ 2022-06-23 08:43 Amit Kapila <[email protected]> parent: [email protected] <[email protected]> 0 siblings, 1 reply; 66+ messages in thread From: Amit Kapila @ 2022-06-23 08:43 UTC (permalink / raw) To: [email protected] <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Peter Smith <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; PostgreSQL Hackers <[email protected]> On Thu, Jun 23, 2022 at 12:51 PM [email protected] <[email protected]> wrote: > > On Mon, Jun 20, 2022 at 11:00 AM Amit Kapila <[email protected]> wrote: > > I have improved the comments in this and other related sections of the > > patch. See attached. > Thanks for your comments and patch! > Improved the comments as you suggested. > > > > > 3. > > > > + > > > > + <para> > > > > + Setting streaming mode to <literal>apply</literal> could export invalid > > LSN > > > > + as finish LSN of failed transaction. Changing the streaming mode and > > making > > > > + the same conflict writes the finish LSN of the failed transaction in the > > > > + server log if required. > > > > + </para> > > > > > > > > How will the user identify that this is an invalid LSN value and she > > > > shouldn't use it to SKIP the transaction? Can we change the second > > > > sentence to: "User should change the streaming mode to 'on' if they > > > > would instead wish to see the finish LSN on error. Users can use > > > > finish LSN to SKIP applying the transaction." I think we can give > > > > reference to docs where the SKIP feature is explained. > > > Improved the sentence as suggested. > > > > > > > You haven't answered first part of the comment: "How will the user > > identify that this is an invalid LSN value and she shouldn't use it to > > SKIP the transaction?". Have you checked what value it displays? For > > example, in one of the case in apply_error_callback as shown in below > > code, we don't even display finish LSN if it is invalid. > > else if (XLogRecPtrIsInvalid(errarg->finish_lsn)) > > errcontext("processing remote data for replication origin \"%s\" > > during \"%s\" in transaction %u", > > errarg->origin_name, > > logicalrep_message_type(errarg->command), > > errarg->remote_xid); > I am sorry that I missed something in my previous reply. > The invalid LSN value here is to say InvalidXLogRecPtr (0/0). > Here is an example : > ``` > 2022-06-23 14:30:11.343 CST [822333] logical replication worker CONTEXT: processing remote data for replication origin "pg_16389" during "INSERT" for replication target relation "public.tab" in transaction 727 finished at 0/0 > ``` > I don't think it is a good idea to display invalid values. We can mask this as we are doing in other cases in function apply_error_callback. The ideal way is that we provide a view/system table for users to check these errors but that is a matter of another patch. So users probably need to check Logs to see if the error is from a background apply worker to decide whether or not to switch streaming mode. -- With Regards, Amit Kapila. ^ permalink raw reply [nested|flat] 66+ messages in thread
* RE: Perform streaming logical transactions by background workers and parallel apply @ 2022-06-28 03:21 [email protected] <[email protected]> parent: Amit Kapila <[email protected]> 0 siblings, 5 replies; 66+ messages in thread From: [email protected] @ 2022-06-28 03:21 UTC (permalink / raw) To: Amit Kapila <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Peter Smith <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; PostgreSQL Hackers <[email protected]> On Thu, Jun 23, 2022 at 16:44 PM Amit Kapila <[email protected]> wrote: > On Thu, Jun 23, 2022 at 12:51 PM [email protected] > <[email protected]> wrote: > > > > On Mon, Jun 20, 2022 at 11:00 AM Amit Kapila <[email protected]> > wrote: > > > I have improved the comments in this and other related sections of the > > > patch. See attached. > > Thanks for your comments and patch! > > Improved the comments as you suggested. > > > > > > > 3. > > > > > + > > > > > + <para> > > > > > + Setting streaming mode to <literal>apply</literal> could export invalid > > > LSN > > > > > + as finish LSN of failed transaction. Changing the streaming mode and > > > making > > > > > + the same conflict writes the finish LSN of the failed transaction in the > > > > > + server log if required. > > > > > + </para> > > > > > > > > > > How will the user identify that this is an invalid LSN value and she > > > > > shouldn't use it to SKIP the transaction? Can we change the second > > > > > sentence to: "User should change the streaming mode to 'on' if they > > > > > would instead wish to see the finish LSN on error. Users can use > > > > > finish LSN to SKIP applying the transaction." I think we can give > > > > > reference to docs where the SKIP feature is explained. > > > > Improved the sentence as suggested. > > > > > > > > > > You haven't answered first part of the comment: "How will the user > > > identify that this is an invalid LSN value and she shouldn't use it to > > > SKIP the transaction?". Have you checked what value it displays? For > > > example, in one of the case in apply_error_callback as shown in below > > > code, we don't even display finish LSN if it is invalid. > > > else if (XLogRecPtrIsInvalid(errarg->finish_lsn)) > > > errcontext("processing remote data for replication origin \"%s\" > > > during \"%s\" in transaction %u", > > > errarg->origin_name, > > > logicalrep_message_type(errarg->command), > > > errarg->remote_xid); > > I am sorry that I missed something in my previous reply. > > The invalid LSN value here is to say InvalidXLogRecPtr (0/0). > > Here is an example : > > ``` > > 2022-06-23 14:30:11.343 CST [822333] logical replication worker CONTEXT: > processing remote data for replication origin "pg_16389" during "INSERT" for > replication target relation "public.tab" in transaction 727 finished at 0/0 > > ``` > > > > I don't think it is a good idea to display invalid values. We can mask > this as we are doing in other cases in function apply_error_callback. > The ideal way is that we provide a view/system table for users to > check these errors but that is a matter of another patch. So users > probably need to check Logs to see if the error is from a background > apply worker to decide whether or not to switch streaming mode. Thanks for your comments. I improved it as you suggested. I mask the LSN if it is invalid LSN(0/0). Also, I improved the related pg-doc as following: ``` When the streaming mode is <literal>parallel</literal>, the finish LSN of failed transactions may not be logged. In that case, it may be necessary to change the streaming mode to <literal>on</literal> and cause the same conflicts again so the finish LSN of the failed transaction will be written to the server log. For the usage of finish LSN, please refer to <link linkend="sql-altersubscription"><command>ALTER SUBSCRIPTION ... SKIP</command></link>. ``` After improving this (mask invalid LSN), I found that this improvement and parallel apply patch do not seem to have a strong correlation. Would it be better to improve and commit in another separate patch? I also improved patches as suggested by Peter-san in [1] and [2]. Thanks for Shi Yu to improve the patches by addressing the comments in [2]. Attach the new patches. [1] - https://www.postgresql.org/message-id/CAHut%2BPtu_eWOVWAKrwkUFdTAh_r-RZsbDFkFmKwEAmxws%3DSh5w%40mail... [2] - https://www.postgresql.org/message-id/CAHut%2BPsDzRu6PD1uSRkftRXef-KwrOoYrcq7Cm0v4otisi5M%2Bg%40mail... Regards, Wang wei Attachments: [application/octet-stream] v14-0001-Perform-streaming-logical-transactions-by-backgr.patch (103.1K, ../../OS3PR01MB62758DBE8FA12BA72A43AC819EB89@OS3PR01MB6275.jpnprd01.prod.outlook.com/2-v14-0001-Perform-streaming-logical-transactions-by-backgr.patch) download | inline diff: From aec2c6d6d44ebf5d070e042b4420f6154ddb44c3 Mon Sep 17 00:00:00 2001 From: "houzj.fnst" <[email protected]> Date: Wed, 20 Apr 2022 16:45:07 +0800 Subject: [PATCH v14 1/4] Perform streaming logical transactions by background workers Currently, for large transactions, the publisher sends the data in multiple streams (changes divided into chunks depending upon logical_decoding_work_mem), and then on the subscriber-side, the apply worker writes the changes into temporary files and once it receives the commit, it reads from the file and applies the entire transaction. To improve the performance of such transactions, we can instead allow them to be applied via background workers. In this approach, we assign a new apply background worker (if available) as soon as the xact's first stream is received and the main apply worker will send changes to this new worker via shared memory. The apply background worker will directly apply the change instead of writing it to temporary files. We keep this worker assigned till the transaction commit is received and also wait for the worker to finish at commit. This preserves commit ordering and avoids writing to and reading from file in most cases. We still need to spill if there is no worker available. This patch also extends the SUBSCRIPTION 'streaming' parameter so that the user can control whether to apply the streaming transaction in an apply background worker or spill the change to disk. The user can set the streaming parameter to 'on/off', 'parallel'. For now, 'parallel' means the streaming will be applied via a apply background worker if available. 'on' means the streaming transaction will be spilled to disk. By the way, we do not change the default behaviour. --- doc/src/sgml/catalogs.sgml | 10 +- doc/src/sgml/config.sgml | 25 + doc/src/sgml/logical-replication.sgml | 10 + doc/src/sgml/protocol.sgml | 19 + doc/src/sgml/ref/create_subscription.sgml | 24 +- src/backend/access/transam/xact.c | 13 + src/backend/commands/subscriptioncmds.c | 66 +- src/backend/postmaster/bgworker.c | 3 + src/backend/replication/logical/Makefile | 3 +- .../replication/logical/applybgworker.c | 775 ++++++++++++++++++ src/backend/replication/logical/decode.c | 10 +- src/backend/replication/logical/launcher.c | 130 ++- src/backend/replication/logical/origin.c | 26 +- src/backend/replication/logical/proto.c | 35 +- .../replication/logical/reorderbuffer.c | 10 +- src/backend/replication/logical/tablesync.c | 10 +- src/backend/replication/logical/worker.c | 682 +++++++++++---- src/backend/replication/pgoutput/pgoutput.c | 2 +- src/backend/utils/activity/wait_event.c | 3 + src/backend/utils/misc/guc.c | 12 + src/backend/utils/misc/postgresql.conf.sample | 1 + src/bin/pg_dump/pg_dump.c | 6 +- src/include/catalog/pg_subscription.h | 18 +- src/include/replication/logicallauncher.h | 1 + src/include/replication/logicalproto.h | 19 +- src/include/replication/logicalworker.h | 1 + src/include/replication/origin.h | 2 +- src/include/replication/reorderbuffer.h | 7 +- src/include/replication/worker_internal.h | 102 ++- src/include/utils/wait_event.h | 1 + src/test/regress/expected/subscription.out | 2 +- src/tools/pgindent/typedefs.list | 5 + 32 files changed, 1814 insertions(+), 219 deletions(-) create mode 100644 src/backend/replication/logical/applybgworker.c diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index 25b02c4e37..f73fae39b6 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -7873,11 +7873,15 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l <row> <entry role="catalog_table_entry"><para role="column_definition"> - <structfield>substream</structfield> <type>bool</type> + <structfield>substream</structfield> <type>char</type> </para> <para> - If true, the subscription will allow streaming of in-progress - transactions + Controls how to handle the streaming of in-progress transactions: + <literal>f</literal> = disallow streaming of in-progress transactions, + <literal>t</literal> = spill the changes of in-progress transactions to + disk and apply at once after the transaction is committed on the + publisher, + <literal>p</literal> = apply changes directly using a background worker </para></entry> </row> diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml index 48478b1024..faed900c50 100644 --- a/doc/src/sgml/config.sgml +++ b/doc/src/sgml/config.sgml @@ -4970,6 +4970,31 @@ ANY <replaceable class="parameter">num_sync</replaceable> ( <replaceable class=" </listitem> </varlistentry> + <varlistentry id="guc-max-apply-bgworkers-per-subscription" xreflabel="max_apply_bgworkers_per_subscription"> + <term><varname>max_apply_bgworkers_per_subscription</varname> (<type>integer</type>) + <indexterm> + <primary><varname>max_apply_bgworkers_per_subscription</varname> configuration parameter</primary> + </indexterm> + </term> + <listitem> + <para> + Maximum number of apply background workers per subscription. This + parameter controls the amount of parallelism of the streaming of + in-progress transactions when subscription parameter + <literal>streaming = parallel</literal>. + </para> + <para> + The apply background workers are taken from the pool defined by + <varname>max_logical_replication_workers</varname>. + </para> + <para> + The default value is 2. This parameter can only be set in the + <filename>postgresql.conf</filename> file or on the server command + line. + </para> + </listitem> + </varlistentry> + </variablelist> </sect2> diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml index bdf1e7b727..92997f9299 100644 --- a/doc/src/sgml/logical-replication.sgml +++ b/doc/src/sgml/logical-replication.sgml @@ -1153,6 +1153,16 @@ CONTEXT: processing remote data for replication origin "pg_16395" during "INSER might not violate any constraint. This can easily make the subscriber inconsistent. </para> + + <para> + When the streaming mode is <literal>parallel</literal>, the finish LSN of + failed transactions may not be logged. In that case, it may be necessary to + change the streaming mode to <literal>on</literal> and cause the same + conflicts again so the finish LSN of the failed transaction will be written + to the server log. For the usage of finish LSN, please refer to <link + linkend="sql-altersubscription"><command>ALTER SUBSCRIPTION ... + SKIP</command></link>. + </para> </sect1> <sect1 id="logical-replication-restrictions"> diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml index a94743b587..29f3a698b7 100644 --- a/doc/src/sgml/protocol.sgml +++ b/doc/src/sgml/protocol.sgml @@ -6809,6 +6809,25 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;" </listitem> </varlistentry> + <varlistentry> + <term>Int64 (XLogRecPtr)</term> + <listitem> + <para> + The LSN of the abort. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Int64 (TimestampTz)</term> + <listitem> + <para> + Abort timestamp of the transaction. The value is in number + of microseconds since PostgreSQL epoch (2000-01-01). + </para> + </listitem> + </varlistentry> + <varlistentry> <term>Int32 (TransactionId)</term> <listitem> diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml index 34b3264b26..7cf0d2a977 100644 --- a/doc/src/sgml/ref/create_subscription.sgml +++ b/doc/src/sgml/ref/create_subscription.sgml @@ -217,13 +217,29 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl </varlistentry> <varlistentry> - <term><literal>streaming</literal> (<type>boolean</type>)</term> + <term><literal>streaming</literal> (<type>enum</type>)</term> <listitem> <para> Specifies whether to enable streaming of in-progress transactions - for this subscription. By default, all transactions - are fully decoded on the publisher and only then sent to the - subscriber as a whole. + for this subscription. The default value is <literal>off</literal>, + meaning all transactions are fully decoded on the publisher and only + then sent to the subscriber as a whole. + </para> + + <para> + If set to <literal>on</literal>, the changes of transaction are + written to temporary files and then applied at once after the + transaction is committed on the publisher. + </para> + + <para> + If set to <literal>parallel</literal>, incoming changes are directly + applied via one of the apply background workers, if available. If no + background worker is free to handle streaming transaction then the + changes are written to a file and applied after the transaction is + committed. Note that if an error happens when applying changes in a + background worker, the finish LSN of the remote transaction might + not be reported in the server log. </para> </listitem> </varlistentry> diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c index 47d80b0d25..678540ba25 100644 --- a/src/backend/access/transam/xact.c +++ b/src/backend/access/transam/xact.c @@ -1711,6 +1711,7 @@ RecordTransactionAbort(bool isSubXact) int nchildren; TransactionId *children; TimestampTz xact_time; + bool replorigin; /* * If we haven't been assigned an XID, nobody will care whether we aborted @@ -1741,6 +1742,13 @@ RecordTransactionAbort(bool isSubXact) elog(PANIC, "cannot abort transaction %u, it was already committed", xid); + /* + * Are we using the replication origins feature? Or, in other words, + * are we replaying remote actions? + */ + replorigin = (replorigin_session_origin != InvalidRepOriginId && + replorigin_session_origin != DoNotReplicateId); + /* Fetch the data we need for the abort record */ nrels = smgrGetPendingDeletes(false, &rels); nchildren = xactGetCommittedChildren(&children); @@ -1765,6 +1773,11 @@ RecordTransactionAbort(bool isSubXact) MyXactFlags, InvalidTransactionId, NULL); + if (replorigin) + /* Move LSNs forward for this replication origin */ + replorigin_session_advance(replorigin_session_origin_lsn, + XactLastRecEnd); + /* * Report the latest async abort LSN, so that the WAL writer knows to * flush this abort. There's nothing to be gained by delaying this, since diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c index 83e6eae855..467ff11e39 100644 --- a/src/backend/commands/subscriptioncmds.c +++ b/src/backend/commands/subscriptioncmds.c @@ -83,7 +83,7 @@ typedef struct SubOpts bool copy_data; bool refresh; bool binary; - bool streaming; + char streaming; bool twophase; bool disableonerr; XLogRecPtr lsn; @@ -95,6 +95,62 @@ static List *merge_publications(List *oldpublist, List *newpublist, bool addpub, static void ReportSlotConnectionError(List *rstates, Oid subid, char *slotname, char *err); +/* + * Extract the streaming mode value from a DefElem. This is like + * defGetBoolean() but also accepts the special value of "parallel". + */ +static char +defGetStreamingMode(DefElem *def) +{ + /* + * If no parameter given, assume "true" is meant. + */ + if (def->arg == NULL) + return SUBSTREAM_ON; + + /* + * Allow 0, 1, "false", "true", "off", "on" or "parallel". + */ + switch (nodeTag(def->arg)) + { + case T_Integer: + switch (intVal(def->arg)) + { + case 0: + return SUBSTREAM_OFF; + case 1: + return SUBSTREAM_ON; + default: + /* otherwise, error out below */ + break; + } + break; + default: + { + char *sval = defGetString(def); + + /* + * The set of strings accepted here should match up with the + * grammar's opt_boolean_or_string production. + */ + if (pg_strcasecmp(sval, "false") == 0 || + pg_strcasecmp(sval, "off") == 0) + return SUBSTREAM_OFF; + if (pg_strcasecmp(sval, "true") == 0 || + pg_strcasecmp(sval, "on") == 0) + return SUBSTREAM_ON; + if (pg_strcasecmp(sval, "parallel") == 0) + return SUBSTREAM_PARALLEL; + } + break; + } + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s requires a Boolean value or \"parallel\"", + def->defname))); + return SUBSTREAM_OFF; /* keep compiler quiet */ +} + /* * Common option parsing function for CREATE and ALTER SUBSCRIPTION commands. * @@ -132,7 +188,7 @@ parse_subscription_options(ParseState *pstate, List *stmt_options, if (IsSet(supported_opts, SUBOPT_BINARY)) opts->binary = false; if (IsSet(supported_opts, SUBOPT_STREAMING)) - opts->streaming = false; + opts->streaming = SUBSTREAM_OFF; if (IsSet(supported_opts, SUBOPT_TWOPHASE_COMMIT)) opts->twophase = false; if (IsSet(supported_opts, SUBOPT_DISABLE_ON_ERR)) @@ -233,7 +289,7 @@ parse_subscription_options(ParseState *pstate, List *stmt_options, errorConflictingDefElem(defel, pstate); opts->specified_opts |= SUBOPT_STREAMING; - opts->streaming = defGetBoolean(defel); + opts->streaming = defGetStreamingMode(defel); } else if (strcmp(defel->defname, "two_phase") == 0) { @@ -601,7 +657,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt, values[Anum_pg_subscription_subowner - 1] = ObjectIdGetDatum(owner); values[Anum_pg_subscription_subenabled - 1] = BoolGetDatum(opts.enabled); values[Anum_pg_subscription_subbinary - 1] = BoolGetDatum(opts.binary); - values[Anum_pg_subscription_substream - 1] = BoolGetDatum(opts.streaming); + values[Anum_pg_subscription_substream - 1] = CharGetDatum(opts.streaming); values[Anum_pg_subscription_subtwophasestate - 1] = CharGetDatum(opts.twophase ? LOGICALREP_TWOPHASE_STATE_PENDING : @@ -1060,7 +1116,7 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt, if (IsSet(opts.specified_opts, SUBOPT_STREAMING)) { values[Anum_pg_subscription_substream - 1] = - BoolGetDatum(opts.streaming); + CharGetDatum(opts.streaming); replaces[Anum_pg_subscription_substream - 1] = true; } diff --git a/src/backend/postmaster/bgworker.c b/src/backend/postmaster/bgworker.c index 40601aefd9..40ccb8993c 100644 --- a/src/backend/postmaster/bgworker.c +++ b/src/backend/postmaster/bgworker.c @@ -128,6 +128,9 @@ static const struct }, { "ApplyWorkerMain", ApplyWorkerMain + }, + { + "ApplyBgworkerMain", ApplyBgworkerMain } }; diff --git a/src/backend/replication/logical/Makefile b/src/backend/replication/logical/Makefile index c4e2fdeb71..059fe86d93 100644 --- a/src/backend/replication/logical/Makefile +++ b/src/backend/replication/logical/Makefile @@ -26,6 +26,7 @@ OBJS = \ reorderbuffer.o \ snapbuild.o \ tablesync.o \ - worker.o + worker.o \ + applybgworker.o include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/replication/logical/applybgworker.c b/src/backend/replication/logical/applybgworker.c new file mode 100644 index 0000000000..dd70d333e0 --- /dev/null +++ b/src/backend/replication/logical/applybgworker.c @@ -0,0 +1,775 @@ +/*------------------------------------------------------------------------- + * applybgworker.c + * Support routines for applying xact by apply background worker + * + * Copyright (c) 2016-2022, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/replication/logical/applybgworker.c + * + * This file contains routines that are intended to support setting up, using, + * and tearing down a ApplyBgworkerState. + * + * Refer to the comments in file header of logical/worker.c to see more + * information about apply background worker. + * + *------------------------------------------------------------------------- + */ + +#include "postgres.h" + +#include "libpq/pqformat.h" +#include "mb/pg_wchar.h" +#include "pgstat.h" +#include "postmaster/interrupt.h" +#include "replication/logicallauncher.h" +#include "replication/logicalworker.h" +#include "replication/origin.h" +#include "replication/walreceiver.h" +#include "replication/worker_internal.h" +#include "storage/ipc.h" +#include "storage/procarray.h" +#include "tcop/tcopprot.h" +#include "utils/inval.h" +#include "utils/memutils.h" +#include "utils/resowner.h" +#include "utils/syscache.h" + +#define PG_LOGICAL_APPLY_SHM_MAGIC 0x79fb2447 // TODO Consider change + +/* + * DSM keys for apply background worker. Unlike other parallel execution code, + * since we don't need to worry about DSM keys conflicting with plan_node_id we + * can use small integers. + */ +#define APPLY_BGWORKER_KEY_SHARED 1 +#define APPLY_BGWORKER_KEY_MQ 2 + +/* queue size of DSM, 16 MB for now. */ +#define DSM_QUEUE_SIZE 160000000 + +/* + * There are three fields in message: start_lsn, end_lsn and send_time. Because + * we have updated these statistics in apply worker, we could ignore these + * fields in apply background worker. (see function LogicalRepApplyLoop) + */ +#define IGNORE_SIZE_IN_MESSAGE (3 * sizeof(uint64)) + +/* + * Entry for a hash table we use to map from xid to our apply background worker + * state. + */ +typedef struct ApplyBgworkerEntry +{ + TransactionId xid; + ApplyBgworkerState *wstate; +} ApplyBgworkerEntry; + +/* Apply background workers hash table (initialized on first use) */ +static HTAB *ApplyWorkersHash = NULL; +static List *ApplyWorkersFreeList = NIL; +static List *ApplyWorkersList = NIL; + +/* + * Information shared between main apply worker and apply background worker. + */ +volatile ApplyBgworkerShared *MyParallelState = NULL; + +List *subxactlist = NIL; + +static bool apply_bgworker_can_start(TransactionId xid); +static ApplyBgworkerState *apply_bgworker_setup(void); +static void apply_bgworker_setup_dsm(ApplyBgworkerState *wstate); + +/* + * Confirm if we can try to start a new apply background worker. + */ +static bool +apply_bgworker_can_start(TransactionId xid) +{ + if (!TransactionIdIsValid(xid)) + return false; + + /* + * We don't start new background worker if we are not in streaming parallel + * mode. + */ + if (MySubscription->stream != SUBSTREAM_PARALLEL) + return false; + + /* + * We don't start new background worker if user has set skiplsn as it's + * possible that user want to skip the streaming transaction. For + * streaming transaction, we need to spill the transaction to disk so that + * we can get the last LSN of the transaction to judge whether to skip + * before starting to apply the change. + */ + if (!XLogRecPtrIsInvalid(MySubscription->skiplsn)) + return false; + + /* + * For streaming transactions that are being applied in apply background + * worker, we cannot decide whether to apply the change for a relation + * that is not in the READY state (see should_apply_changes_for_rel) as we + * won't know remote_final_lsn by that time. So, we don't start new apply + * background worker in this case. + */ + if (!AllTablesyncsReady()) + return false; + + return true; +} + +/* + * Try to start worker inside ApplyWorkersHash for requested xid. + */ +ApplyBgworkerState * +apply_bgworker_start(TransactionId xid) +{ + bool found; + ApplyBgworkerState *wstate; + ApplyBgworkerEntry *entry = NULL; + + if (!apply_bgworker_can_start(xid)) + return NULL; + + /* First time through, initialize apply workers hashtable */ + if (ApplyWorkersHash == NULL) + { + HASHCTL ctl; + + MemSet(&ctl, 0, sizeof(ctl)); + ctl.keysize = sizeof(TransactionId); + ctl.entrysize = sizeof(ApplyBgworkerEntry); + ctl.hcxt = ApplyContext; + + ApplyWorkersHash = hash_create("logical apply workers hash", 8, &ctl, + HASH_ELEM | HASH_BLOBS | HASH_CONTEXT); + } + + /* + * Now, we try to get an apply background worker. If there is at least one + * worker in the free list, then take one. Otherwise, we try to start a + * new apply background worker. + */ + if (list_length(ApplyWorkersFreeList) > 0) + { + wstate = (ApplyBgworkerState *) llast(ApplyWorkersFreeList); + ApplyWorkersFreeList = list_delete_last(ApplyWorkersFreeList); + } + else + { + wstate = apply_bgworker_setup(); + + if (wstate == NULL) + return NULL; + } + + /* + * Create entry for requested transaction. + */ + entry = hash_search(ApplyWorkersHash, &xid, HASH_ENTER, &found); + if (found) + elog(ERROR, "hash table corrupted"); + + /* Fill up the hash entry */ + wstate->pstate->status = APPLY_BGWORKER_BUSY; + wstate->pstate->server_version = walrcv_server_version(LogRepWorkerWalRcvConn); + wstate->pstate->stream_xid = xid; + entry->wstate = wstate; + entry->xid = xid; + + return wstate; +} + +/* + * Try to look up worker inside ApplyWorkersHash for requested xid. + */ +ApplyBgworkerState * +apply_bgworker_find(TransactionId xid) +{ + bool found; + ApplyBgworkerEntry *entry = NULL; + + if (!TransactionIdIsValid(xid)) + return NULL; + + if (ApplyWorkersHash == NULL) + return NULL; + + /* + * Find entry for requested transaction. + */ + entry = hash_search(ApplyWorkersHash, &xid, HASH_FIND, &found); + if (found) + { + entry->wstate->pstate->status = APPLY_BGWORKER_BUSY; + return entry->wstate; + } + else + return NULL; +} + +/* + * Add the worker to the free list and remove the entry from the hash table. + */ +void +apply_bgworker_free(ApplyBgworkerState *wstate) +{ + MemoryContext oldctx; + TransactionId xid = wstate->pstate->stream_xid; + + Assert(wstate->pstate->status == APPLY_BGWORKER_FINISHED); + + oldctx = MemoryContextSwitchTo(ApplyContext); + + hash_search(ApplyWorkersHash, &xid, HASH_REMOVE, NULL); + + elog(DEBUG1, "adding finished apply worker #%u for xid %u to the free list", + wstate->pstate->n, wstate->pstate->stream_xid); + + ApplyWorkersFreeList = lappend(ApplyWorkersFreeList, wstate); + + MemoryContextSwitchTo(oldctx); +} + +/* Apply Background Worker main loop */ +static void +LogicalApplyBgwLoop(shm_mq_handle *mqh, volatile ApplyBgworkerShared *pst) +{ + shm_mq_result shmq_res; + PGPROC *registrant; + ErrorContextCallback errcallback; + + registrant = BackendPidGetProc(MyBgworkerEntry->bgw_notify_pid); + SetLatch(®istrant->procLatch); + + /* + * Push apply error context callback. Fields will be filled applying + * applying a change. + */ + errcallback.callback = apply_error_callback; + errcallback.previous = error_context_stack; + error_context_stack = &errcallback; + + for (;;) + { + void *data; + Size len; + int c; + StringInfoData s; + MemoryContext oldctx; + + CHECK_FOR_INTERRUPTS(); + + /* Ensure we are reading the data into our memory context. */ + oldctx = MemoryContextSwitchTo(ApplyMessageContext); + + shmq_res = shm_mq_receive(mqh, &len, &data, false); + + if (shmq_res != SHM_MQ_SUCCESS) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("lost connection to the main apply worker"))); + + if (len == 0) + break; + + s.cursor = 0; + s.maxlen = -1; + s.data = (char *) data; + s.len = len; + + /* + * We use first byte of message for additional communication between + * main Logical replication worker and apply background workers, so if + * it differs from 'w', then process it first. + */ + c = pq_getmsgbyte(&s); + switch (c) + { + /* End message of streaming chunk */ + case LOGICAL_REP_MSG_STREAM_STOP: + elog(DEBUG1, "[Apply BGW #%u] ended processing streaming chunk," + "waiting on shm_mq_receive", pst->n); + + in_streamed_transaction = false; + pgstat_report_activity(STATE_IDLEINTRANSACTION, NULL); + continue; + case 'w': + break; + default: + elog(ERROR, "[Apply BGW #%u] unexpected message \"%c\"", + pst->n, c); + break; + } + + /* Ignore statistics fields that have been updated. */ + s.cursor += IGNORE_SIZE_IN_MESSAGE; + + apply_dispatch(&s); + + if (ConfigReloadPending) + { + ConfigReloadPending = false; + ProcessConfigFile(PGC_SIGHUP); + } + + MemoryContextSwitchTo(oldctx); + MemoryContextReset(ApplyMessageContext); + } + + MemoryContextSwitchTo(TopMemoryContext); + MemoryContextReset(ApplyContext); + + /* Pop the error context stack */ + error_context_stack = errcallback.previous; + + elog(DEBUG1, "[Apply BGW #%u] exiting", pst->n); + + /* Signal main process that we are done. */ + SetLatch(®istrant->procLatch); +} + +/* + * Set the exit status so that the main apply worker can realize we have + * shutdown. + */ +static void +apply_bgworker_shutdown(int code, Datum arg) +{ + SpinLockAcquire(&MyParallelState->mutex); + MyParallelState->status = APPLY_BGWORKER_EXIT; + SpinLockRelease(&MyParallelState->mutex); + + dsm_detach((dsm_segment *) DatumGetPointer(arg)); +} + +/* + * Apply Background Worker entry point + */ +void +ApplyBgworkerMain(Datum main_arg) +{ + volatile ApplyBgworkerShared *pst; + + dsm_handle handle; + dsm_segment *seg; + shm_toc *toc; + shm_mq *mq; + shm_mq_handle *mqh; + MemoryContext oldcontext; + RepOriginId originid; + int worker_slot = DatumGetInt32(main_arg); + char originname[NAMEDATALEN]; + + MemoryContextSwitchTo(TopMemoryContext); + + /* Init the memory context for the apply background worker to work in. */ + ApplyContext = AllocSetContextCreate(TopMemoryContext, + "ApplyContext", + ALLOCSET_DEFAULT_SIZES); + + /* + * Init the ApplyMessageContext which we clean up after each replication + * protocol message. + */ + ApplyMessageContext = AllocSetContextCreate(ApplyContext, + "ApplyMessageContext", + ALLOCSET_DEFAULT_SIZES); + + /* Setup signal handling */ + pqsignal(SIGHUP, SignalHandlerForConfigReload); + pqsignal(SIGTERM, die); + BackgroundWorkerUnblockSignals(); + + /* + * Connect to the dynamic shared memory segment. + * + * The backend that registered this worker passed us the ID of a shared + * memory segment to which we must attach for further instructions. In + * order to attach to dynamic shared memory, we need a resource owner. + * Once we've mapped the segment in our address space, attach to the table + * of contents so we can locate the various data structures we'll need to + * find within the segment. + */ + CurrentResourceOwner = ResourceOwnerCreate(NULL, "Logical apply worker"); + memcpy(&handle, MyBgworkerEntry->bgw_extra, sizeof(dsm_handle)); + seg = dsm_attach(handle); + if (seg == NULL) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("unable to map dynamic shared memory segment"))); + toc = shm_toc_attach(PG_LOGICAL_APPLY_SHM_MAGIC, dsm_segment_address(seg)); + if (toc == NULL) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("bad magic number in dynamic shared memory segment"))); + + before_shmem_exit(apply_bgworker_shutdown, PointerGetDatum(seg)); + + /* Look up the parallel state. */ + pst = shm_toc_lookup(toc, APPLY_BGWORKER_KEY_SHARED, false); + MyParallelState = pst; + + /* + * Attach to the message queue. + */ + mq = shm_toc_lookup(toc, APPLY_BGWORKER_KEY_MQ, false); + shm_mq_set_receiver(mq, MyProc); + mqh = shm_mq_attach(mq, seg, NULL); + + /* Run as replica session replication role. */ + SetConfigOption("session_replication_role", "replica", + PGC_SUSET, PGC_S_OVERRIDE); + + /* + * Now, we have initialized DSM. Attach to slot. + */ + logicalrep_worker_attach(worker_slot); + + /* Connect to our database. */ + BackgroundWorkerInitializeConnectionByOid(MyLogicalRepWorker->dbid, + MyLogicalRepWorker->userid, + 0); + + /* + * Set always-secure search path, so malicious users can't redirect user + * code (e.g. pg_index.indexprs). + */ + SetConfigOption("search_path", "", PGC_SUSET, PGC_S_OVERRIDE); + + /* + * Set the client encoding to the database encoding, since that is what + * the leader will expect. + */ + SetClientEncoding(GetDatabaseEncoding()); + + stream_xid = pst->stream_xid; + + StartTransactionCommand(); + oldcontext = MemoryContextSwitchTo(ApplyContext); + + MySubscription = GetSubscription(MyLogicalRepWorker->subid, true); + if (!MySubscription) + { + ereport(LOG, + (errmsg("logical replication apply worker for subscription %u will not " + "start because the subscription was removed during startup", + MyLogicalRepWorker->subid))); + proc_exit(0); + } + + MySubscriptionValid = true; + MemoryContextSwitchTo(oldcontext); + + /* Setup synchronous commit according to the user's wishes */ + SetConfigOption("synchronous_commit", MySubscription->synccommit, + PGC_BACKEND, PGC_S_OVERRIDE); + + /* Keep us informed about subscription changes. */ + CacheRegisterSyscacheCallback(SUBSCRIPTIONOID, + subscription_change_cb, + (Datum) 0); + + CommitTransactionCommand(); + + /* Setup replication origin tracking. */ + StartTransactionCommand(); + snprintf(originname, sizeof(originname), "pg_%u", MySubscription->oid); + originid = replorigin_by_name(originname, true); + if (!OidIsValid(originid)) + originid = replorigin_create(originname); + + /* + * The apply background worker doesn't need to monopolize this replication + * origin which was already acquired by its leader process. + */ + replorigin_session_setup(originid, false); + replorigin_session_origin = originid; + CommitTransactionCommand(); + + /* + * Allocate the origin name in long-lived context for error context + * message. + */ + apply_error_callback_arg.origin_name = MemoryContextStrdup(ApplyContext, + originname); + + elog(DEBUG1, "[Apply BGW #%u] started", pst->n); + + LogicalApplyBgwLoop(mqh, pst); + + /* + * We're done. Explicitly detach the shared memory segment so that we + * don't get a resource leak warning at commit time. This will fire any + * on_dsm_detach callbacks we've registered, as well. Once that's done, + * we can go ahead and exit. + */ + dsm_detach(seg); + proc_exit(0); +} + +/* + * Set up a dynamic shared memory segment. + * + * We set up a control region that contains a ApplyBgworkerShared, + * plus one region per message queue. There are as many message queues as + * the number of workers. + */ +static void +apply_bgworker_setup_dsm(ApplyBgworkerState *wstate) +{ + shm_toc_estimator e; + Size segsize; + dsm_segment *seg; + shm_toc *toc; + ApplyBgworkerShared *pst; + shm_mq *mq; + int64 queue_size = DSM_QUEUE_SIZE; + + /* + * Estimate how much shared memory we need. + * + * Because the TOC machinery may choose to insert padding of oddly-sized + * requests, we must estimate each chunk separately. + * + * We need one key to register the location of the header, and we need + * another key to track the location of the message queue. + */ + shm_toc_initialize_estimator(&e); + shm_toc_estimate_chunk(&e, sizeof(ApplyBgworkerShared)); + shm_toc_estimate_chunk(&e, (Size) queue_size); + + shm_toc_estimate_keys(&e, 1 + 1); + segsize = shm_toc_estimate(&e); + + /* Create the shared memory segment and establish a table of contents. */ + seg = dsm_create(shm_toc_estimate(&e), 0); + toc = shm_toc_create(PG_LOGICAL_APPLY_SHM_MAGIC, dsm_segment_address(seg), + segsize); + + /* Set up the header region. */ + pst = shm_toc_allocate(toc, sizeof(ApplyBgworkerShared)); + SpinLockInit(&pst->mutex); + pst->status = APPLY_BGWORKER_BUSY; + pst->server_version = walrcv_server_version(LogRepWorkerWalRcvConn); + pst->stream_xid = stream_xid; + pst->n = list_length(ApplyWorkersList) + 1; + + shm_toc_insert(toc, APPLY_BGWORKER_KEY_SHARED, pst); + + /* Set up message queue for the worker. */ + mq = shm_mq_create(shm_toc_allocate(toc, (Size) queue_size), + (Size) queue_size); + shm_toc_insert(toc, APPLY_BGWORKER_KEY_MQ, mq); + shm_mq_set_sender(mq, MyProc); + + /* Attach the queue. */ + wstate->mq_handle = shm_mq_attach(mq, seg, NULL); + + /* Return results to caller. */ + wstate->dsm_seg = seg; + wstate->pstate = pst; +} + +/* + * Start apply background worker process and allocate shared memory for it. + */ +static ApplyBgworkerState * +apply_bgworker_setup(void) +{ + MemoryContext oldcontext; + bool launched; + ApplyBgworkerState *wstate; + int napplyworkers; + + elog(DEBUG1, "setting up apply worker #%u", list_length(ApplyWorkersList) + 1); + + /* Check If there are free worker slot(s) */ + LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); + napplyworkers = logicalrep_apply_background_worker_count(MyLogicalRepWorker->subid); + LWLockRelease(LogicalRepWorkerLock); + if (napplyworkers >= max_apply_bgworkers_per_subscription) + return NULL; + + oldcontext = MemoryContextSwitchTo(ApplyContext); + + wstate = (ApplyBgworkerState *) palloc0(sizeof(ApplyBgworkerState)); + + /* Setup shared memory */ + apply_bgworker_setup_dsm(wstate); + + launched = logicalrep_worker_launch(MyLogicalRepWorker->dbid, + MySubscription->oid, + MySubscription->name, + MyLogicalRepWorker->userid, + InvalidOid, + dsm_segment_handle(wstate->dsm_seg)); + + if (launched) + ApplyWorkersList = lappend(ApplyWorkersList, wstate); + else + { + shm_mq_detach(wstate->mq_handle); + dsm_detach(wstate->dsm_seg); + pfree(wstate); + + wstate->mq_handle = NULL; + wstate->dsm_seg = NULL; + wstate = NULL; + } + + MemoryContextSwitchTo(oldcontext); + + return wstate; +} + +/* + * Send the data to the specified apply background worker via shared-memory queue. + */ +void +apply_bgworker_send_data(ApplyBgworkerState *wstate, Size nbytes, const void *data) +{ + shm_mq_result result; + + result = shm_mq_send(wstate->mq_handle, nbytes, data, false, true); + + if (result != SHM_MQ_SUCCESS) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("could not send tuples to shared-memory queue"))); +} + +/* + * Wait until the status of apply background worker reaches the + * 'wait_for_status' + */ +void +apply_bgworker_wait_for(ApplyBgworkerState *wstate, + ApplyBgworkerStatus wait_for_status) +{ + for (;;) + { + char status; + + SpinLockAcquire(&wstate->pstate->mutex); + status = wstate->pstate->status; + SpinLockRelease(&wstate->pstate->mutex); + + /* Done if already in correct status. */ + if (status == wait_for_status) + break; + + /* If any workers (or the postmaster) have died, we have failed. */ + if (status == APPLY_BGWORKER_EXIT) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("background worker %u failed to apply transaction %u", + wstate->pstate->n, wstate->pstate->stream_xid))); + + /* Wait to be signalled. */ + WaitLatch(MyLatch, WL_LATCH_SET | WL_EXIT_ON_PM_DEATH, 0, + WAIT_EVENT_LOGICAL_APPLY_WORKER_STATE_CHANGE); + + /* Reset the latch so we don't spin. */ + ResetLatch(MyLatch); + + /* An interrupt may have occurred while we were waiting. */ + CHECK_FOR_INTERRUPTS(); + } +} + +/* + * Check the status of workers and report an error if any apply background + * worker has exited unexpectedly. + * + * Exit if any relation is not in the READY state and if any worker is handling + * the streaming transaction at the same time. Because for streaming + * transactions that is being applied in apply background worker, we cannot + * decide whether to apply the change for a relation that is not in the READY + * state (see should_apply_changes_for_rel) as we won't know remote_final_lsn + * by that time. + */ +void +apply_bgworker_check_status(void) +{ + ListCell *lc; + + if (am_apply_bgworker() || MySubscription->stream != SUBSTREAM_PARALLEL) + return; + + foreach(lc, ApplyWorkersList) + { + ApplyBgworkerState *wstate = (ApplyBgworkerState *) lfirst(lc); + + /* + * We don't lock here as in the worst case we will just detect the + * failure of worker a bit later. + */ + if (wstate->pstate->status == APPLY_BGWORKER_EXIT) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("background worker %u exited unexpectedly", + wstate->pstate->n))); + } + + if (list_length(ApplyWorkersFreeList) != list_length(ApplyWorkersList) && + !AllTablesyncsReady()) + { + ereport(LOG, + (errmsg("logical replication apply workers for subscription \"%s\" will restart", + MySubscription->name), + errdetail("Cannot handle streamed replication transaction by apply " + "background workers until all tables are synchronized"))); + + proc_exit(0); + } +} + +/* Set the apply background worker status */ +void +apply_bgworker_set_status(ApplyBgworkerStatus status) +{ + if (!am_apply_bgworker()) + return; + + elog(DEBUG1, "[Apply BGW #%u] set status to %d", MyParallelState->n, status); + + SpinLockAcquire(&MyParallelState->mutex); + MyParallelState->status = status; + SpinLockRelease(&MyParallelState->mutex); +} + +/* + * Define a savepoint for a subxact in apply background worker if needed. + * + * Inside apply background worker we can figure out that new subtransaction was + * started if new change arrived with different xid. In that case we can define + * named savepoint, so that we were able to commit/rollback it separately + * later. + * Special case is if the first change comes from subtransaction, then + * we check that current_xid differs from stream_xid. + */ +void +apply_bgworker_subxact_info_add(TransactionId current_xid) +{ + if (current_xid != stream_xid && + !list_member_int(subxactlist, (int) current_xid)) + { + MemoryContext oldctx; + char spname[MAXPGPATH]; + + snprintf(spname, MAXPGPATH, "savepoint_for_xid_%u", current_xid); + + elog(DEBUG1, "[Apply BGW #%u] defining savepoint %s", + MyParallelState->n, spname); + + DefineSavepoint(spname); + CommitTransactionCommand(); + + oldctx = MemoryContextSwitchTo(ApplyContext); + subxactlist = lappend_int(subxactlist, (int) current_xid); + MemoryContextSwitchTo(oldctx); + } +} diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c index aa2427ba73..00fdde0830 100644 --- a/src/backend/replication/logical/decode.c +++ b/src/backend/replication/logical/decode.c @@ -651,9 +651,10 @@ DecodeCommit(LogicalDecodingContext *ctx, XLogRecordBuffer *buf, { for (i = 0; i < parsed->nsubxacts; i++) { - ReorderBufferForget(ctx->reorder, parsed->subxacts[i], buf->origptr); + ReorderBufferForget(ctx->reorder, parsed->subxacts[i], buf->origptr, + commit_time); } - ReorderBufferForget(ctx->reorder, xid, buf->origptr); + ReorderBufferForget(ctx->reorder, xid, buf->origptr, commit_time); return; } @@ -821,10 +822,11 @@ DecodeAbort(LogicalDecodingContext *ctx, XLogRecordBuffer *buf, for (i = 0; i < parsed->nsubxacts; i++) { ReorderBufferAbort(ctx->reorder, parsed->subxacts[i], - buf->record->EndRecPtr); + buf->record->EndRecPtr, abort_time); } - ReorderBufferAbort(ctx->reorder, xid, buf->record->EndRecPtr); + ReorderBufferAbort(ctx->reorder, xid, buf->record->EndRecPtr, + abort_time); } /* update the decoding stats */ diff --git a/src/backend/replication/logical/launcher.c b/src/backend/replication/logical/launcher.c index 2bdab53e19..63b803ce94 100644 --- a/src/backend/replication/logical/launcher.c +++ b/src/backend/replication/logical/launcher.c @@ -54,6 +54,7 @@ int max_logical_replication_workers = 4; int max_sync_workers_per_subscription = 2; +int max_apply_bgworkers_per_subscription = 2; LogicalRepWorker *MyLogicalRepWorker = NULL; @@ -73,6 +74,7 @@ static void logicalrep_launcher_onexit(int code, Datum arg); static void logicalrep_worker_onexit(int code, Datum arg); static void logicalrep_worker_detach(void); static void logicalrep_worker_cleanup(LogicalRepWorker *worker); +static void logicalrep_worker_stop_internal(LogicalRepWorker *worker); static bool on_commit_launcher_wakeup = false; @@ -151,8 +153,10 @@ get_subscription_list(void) * * This is only needed for cleaning up the shared memory in case the worker * fails to attach. + * + * Returns false if the attach fails. Otherwise return true. */ -static void +static bool WaitForReplicationWorkerAttach(LogicalRepWorker *worker, uint16 generation, BackgroundWorkerHandle *handle) @@ -168,11 +172,11 @@ WaitForReplicationWorkerAttach(LogicalRepWorker *worker, LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); - /* Worker either died or has started; no need to do anything. */ + /* Worker either died or has started. Return false if died. */ if (!worker->in_use || worker->proc) { LWLockRelease(LogicalRepWorkerLock); - return; + return worker->in_use ? true : false; } LWLockRelease(LogicalRepWorkerLock); @@ -187,7 +191,7 @@ WaitForReplicationWorkerAttach(LogicalRepWorker *worker, if (generation == worker->generation) logicalrep_worker_cleanup(worker); LWLockRelease(LogicalRepWorkerLock); - return; + return false; } /* @@ -223,6 +227,13 @@ logicalrep_worker_find(Oid subid, Oid relid, bool only_running) { LogicalRepWorker *w = &LogicalRepCtx->workers[i]; + /* + * We are only interested in the main apply worker or table sync worker + * here. + */ + if (w->subworker) + continue; + if (w->in_use && w->subid == subid && w->relid == relid && (!only_running || w->proc)) { @@ -259,11 +270,11 @@ logicalrep_workers_find(Oid subid, bool only_running) } /* - * Start new apply background worker, if possible. + * Start new background worker, if possible. */ -void +bool logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, Oid userid, - Oid relid) + Oid relid, dsm_handle subworker_dsm) { BackgroundWorker bgw; BackgroundWorkerHandle *bgw_handle; @@ -273,6 +284,10 @@ logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, Oid userid, LogicalRepWorker *worker = NULL; int nsyncworkers; TimestampTz now; + bool is_subworker = (subworker_dsm != DSM_HANDLE_INVALID); + + /* We don't support table sync in subworker */ + Assert(!(is_subworker && OidIsValid(relid))); ereport(DEBUG1, (errmsg_internal("starting logical replication worker for subscription \"%s\"", @@ -350,7 +365,7 @@ retry: if (OidIsValid(relid) && nsyncworkers >= max_sync_workers_per_subscription) { LWLockRelease(LogicalRepWorkerLock); - return; + return false; } /* @@ -364,7 +379,7 @@ retry: (errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED), errmsg("out of logical replication worker slots"), errhint("You might need to increase max_logical_replication_workers."))); - return; + return false; } /* Prepare the worker slot. */ @@ -379,6 +394,7 @@ retry: worker->relstate = SUBREL_STATE_UNKNOWN; worker->relstate_lsn = InvalidXLogRecPtr; worker->stream_fileset = NULL; + worker->subworker = is_subworker; worker->last_lsn = InvalidXLogRecPtr; TIMESTAMP_NOBEGIN(worker->last_send_time); TIMESTAMP_NOBEGIN(worker->last_recv_time); @@ -396,19 +412,31 @@ retry: BGWORKER_BACKEND_DATABASE_CONNECTION; bgw.bgw_start_time = BgWorkerStart_RecoveryFinished; snprintf(bgw.bgw_library_name, BGW_MAXLEN, "postgres"); - snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyWorkerMain"); + + if (is_subworker) + snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyBgworkerMain"); + else + snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyWorkerMain"); + if (OidIsValid(relid)) snprintf(bgw.bgw_name, BGW_MAXLEN, "logical replication worker for subscription %u sync %u", subid, relid); + else if (is_subworker) + snprintf(bgw.bgw_name, BGW_MAXLEN, + "logical replication apply background worker for subscription %u", subid); else snprintf(bgw.bgw_name, BGW_MAXLEN, "logical replication worker for subscription %u", subid); + snprintf(bgw.bgw_type, BGW_MAXLEN, "logical replication worker"); bgw.bgw_restart_time = BGW_NEVER_RESTART; bgw.bgw_notify_pid = MyProcPid; bgw.bgw_main_arg = Int32GetDatum(slot); + if (is_subworker) + memcpy(bgw.bgw_extra, &subworker_dsm, sizeof(dsm_handle)); + if (!RegisterDynamicBackgroundWorker(&bgw, &bgw_handle)) { /* Failed to start worker, so clean up the worker slot. */ @@ -421,11 +449,11 @@ retry: (errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED), errmsg("out of background worker slots"), errhint("You might need to increase max_worker_processes."))); - return; + return false; } /* Now wait until it attaches. */ - WaitForReplicationWorkerAttach(worker, generation, bgw_handle); + return WaitForReplicationWorkerAttach(worker, generation, bgw_handle); } /* @@ -436,18 +464,27 @@ void logicalrep_worker_stop(Oid subid, Oid relid) { LogicalRepWorker *worker; - uint16 generation; LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); worker = logicalrep_worker_find(subid, relid, false); - /* No worker, nothing to do. */ - if (!worker) - { - LWLockRelease(LogicalRepWorkerLock); - return; - } + if (worker) + logicalrep_worker_stop_internal(worker); + + LWLockRelease(LogicalRepWorkerLock); +} + +/* + * Workhorse for logicalrep_worker_stop() and logicalrep_worker_detach(). Stop + * the worker and wait for it to die. + */ +static void +logicalrep_worker_stop_internal(LogicalRepWorker *worker) +{ + uint16 generation; + + Assert(LWLockHeldByMe(LogicalRepWorkerLock)); /* * Remember which generation was our worker so we can check if what we see @@ -485,10 +522,7 @@ logicalrep_worker_stop(Oid subid, Oid relid) * different, meaning that a different worker has taken the slot. */ if (!worker->in_use || worker->generation != generation) - { - LWLockRelease(LogicalRepWorkerLock); return; - } /* Worker has assigned proc, so it has started. */ if (worker->proc) @@ -522,8 +556,6 @@ logicalrep_worker_stop(Oid subid, Oid relid) LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); } - - LWLockRelease(LogicalRepWorkerLock); } /* @@ -599,6 +631,29 @@ logicalrep_worker_attach(int slot) static void logicalrep_worker_detach(void) { + /* + * If we are the main apply worker, stop all the apply background workers + * we started before. + */ + if (!MyLogicalRepWorker->subworker) + { + List *workers; + ListCell *lc; + + LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); + + workers = logicalrep_workers_find(MyLogicalRepWorker->subid, true); + foreach(lc, workers) + { + LogicalRepWorker *w = (LogicalRepWorker *) lfirst(lc); + + if (w->subworker) + logicalrep_worker_stop_internal(w); + } + + LWLockRelease(LogicalRepWorkerLock); + } + /* Block concurrent access. */ LWLockAcquire(LogicalRepWorkerLock, LW_EXCLUSIVE); @@ -621,6 +676,7 @@ logicalrep_worker_cleanup(LogicalRepWorker *worker) worker->userid = InvalidOid; worker->subid = InvalidOid; worker->relid = InvalidOid; + worker->subworker = false; } /* @@ -679,6 +735,30 @@ logicalrep_sync_worker_count(Oid subid) return res; } +/* + * Count the number of registered (not necessarily running) apply background + * workers for a subscription. + */ +int +logicalrep_apply_background_worker_count(Oid subid) +{ + int i; + int res = 0; + + Assert(LWLockHeldByMe(LogicalRepWorkerLock)); + + /* Search for attached worker for a given subscription id. */ + for (i = 0; i < max_logical_replication_workers; i++) + { + LogicalRepWorker *w = &LogicalRepCtx->workers[i]; + + if (w->subid == subid && w->subworker) + res++; + } + + return res; +} + /* * ApplyLauncherShmemSize * Compute space needed for replication launcher shared memory @@ -868,7 +948,7 @@ ApplyLauncherMain(Datum main_arg) wait_time = wal_retrieve_retry_interval; logicalrep_worker_launch(sub->dbid, sub->oid, sub->name, - sub->owner, InvalidOid); + sub->owner, InvalidOid, DSM_HANDLE_INVALID); } } diff --git a/src/backend/replication/logical/origin.c b/src/backend/replication/logical/origin.c index 21937ab2d3..50c567fb6e 100644 --- a/src/backend/replication/logical/origin.c +++ b/src/backend/replication/logical/origin.c @@ -1063,12 +1063,21 @@ ReplicationOriginExitCleanup(int code, Datum arg) * array doesn't have to be searched when calling * replorigin_session_advance(). * - * Obviously only one such cached origin can exist per process and the current + * Normally only one such cached origin can exist per process and the current * cached value can only be set again after the previous value is torn down * with replorigin_session_reset(). + * + * However, if the function parameter 'must_acquire' is false, we allow the + * process to use the same slot already acquired by another process. It's safe + * because 1) The only caller (apply background workers) will maintain the + * commit order by allowing only one process to commit at a time, so no two + * workers will be operating on the same origin at the same time (see comments + * in logical/worker.c). 2) Even though we try to advance the session's origin + * concurrently, it's safe to do so as we change/advance the session_origin + * LSNs under replicate_state LWLock. */ void -replorigin_session_setup(RepOriginId node) +replorigin_session_setup(RepOriginId node, bool must_acquire) { static bool registered_cleanup; int i; @@ -1110,7 +1119,7 @@ replorigin_session_setup(RepOriginId node) if (curstate->roident != node) continue; - else if (curstate->acquired_by != 0) + else if (curstate->acquired_by != 0 && must_acquire) { ereport(ERROR, (errcode(ERRCODE_OBJECT_IN_USE), @@ -1141,7 +1150,14 @@ replorigin_session_setup(RepOriginId node) Assert(session_replication_state->roident != InvalidRepOriginId); - session_replication_state->acquired_by = MyProcPid; + if (must_acquire) + session_replication_state->acquired_by = MyProcPid; + else if (session_replication_state->acquired_by == 0) + ereport(ERROR, + (errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED), + errmsg("apply background worker could not find replication state slot for replication origin with OID %u", + node), + errdetail("There is no replication state slot set by its main apply worker."))); LWLockRelease(ReplicationOriginLock); @@ -1321,7 +1337,7 @@ pg_replication_origin_session_setup(PG_FUNCTION_ARGS) name = text_to_cstring((text *) DatumGetPointer(PG_GETARG_DATUM(0))); origin = replorigin_by_name(name, false); - replorigin_session_setup(origin); + replorigin_session_setup(origin, true); replorigin_session_origin = origin; diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c index ff8513e2d2..b3718f0829 100644 --- a/src/backend/replication/logical/proto.c +++ b/src/backend/replication/logical/proto.c @@ -1166,28 +1166,47 @@ logicalrep_read_stream_commit(StringInfo in, LogicalRepCommitData *commit_data) */ void logicalrep_write_stream_abort(StringInfo out, TransactionId xid, - TransactionId subxid) + ReorderBufferTXN *txn, XLogRecPtr abort_lsn) { pq_sendbyte(out, LOGICAL_REP_MSG_STREAM_ABORT); - Assert(TransactionIdIsValid(xid) && TransactionIdIsValid(subxid)); + Assert(TransactionIdIsValid(xid) && TransactionIdIsValid(txn->xid)); /* transaction ID */ pq_sendint32(out, xid); - pq_sendint32(out, subxid); + pq_sendint32(out, txn->xid); + pq_sendint64(out, abort_lsn); + pq_sendint64(out, txn->xact_time.abort_time); } /* * Read STREAM ABORT from the output stream. */ void -logicalrep_read_stream_abort(StringInfo in, TransactionId *xid, - TransactionId *subxid) +logicalrep_read_stream_abort(StringInfo in, + LogicalRepStreamAbortData *abort_data, + bool include_abort_lsn) { - Assert(xid && subxid); + Assert(abort_data); + + abort_data->xid = pq_getmsgint(in, 4); + abort_data->subxid = pq_getmsgint(in, 4); - *xid = pq_getmsgint(in, 4); - *subxid = pq_getmsgint(in, 4); + /* + * If the version of the publisher is lower than the version of the + * subscriber, it may not support sending these two fields. So these + * fields are only taken if they are included. + */ + if (include_abort_lsn) + { + abort_data->abort_lsn = pq_getmsgint64(in); + abort_data->abort_time = pq_getmsgint64(in); + } + else + { + abort_data->abort_lsn = InvalidXLogRecPtr; + abort_data->abort_time = 0; + } } /* diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c index 8da5f9089c..3d2bbdb38d 100644 --- a/src/backend/replication/logical/reorderbuffer.c +++ b/src/backend/replication/logical/reorderbuffer.c @@ -2826,7 +2826,8 @@ ReorderBufferFinishPrepared(ReorderBuffer *rb, TransactionId xid, * disk. */ void -ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn) +ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn, + TimestampTz abort_time) { ReorderBufferTXN *txn; @@ -2837,6 +2838,8 @@ ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn) if (txn == NULL) return; + txn->xact_time.abort_time = abort_time; + /* For streamed transactions notify the remote node about the abort. */ if (rbtxn_is_streamed(txn)) { @@ -2911,7 +2914,8 @@ ReorderBufferAbortOld(ReorderBuffer *rb, TransactionId oldestRunningXid) * to this xid might re-create the transaction incompletely. */ void -ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn) +ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn, + TimestampTz abort_time) { ReorderBufferTXN *txn; @@ -2922,6 +2926,8 @@ ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn) if (txn == NULL) return; + txn->xact_time.abort_time = abort_time; + /* For streamed transactions notify the remote node about the abort. */ if (rbtxn_is_streamed(txn)) rb->stream_abort(rb, txn, lsn); diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c index 670c6fcada..8ffba7e2e5 100644 --- a/src/backend/replication/logical/tablesync.c +++ b/src/backend/replication/logical/tablesync.c @@ -568,7 +568,8 @@ process_syncing_tables_for_apply(XLogRecPtr current_lsn) MySubscription->oid, MySubscription->name, MyLogicalRepWorker->userid, - rstate->relid); + rstate->relid, + DSM_HANDLE_INVALID); hentry->last_start_time = now; } } @@ -589,6 +590,9 @@ process_syncing_tables_for_apply(XLogRecPtr current_lsn) void process_syncing_tables(XLogRecPtr current_lsn) { + if (MyLogicalRepWorker->subworker) + return; + if (am_tablesync_worker()) process_syncing_tables_for_sync(current_lsn); else @@ -1273,7 +1277,7 @@ LogicalRepSyncTableStart(XLogRecPtr *origin_startpos) * time this tablesync was launched. */ originid = replorigin_by_name(originname, false); - replorigin_session_setup(originid); + replorigin_session_setup(originid, true); replorigin_session_origin = originid; *origin_startpos = replorigin_session_get_progress(false); @@ -1384,7 +1388,7 @@ LogicalRepSyncTableStart(XLogRecPtr *origin_startpos) true /* go backward */ , true /* WAL log */ ); UnlockRelationOid(ReplicationOriginRelationId, RowExclusiveLock); - replorigin_session_setup(originid); + replorigin_session_setup(originid, true); replorigin_session_origin = originid; } else diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c index 38e3b1c1b3..84938a87be 100644 --- a/src/backend/replication/logical/worker.c +++ b/src/backend/replication/logical/worker.c @@ -22,8 +22,28 @@ * STREAMED TRANSACTIONS * --------------------- * Streamed transactions (large transactions exceeding a memory limit on the - * upstream) are not applied immediately, but instead, the data is written - * to temporary files and then applied at once when the final commit arrives. + * upstream) are applied using one of two approaches. + * + * 1) Separate background workers + * + * Assign a new apply background worker (if available) as soon as the xact's + * first stream is received. The main apply worker will send changes to this + * new worker via shared memory. We keep this worker assigned till the + * transaction commit is received and also wait for the worker to finish at + * commit. This preserves commit ordering and avoids file I/O in most cases. We + * still need to spill to a file if there is no worker available. It is + * important to maintain commit order to avoid failures due to (a) transaction + * dependencies, say if we insert a row in the first transaction and update it + * in the second transaction then allowing to apply both in parallel can lead + * to failure in the update. (b) deadlocks, allowing transactions that update + * the same set of rows/tables in opposite order to be applied in parallel can + * lead to deadlocks. + * + * 2) Write to temporary files and apply when the final commit arrives + * + * If no worker is available to handle streamed transaction, the data is + * written to temporary files and then applied at once when the final commit + * arrives. * * Unlike the regular (non-streamed) case, handling streamed transactions has * to handle aborts of both the toplevel transaction and subtransactions. This @@ -219,20 +239,7 @@ typedef struct ApplyExecutionData PartitionTupleRouting *proute; /* partition routing info */ } ApplyExecutionData; -/* Struct for saving and restoring apply errcontext information */ -typedef struct ApplyErrorCallbackArg -{ - LogicalRepMsgType command; /* 0 if invalid */ - LogicalRepRelMapEntry *rel; - - /* Remote node information */ - int remote_attnum; /* -1 if invalid */ - TransactionId remote_xid; - XLogRecPtr finish_lsn; - char *origin_name; -} ApplyErrorCallbackArg; - -static ApplyErrorCallbackArg apply_error_callback_arg = +ApplyErrorCallbackArg apply_error_callback_arg = { .command = 0, .rel = NULL, @@ -242,7 +249,7 @@ static ApplyErrorCallbackArg apply_error_callback_arg = .origin_name = NULL, }; -static MemoryContext ApplyMessageContext = NULL; +MemoryContext ApplyMessageContext = NULL; MemoryContext ApplyContext = NULL; /* per stream context for streaming transactions */ @@ -251,27 +258,38 @@ static MemoryContext LogicalStreamingContext = NULL; WalReceiverConn *LogRepWorkerWalRcvConn = NULL; Subscription *MySubscription = NULL; -static bool MySubscriptionValid = false; +bool MySubscriptionValid = false; bool in_remote_transaction = false; static XLogRecPtr remote_final_lsn = InvalidXLogRecPtr; /* fields valid only when processing streamed transaction */ -static bool in_streamed_transaction = false; +bool in_streamed_transaction = false; + +TransactionId stream_xid = InvalidTransactionId; +static ApplyBgworkerState *stream_apply_worker = NULL; -static TransactionId stream_xid = InvalidTransactionId; +/* Check if we are applying the transaction in apply background worker */ +#define apply_bgworker_active() (in_streamed_transaction && stream_apply_worker != NULL) + +/* + * The number of changes during one streaming block (only for apply background + * workers) + */ +static uint32 nchanges = 0; /* * We enable skipping all data modification changes (INSERT, UPDATE, etc.) for * the subscription if the remote transaction's finish LSN matches the subskiplsn. * Once we start skipping changes, we don't stop it until we skip all changes of * the transaction even if pg_subscription is updated and MySubscription->skiplsn - * gets changed or reset during that. Also, in streaming transaction cases, we - * don't skip receiving and spooling the changes since we decide whether or not + * gets changed or reset during that. Also, in streaming transaction cases (streaming = on), + * we don't skip receiving and spooling the changes since we decide whether or not * to skip applying the changes when starting to apply changes. The subskiplsn is * cleared after successfully skipping the transaction or applying non-empty * transaction. The latter prevents the mistakenly specified subskiplsn from - * being left. + * being left. Note that we cannot skip the streaming transaction in parallel + * mode, because we cannot get the finish LSN before applying the changes. */ static XLogRecPtr skip_xact_finish_lsn = InvalidXLogRecPtr; #define is_skipping_changes() (unlikely(!XLogRecPtrIsInvalid(skip_xact_finish_lsn))) @@ -324,9 +342,6 @@ static void maybe_reread_subscription(void); static void DisableSubscriptionAndExit(void); -/* prototype needed because of stream_commit */ -static void apply_dispatch(StringInfo s); - static void apply_handle_commit_internal(LogicalRepCommitData *commit_data); static void apply_handle_insert_internal(ApplyExecutionData *edata, ResultRelInfo *relinfo, @@ -359,7 +374,6 @@ static void stop_skipping_changes(void); static void clear_subscription_skip_lsn(XLogRecPtr finish_lsn); /* Functions for apply error callback */ -static void apply_error_callback(void *arg); static inline void set_apply_error_context_xact(TransactionId xid, XLogRecPtr lsn); static inline void reset_apply_error_context_info(void); @@ -426,38 +440,84 @@ end_replication_step(void) } /* - * Handle streamed transactions. + * Handle streamed transactions for both main apply worker and apply background + * worker. + * + * In streaming case (receiving a block of streamed transaction), for + * SUBSTREAM_ON mode, we simply redirect it to a file for the proper toplevel + * transaction, and for SUBSTREAM_PARALLEL mode, we send the changes to + * background apply worker (LOGICAL_REP_MSG_RELATION or LOGICAL_REP_MSG_TYPE + * changes will also be applied in main apply worker). * - * If in streaming mode (receiving a block of streamed transaction), we - * simply redirect it to a file for the proper toplevel transaction. + * For non-streamed transactions, returns false; + * For streamed transactions, returns true if in main apply worker, false + * otherwise. * - * Returns true for streamed transactions, false otherwise (regular mode). + * But there are two exceptions: If we apply streamed transaction in main apply + * worker with parallel mode, it will return false when we address + * LOGICAL_REP_MSG_RELATION or LOGICAL_REP_MSG_TYPE changes. */ static bool handle_streamed_transaction(LogicalRepMsgType action, StringInfo s) { - TransactionId xid; + TransactionId current_xid = InvalidTransactionId; /* not in streaming mode */ - if (!in_streamed_transaction) + if (!(in_streamed_transaction || am_apply_bgworker())) return false; - Assert(stream_fd != NULL); Assert(TransactionIdIsValid(stream_xid)); /* * We should have received XID of the subxact as the first part of the * message, so extract it. */ - xid = pq_getmsgint(s, 4); + current_xid = pq_getmsgint(s, 4); - if (!TransactionIdIsValid(xid)) + if (!TransactionIdIsValid(current_xid)) ereport(ERROR, (errcode(ERRCODE_PROTOCOL_VIOLATION), errmsg_internal("invalid transaction ID in streamed replication transaction"))); - /* Add the new subxact to the array (unless already there). */ - subxact_info_add(xid); + if (am_apply_bgworker()) + { + /* define a savepoint for a subxact if needed. */ + apply_bgworker_subxact_info_add(current_xid); + + return false; + } + + if (apply_bgworker_active()) + { + /* + * This is the main apply worker, and there is an apply background + * worker. So we apply the changes of this transaction in an apply + * background worker. Pass the data to the worker. + */ + apply_bgworker_send_data(stream_apply_worker, s->len, s->data); + nchanges += 1; + + /* + * XXX The publisher side doesn't always send relation/type update + * messages after the streaming transaction, so also update the + * relation/type in main apply worker here. See function + * cleanup_rel_sync_cache. + */ + if (action == LOGICAL_REP_MSG_RELATION || + action == LOGICAL_REP_MSG_TYPE) + return false; + + return true; + } + + /* + * This is the main apply worker, but there is no apply background + * worker. So we write to temporary files and apply when the final + * commit arrives. + * + * Add the new subxact to the array (unless already there). + */ + subxact_info_add(current_xid); /* write the change to the current file */ stream_write_change(action, s); @@ -844,6 +904,9 @@ apply_handle_commit(StringInfo s) apply_handle_commit_internal(&commit_data); + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); + /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(commit_data.end_lsn); @@ -898,7 +961,9 @@ apply_handle_prepare_internal(LogicalRepPreparedTxnData *prepare_data) * BeginTransactionBlock is necessary to balance the EndTransactionBlock * called within the PrepareTransactionBlock below. */ - BeginTransactionBlock(); + if (!IsTransactionBlock()) + BeginTransactionBlock(); + CommitTransactionCommand(); /* Completes the preceding Begin command. */ /* @@ -950,6 +1015,9 @@ apply_handle_prepare(StringInfo s) in_remote_transaction = false; + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); + /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(prepare_data.end_lsn); @@ -1064,10 +1132,6 @@ apply_handle_rollback_prepared(StringInfo s) /* * Handle STREAM PREPARE. - * - * Logic is in two parts: - * 1. Replay all the spooled operations - * 2. Mark the transaction as prepared */ static void apply_handle_stream_prepare(StringInfo s) @@ -1088,24 +1152,76 @@ apply_handle_stream_prepare(StringInfo s) logicalrep_read_stream_prepare(s, &prepare_data); set_apply_error_context_xact(prepare_data.xid, prepare_data.prepare_lsn); - elog(DEBUG1, "received prepare for streamed transaction %u", prepare_data.xid); + if (am_apply_bgworker()) + { + /* Mark the transaction as prepared. */ + apply_handle_prepare_internal(&prepare_data); - /* Replay all the spooled operations. */ - apply_spooled_messages(prepare_data.xid, prepare_data.prepare_lsn); + CommitTransactionCommand(); - /* Mark the transaction as prepared. */ - apply_handle_prepare_internal(&prepare_data); + pgstat_report_stat(false); - CommitTransactionCommand(); + list_free(subxactlist); + subxactlist = NIL; - pgstat_report_stat(false); + apply_bgworker_set_status(APPLY_BGWORKER_FINISHED); + } + else + { + /* This is the main apply worker. */ + ApplyBgworkerState *wstate = apply_bgworker_find(prepare_data.xid); - store_flush_position(prepare_data.end_lsn); + elog(DEBUG1, "received prepare for streamed transaction %u", + prepare_data.xid); + + /* + * Check if we are processing this transaction in an apply background + * worker. + */ + if (wstate) + { + apply_bgworker_send_data(wstate, s->len, s->data); + + /* + * Wait for apply background worker to finish. This is required to + * maintain commit order which avoids failures due to transaction + * dependencies and deadlocks. + */ + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_FINISHED); + apply_bgworker_free(wstate); + + pgstat_report_stat(false); + store_flush_position(prepare_data.end_lsn); + } + else + { + /* + * The transaction has been serialized to file, so replay all the + * spooled operations. + */ + apply_spooled_messages(prepare_data.xid, prepare_data.prepare_lsn); + + /* Mark the transaction as prepared. */ + apply_handle_prepare_internal(&prepare_data); + + CommitTransactionCommand(); + + pgstat_report_stat(false); + + store_flush_position(prepare_data.end_lsn); + + in_remote_transaction = false; + + /* Unlink the files with serialized changes and subxact info. */ + stream_cleanup_files(MyLogicalRepWorker->subid, prepare_data.xid); + } + } in_remote_transaction = false; + stream_apply_worker = NULL; - /* unlink the files with serialized changes and subxact info. */ - stream_cleanup_files(MyLogicalRepWorker->subid, prepare_data.xid); + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(prepare_data.end_lsn); @@ -1155,15 +1271,6 @@ apply_handle_stream_start(StringInfo s) (errcode(ERRCODE_PROTOCOL_VIOLATION), errmsg_internal("duplicate STREAM START message"))); - /* - * Start a transaction on stream start, this transaction will be committed - * on the stream stop unless it is a tablesync worker in which case it - * will be committed after processing all the messages. We need the - * transaction for handling the buffile, used for serializing the - * streaming data and subxact info. - */ - begin_replication_step(); - /* notify handle methods we're processing a remote transaction */ in_streamed_transaction = true; @@ -1177,36 +1284,95 @@ apply_handle_stream_start(StringInfo s) set_apply_error_context_xact(stream_xid, InvalidXLogRecPtr); - /* - * Initialize the worker's stream_fileset if we haven't yet. This will be - * used for the entire duration of the worker so create it in a permanent - * context. We create this on the very first streaming message from any - * transaction and then use it for this and other streaming transactions. - * Now, we could create a fileset at the start of the worker as well but - * then we won't be sure that it will ever be used. - */ - if (MyLogicalRepWorker->stream_fileset == NULL) + if (am_apply_bgworker()) { - MemoryContext oldctx; - - oldctx = MemoryContextSwitchTo(ApplyContext); + /* + * Make sure the handle apply_dispatch methods are aware we're in a + * remote transaction. + */ + in_remote_transaction = true; - MyLogicalRepWorker->stream_fileset = palloc(sizeof(FileSet)); - FileSetInit(MyLogicalRepWorker->stream_fileset); + /* Begin the transaction. */ + AcceptInvalidationMessages(); + maybe_reread_subscription(); - MemoryContextSwitchTo(oldctx); + StartTransactionCommand(); + BeginTransactionBlock(); + CommitTransactionCommand(); } + else + { + /* + * This is the main apply worker. Check if there is any free apply + * background worker we can use to process this transaction. + */ + if (first_segment) + stream_apply_worker = apply_bgworker_start(stream_xid); + else + stream_apply_worker = apply_bgworker_find(stream_xid); - /* open the spool file for this transaction */ - stream_open_file(MyLogicalRepWorker->subid, stream_xid, first_segment); + if (stream_apply_worker) + { + /* + * If we have found a free worker or if we are already applying this + * transaction in an apply background worker, then we pass the data to + * that worker. + */ + if (first_segment) + apply_bgworker_send_data(stream_apply_worker, s->len, s->data); - /* if this is not the first segment, open existing subxact file */ - if (!first_segment) - subxact_info_read(MyLogicalRepWorker->subid, stream_xid); + nchanges = 0; + elog(DEBUG1, "starting streaming of xid %u", stream_xid); + } - pgstat_report_activity(STATE_RUNNING, NULL); + /* + * If no worker is available for the first stream start, we start to + * serialize all the changes of the transaction. + */ + else + { + /* + * Start a transaction on stream start, this transaction will be + * committed on the stream stop unless it is a tablesync worker in + * which case it will be committed after processing all the messages. + * We need the transaction for handling the buffile, used for + * serializing the streaming data and subxact info. + */ + begin_replication_step(); - end_replication_step(); + /* + * Initialize the worker's stream_fileset if we haven't yet. This will + * be used for the entire duration of the worker so create it in a + * permanent context. We create this on the very first streaming + * message from any transaction and then use it for this and other + * streaming transactions. Now, we could create a fileset at the start + * of the worker as well but then we won't be sure that it will ever + * be used. + */ + if (MyLogicalRepWorker->stream_fileset == NULL) + { + MemoryContext oldctx; + + oldctx = MemoryContextSwitchTo(ApplyContext); + + MyLogicalRepWorker->stream_fileset = palloc(sizeof(FileSet)); + FileSetInit(MyLogicalRepWorker->stream_fileset); + + MemoryContextSwitchTo(oldctx); + } + + /* Open the spool file for this transaction */ + stream_open_file(MyLogicalRepWorker->subid, stream_xid, first_segment); + + /* if this is not the first segment, open existing subxact file */ + if (!first_segment) + subxact_info_read(MyLogicalRepWorker->subid, stream_xid); + + end_replication_step(); + } + } + + pgstat_report_activity(STATE_RUNNING, NULL); } /* @@ -1220,53 +1386,52 @@ apply_handle_stream_stop(StringInfo s) (errcode(ERRCODE_PROTOCOL_VIOLATION), errmsg_internal("STREAM STOP message without STREAM START"))); - /* - * Close the file with serialized changes, and serialize information about - * subxacts for the toplevel transaction. - */ - subxact_info_write(MyLogicalRepWorker->subid, stream_xid); - stream_close_file(); + if (apply_bgworker_active()) + { + char action = LOGICAL_REP_MSG_STREAM_STOP; - /* We must be in a valid transaction state */ - Assert(IsTransactionState()); + apply_bgworker_send_data(stream_apply_worker, 1, &action); - /* Commit the per-stream transaction */ - CommitTransactionCommand(); + elog(DEBUG1, "stopped streaming of xid %u, %u changes streamed", stream_xid, nchanges); + } + else + { + /* + * Close the file with serialized changes, and serialize information + * about subxacts for the toplevel transaction. + */ + subxact_info_write(MyLogicalRepWorker->subid, stream_xid); + stream_close_file(); - in_streamed_transaction = false; + /* We must be in a valid transaction state */ + Assert(IsTransactionState()); + + /* Commit the per-stream transaction */ + CommitTransactionCommand(); - /* Reset per-stream context */ - MemoryContextReset(LogicalStreamingContext); + /* Reset per-stream context */ + MemoryContextReset(LogicalStreamingContext); + } + + in_streamed_transaction = false; + stream_apply_worker = NULL; pgstat_report_activity(STATE_IDLE, NULL); reset_apply_error_context_info(); } /* - * Handle STREAM abort message. + * Handle STREAM ABORT message when the transaction was spilled to disk. */ static void -apply_handle_stream_abort(StringInfo s) +serialize_stream_abort(TransactionId xid, TransactionId subxid) { - TransactionId xid; - TransactionId subxid; - - if (in_streamed_transaction) - ereport(ERROR, - (errcode(ERRCODE_PROTOCOL_VIOLATION), - errmsg_internal("STREAM ABORT message without STREAM STOP"))); - - logicalrep_read_stream_abort(s, &xid, &subxid); - /* * If the two XIDs are the same, it's in fact abort of toplevel xact, so * just delete the files with serialized info. */ if (xid == subxid) - { - set_apply_error_context_xact(xid, InvalidXLogRecPtr); stream_cleanup_files(MyLogicalRepWorker->subid, xid); - } else { /* @@ -1290,8 +1455,6 @@ apply_handle_stream_abort(StringInfo s) bool found = false; char path[MAXPGPATH]; - set_apply_error_context_xact(subxid, InvalidXLogRecPtr); - subidx = -1; begin_replication_step(); subxact_info_read(MyLogicalRepWorker->subid, xid); @@ -1316,7 +1479,6 @@ apply_handle_stream_abort(StringInfo s) cleanup_subxact_info(); end_replication_step(); CommitTransactionCommand(); - reset_apply_error_context_info(); return; } @@ -1339,6 +1501,139 @@ apply_handle_stream_abort(StringInfo s) end_replication_step(); CommitTransactionCommand(); } +} + +/* + * Handle STREAM ABORT message. + */ +static void +apply_handle_stream_abort(StringInfo s) +{ + TransactionId xid; + TransactionId subxid; + LogicalRepStreamAbortData abort_data; + bool include_abort_lsn = false; + + if (in_streamed_transaction) + ereport(ERROR, + (errcode(ERRCODE_PROTOCOL_VIOLATION), + errmsg_internal("STREAM ABORT message without STREAM STOP"))); + + /* Check whether the publisher sends abort_lsn and abort_time. */ + if (am_apply_bgworker()) + include_abort_lsn = MyParallelState->server_version >= 150000; + + logicalrep_read_stream_abort(s, &abort_data, include_abort_lsn); + + xid = abort_data.xid; + subxid = abort_data.subxid; + + set_apply_error_context_xact(subxid, abort_data.abort_lsn); + + if (am_apply_bgworker()) + { + elog(DEBUG1, "[Apply BGW #%u] aborting current transaction xid=%u, subxid=%u", + MyParallelState->n, GetCurrentTransactionIdIfAny(), + GetCurrentSubTransactionId()); + + /* + * Update origin state so we can restart streaming from correct + * position in case of crash. + */ + if (include_abort_lsn) + { + replorigin_session_origin_lsn = abort_data.abort_lsn; + replorigin_session_origin_timestamp = abort_data.abort_time; + } + + /* + * If the two XIDs are the same, it's in fact abort of toplevel xact, + * so just free the subxactlist. + */ + if (subxid == xid) + { + AbortCurrentTransaction(); + + EndTransactionBlock(false); + CommitTransactionCommand(); + + in_remote_transaction = false; + pgstat_report_activity(STATE_IDLE, NULL); + + list_free(subxactlist); + subxactlist = NIL; + + apply_bgworker_set_status(APPLY_BGWORKER_FINISHED); + } + else + { + /* + * OK, so it's a subxact. Rollback to the savepoint. + * + * We also need to read the subxactlist, determine the offset + * tracked for the subxact, and truncate the list. + */ + int i; + bool found = false; + char spname[MAXPGPATH]; + + snprintf(spname, MAXPGPATH, "savepoint_for_xid_%u", subxid); + + elog(DEBUG1, "[Apply BGW #%u] rolling back to savepoint %s", + MyParallelState->n, spname); + + for (i = list_length(subxactlist) - 1; i >= 0; i--) + { + xid = (TransactionId) list_nth_int(subxactlist, i); + if (xid == subxid) + { + found = true; + break; + } + } + + if (found) + { + RollbackToSavepoint(spname); + CommitTransactionCommand(); + subxactlist = list_truncate(subxactlist, i + 1); + } + + pgstat_report_activity(STATE_IDLEINTRANSACTION, NULL); + } + } + else + { + /* This is the main apply worker. */ + ApplyBgworkerState *wstate = apply_bgworker_find(xid); + + /* + * Check if we are processing this transaction in an apply background + * worker. + */ + if (wstate) + { + apply_bgworker_send_data(wstate, s->len, s->data); + + /* + * Wait for apply background worker to finish. This is required to + * maintain commit order which avoids failures due to transaction + * dependencies and deadlocks. + */ + if (subxid == xid) + { + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_FINISHED); + apply_bgworker_free(wstate); + } + } + + /* + * We are in main apply worker and the transaction has been serialized + * to file. + */ + else + serialize_stream_abort(xid, subxid); + } reset_apply_error_context_info(); } @@ -1468,8 +1763,8 @@ apply_spooled_messages(TransactionId xid, XLogRecPtr lsn) static void apply_handle_stream_commit(StringInfo s) { - TransactionId xid; LogicalRepCommitData commit_data; + TransactionId xid; if (in_streamed_transaction) ereport(ERROR, @@ -1479,14 +1774,80 @@ apply_handle_stream_commit(StringInfo s) xid = logicalrep_read_stream_commit(s, &commit_data); set_apply_error_context_xact(xid, commit_data.commit_lsn); - elog(DEBUG1, "received commit for streamed transaction %u", xid); + if (am_apply_bgworker()) + { + /* + * Update origin state so we can restart streaming from correct + * position in case of crash. + */ + replorigin_session_origin_lsn = commit_data.end_lsn; + replorigin_session_origin_timestamp = commit_data.committime; - apply_spooled_messages(xid, commit_data.commit_lsn); + CommitTransactionCommand(); + EndTransactionBlock(false); + CommitTransactionCommand(); - apply_handle_commit_internal(&commit_data); + in_remote_transaction = false; + + pgstat_report_stat(false); + + list_free(subxactlist); + subxactlist = NIL; + + apply_bgworker_set_status(APPLY_BGWORKER_FINISHED); + } + else + { + /* This is the main apply worker. */ + ApplyBgworkerState *wstate = apply_bgworker_find(xid); + + elog(DEBUG1, "received commit for streamed transaction %u", xid); + + /* + * Check if we are processing this transaction in an apply background + * worker. + */ + if (wstate) + { + /* Send commit message */ + apply_bgworker_send_data(wstate, s->len, s->data); + + /* + * Wait for apply background worker to finish. This is required to + * maintain commit order which avoids failures due to transaction + * dependencies and deadlocks. + */ + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_FINISHED); + + pgstat_report_stat(false); + store_flush_position(commit_data.end_lsn); + stop_skipping_changes(); - /* unlink the files with serialized changes and subxact info */ - stream_cleanup_files(MyLogicalRepWorker->subid, xid); + apply_bgworker_free(wstate); + + /* + * The transaction is either non-empty or skipped, so we clear the + * subskiplsn. + */ + clear_subscription_skip_lsn(commit_data.commit_lsn); + } + else + { + /* + * The transaction has been serialized to file, so replay all the + * spooled operations. + */ + apply_spooled_messages(xid, commit_data.commit_lsn); + + apply_handle_commit_internal(&commit_data); + + /* Unlink the files with serialized changes and subxact info. */ + stream_cleanup_files(MyLogicalRepWorker->subid, xid); + } + } + + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(commit_data.end_lsn); @@ -2463,11 +2824,10 @@ apply_handle_truncate(StringInfo s) end_replication_step(); } - /* * Logical replication protocol message dispatcher. */ -static void +void apply_dispatch(StringInfo s) { LogicalRepMsgType action = pq_getmsgbyte(s); @@ -2636,6 +2996,10 @@ store_flush_position(XLogRecPtr remote_lsn) { FlushPosition *flushpos; + /* We only need to collect the LSN in main apply worker */ + if (am_apply_bgworker()) + return; + /* Need to do this in permanent context */ MemoryContextSwitchTo(ApplyContext); @@ -2650,7 +3014,7 @@ store_flush_position(XLogRecPtr remote_lsn) /* Update statistics of the worker. */ -static void +void UpdateWorkerStats(XLogRecPtr last_lsn, TimestampTz send_time, bool reply) { MyLogicalRepWorker->last_lsn = last_lsn; @@ -2812,6 +3176,9 @@ LogicalRepApplyLoop(XLogRecPtr last_received) AcceptInvalidationMessages(); maybe_reread_subscription(); + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); + /* Process any table synchronization changes. */ process_syncing_tables(last_received); } @@ -3113,7 +3480,7 @@ maybe_reread_subscription(void) /* * Callback from subscription syscache invalidation. */ -static void +void subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue) { MySubscriptionValid = false; @@ -3709,7 +4076,7 @@ ApplyWorkerMain(Datum main_arg) originid = replorigin_by_name(originname, true); if (!OidIsValid(originid)) originid = replorigin_create(originname); - replorigin_session_setup(originid); + replorigin_session_setup(originid, true); replorigin_session_origin = originid; origin_startpos = replorigin_session_get_progress(false); CommitTransactionCommand(); @@ -3756,7 +4123,7 @@ ApplyWorkerMain(Datum main_arg) options.proto.logical.publication_names = MySubscription->publications; options.proto.logical.binary = MySubscription->binary; - options.proto.logical.streaming = MySubscription->stream; + options.proto.logical.streaming = (MySubscription->stream != SUBSTREAM_OFF); options.proto.logical.twophase = false; if (!am_tablesync_worker()) @@ -3914,7 +4281,8 @@ clear_subscription_skip_lsn(XLogRecPtr finish_lsn) XLogRecPtr myskiplsn = MySubscription->skiplsn; bool started_tx = false; - if (likely(XLogRecPtrIsInvalid(myskiplsn))) + if (likely(XLogRecPtrIsInvalid(myskiplsn)) || + am_apply_bgworker()) return; if (!IsTransactionState()) @@ -3986,7 +4354,7 @@ clear_subscription_skip_lsn(XLogRecPtr finish_lsn) } /* Error callback to give more context info about the change being applied */ -static void +void apply_error_callback(void *arg) { ApplyErrorCallbackArg *errarg = &apply_error_callback_arg; @@ -4014,23 +4382,47 @@ apply_error_callback(void *arg) errarg->remote_xid, LSN_FORMAT_ARGS(errarg->finish_lsn)); } - else if (errarg->remote_attnum < 0) - errcontext("processing remote data for replication origin \"%s\" during \"%s\" for replication target relation \"%s.%s\" in transaction %u finished at %X/%X", - errarg->origin_name, - logicalrep_message_type(errarg->command), - errarg->rel->remoterel.nspname, - errarg->rel->remoterel.relname, - errarg->remote_xid, - LSN_FORMAT_ARGS(errarg->finish_lsn)); else - errcontext("processing remote data for replication origin \"%s\" during \"%s\" for replication target relation \"%s.%s\" column \"%s\" in transaction %u finished at %X/%X", - errarg->origin_name, - logicalrep_message_type(errarg->command), - errarg->rel->remoterel.nspname, - errarg->rel->remoterel.relname, - errarg->rel->remoterel.attnames[errarg->remote_attnum], - errarg->remote_xid, - LSN_FORMAT_ARGS(errarg->finish_lsn)); + { + if (errarg->remote_attnum < 0) + { + if (XLogRecPtrIsInvalid(errarg->finish_lsn)) + errcontext("processing remote data for replication origin \"%s\" during \"%s\" for replication target relation \"%s.%s\" in transaction %u", + errarg->origin_name, + logicalrep_message_type(errarg->command), + errarg->rel->remoterel.nspname, + errarg->rel->remoterel.relname, + errarg->remote_xid); + else + errcontext("processing remote data for replication origin \"%s\" during \"%s\" for replication target relation \"%s.%s\" in transaction %u finished at %X/%X", + errarg->origin_name, + logicalrep_message_type(errarg->command), + errarg->rel->remoterel.nspname, + errarg->rel->remoterel.relname, + errarg->remote_xid, + LSN_FORMAT_ARGS(errarg->finish_lsn)); + } + else + { + if (XLogRecPtrIsInvalid(errarg->finish_lsn)) + errcontext("processing remote data for replication origin \"%s\" during \"%s\" for replication target relation \"%s.%s\" column \"%s\" in transaction %u", + errarg->origin_name, + logicalrep_message_type(errarg->command), + errarg->rel->remoterel.nspname, + errarg->rel->remoterel.relname, + errarg->rel->remoterel.attnames[errarg->remote_attnum], + errarg->remote_xid); + else + errcontext("processing remote data for replication origin \"%s\" during \"%s\" for replication target relation \"%s.%s\" column \"%s\" in transaction %u finished at %X/%X", + errarg->origin_name, + logicalrep_message_type(errarg->command), + errarg->rel->remoterel.nspname, + errarg->rel->remoterel.relname, + errarg->rel->remoterel.attnames[errarg->remote_attnum], + errarg->remote_xid, + LSN_FORMAT_ARGS(errarg->finish_lsn)); + } + } } /* Set transaction information of apply error callback */ diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c index 8deae57143..6ebfa24baf 100644 --- a/src/backend/replication/pgoutput/pgoutput.c +++ b/src/backend/replication/pgoutput/pgoutput.c @@ -1833,7 +1833,7 @@ pgoutput_stream_abort(struct LogicalDecodingContext *ctx, Assert(rbtxn_is_streamed(toptxn)); OutputPluginPrepareWrite(ctx, true); - logicalrep_write_stream_abort(ctx->out, toptxn->xid, txn->xid); + logicalrep_write_stream_abort(ctx->out, toptxn->xid, txn, abort_lsn); OutputPluginWrite(ctx, true); cleanup_rel_sync_cache(toptxn->xid, false); diff --git a/src/backend/utils/activity/wait_event.c b/src/backend/utils/activity/wait_event.c index 87c15b9c6f..ba781e6f08 100644 --- a/src/backend/utils/activity/wait_event.c +++ b/src/backend/utils/activity/wait_event.c @@ -388,6 +388,9 @@ pgstat_get_wait_ipc(WaitEventIPC w) case WAIT_EVENT_HASH_GROW_BUCKETS_REINSERT: event_name = "HashGrowBucketsReinsert"; break; + case WAIT_EVENT_LOGICAL_APPLY_WORKER_STATE_CHANGE: + event_name = "LogicalApplyWorkerStateChange"; + break; case WAIT_EVENT_LOGICAL_SYNC_DATA: event_name = "LogicalSyncData"; break; diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c index a7cc49898b..9590359f95 100644 --- a/src/backend/utils/misc/guc.c +++ b/src/backend/utils/misc/guc.c @@ -3220,6 +3220,18 @@ static struct config_int ConfigureNamesInt[] = NULL, NULL, NULL }, + { + {"max_apply_bgworkers_per_subscription", + PGC_SIGHUP, + REPLICATION_SUBSCRIBERS, + gettext_noop("Maximum number of apply background workers per subscription."), + NULL, + }, + &max_apply_bgworkers_per_subscription, + 2, 0, MAX_BACKENDS, + NULL, NULL, NULL + }, + { {"log_rotation_age", PGC_SIGHUP, LOGGING_WHERE, gettext_noop("Sets the amount of time to wait before forcing " diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample index 48ad80cf2e..292808dd83 100644 --- a/src/backend/utils/misc/postgresql.conf.sample +++ b/src/backend/utils/misc/postgresql.conf.sample @@ -360,6 +360,7 @@ #max_logical_replication_workers = 4 # taken from max_worker_processes # (change requires restart) #max_sync_workers_per_subscription = 2 # taken from max_logical_replication_workers +#max_apply_bgworkers_per_subscription = 2 # taken from max_logical_replication_workers #------------------------------------------------------------------------------ diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 7cc9c72e49..53a41b4e3f 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -4450,7 +4450,7 @@ getSubscriptions(Archive *fout) if (fout->remoteVersion >= 140000) appendPQExpBufferStr(query, " s.substream,\n"); else - appendPQExpBufferStr(query, " false AS substream,\n"); + appendPQExpBufferStr(query, " 'f' AS substream,\n"); if (fout->remoteVersion >= 150000) appendPQExpBufferStr(query, @@ -4580,8 +4580,10 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo) if (strcmp(subinfo->subbinary, "t") == 0) appendPQExpBufferStr(query, ", binary = true"); - if (strcmp(subinfo->substream, "f") != 0) + if (strcmp(subinfo->substream, "t") == 0) appendPQExpBufferStr(query, ", streaming = on"); + else if (strcmp(subinfo->substream, "p") == 0) + appendPQExpBufferStr(query, ", streaming = parallel"); if (strcmp(subinfo->subtwophasestate, two_phase_disabled) != 0) appendPQExpBufferStr(query, ", two_phase = on"); diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h index d1260f590c..2995e019a8 100644 --- a/src/include/catalog/pg_subscription.h +++ b/src/include/catalog/pg_subscription.h @@ -68,7 +68,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW bool subbinary; /* True if the subscription wants the * publisher to send data in binary */ - bool substream; /* Stream in-progress transactions. */ + char substream; /* Stream in-progress transactions. + * See SUBSTREAM_xxx constants. */ char subtwophasestate; /* Stream two-phase transactions */ @@ -109,7 +110,8 @@ typedef struct Subscription bool enabled; /* Indicates if the subscription is enabled */ bool binary; /* Indicates if the subscription wants data in * binary format */ - bool stream; /* Allow streaming in-progress transactions. */ + char stream; /* Allow streaming in-progress transactions. + * See SUBSTREAM_xxx constants. */ char twophasestate; /* Allow streaming two-phase transactions */ bool disableonerr; /* Indicates if the subscription should be * automatically disabled if a worker error @@ -120,6 +122,18 @@ typedef struct Subscription List *publications; /* List of publication names to subscribe to */ } Subscription; +/* Disallow streaming in-progress transactions */ +#define SUBSTREAM_OFF 'f' + +/* + * Streaming transactions are written to a temporary file and applied only + * after the transaction is committed on upstream. + */ +#define SUBSTREAM_ON 't' + +/* Streaming transactions are applied immediately via a background worker */ +#define SUBSTREAM_PARALLEL 'p' + extern Subscription *GetSubscription(Oid subid, bool missing_ok); extern void FreeSubscription(Subscription *sub); extern void DisableSubscription(Oid subid); diff --git a/src/include/replication/logicallauncher.h b/src/include/replication/logicallauncher.h index f1e2821e25..ac8ef94381 100644 --- a/src/include/replication/logicallauncher.h +++ b/src/include/replication/logicallauncher.h @@ -14,6 +14,7 @@ extern PGDLLIMPORT int max_logical_replication_workers; extern PGDLLIMPORT int max_sync_workers_per_subscription; +extern PGDLLIMPORT int max_apply_bgworkers_per_subscription; extern void ApplyLauncherRegister(void); extern void ApplyLauncherMain(Datum main_arg); diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h index a771ab8ff3..7bba77c9e7 100644 --- a/src/include/replication/logicalproto.h +++ b/src/include/replication/logicalproto.h @@ -175,6 +175,17 @@ typedef struct LogicalRepRollbackPreparedTxnData char gid[GIDSIZE]; } LogicalRepRollbackPreparedTxnData; +/* + * Transaction protocol information for stream abort. + */ +typedef struct LogicalRepStreamAbortData +{ + TransactionId xid; + TransactionId subxid; + XLogRecPtr abort_lsn; + TimestampTz abort_time; +} LogicalRepStreamAbortData; + extern void logicalrep_write_begin(StringInfo out, ReorderBufferTXN *txn); extern void logicalrep_read_begin(StringInfo in, LogicalRepBeginData *begin_data); @@ -246,9 +257,11 @@ extern void logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn extern TransactionId logicalrep_read_stream_commit(StringInfo out, LogicalRepCommitData *commit_data); extern void logicalrep_write_stream_abort(StringInfo out, TransactionId xid, - TransactionId subxid); -extern void logicalrep_read_stream_abort(StringInfo in, TransactionId *xid, - TransactionId *subxid); + ReorderBufferTXN *txn, + XLogRecPtr abort_lsn); +extern void logicalrep_read_stream_abort(StringInfo in, + LogicalRepStreamAbortData *abort_data, + bool include_abort_lsn); extern char *logicalrep_message_type(LogicalRepMsgType action); #endif /* LOGICAL_PROTO_H */ diff --git a/src/include/replication/logicalworker.h b/src/include/replication/logicalworker.h index cd1b6e8afc..6a1af7f13c 100644 --- a/src/include/replication/logicalworker.h +++ b/src/include/replication/logicalworker.h @@ -13,6 +13,7 @@ #define LOGICALWORKER_H extern void ApplyWorkerMain(Datum main_arg); +extern void ApplyBgworkerMain(Datum main_arg); extern bool IsLogicalWorker(void); diff --git a/src/include/replication/origin.h b/src/include/replication/origin.h index 2d1b5e5c28..c7389b40a7 100644 --- a/src/include/replication/origin.h +++ b/src/include/replication/origin.h @@ -53,7 +53,7 @@ extern XLogRecPtr replorigin_get_progress(RepOriginId node, bool flush); extern void replorigin_session_advance(XLogRecPtr remote_commit, XLogRecPtr local_commit); -extern void replorigin_session_setup(RepOriginId node); +extern void replorigin_session_setup(RepOriginId node, bool must_acquire); extern void replorigin_session_reset(void); extern XLogRecPtr replorigin_session_get_progress(bool flush); diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h index 4a01f877e5..aba1640f4d 100644 --- a/src/include/replication/reorderbuffer.h +++ b/src/include/replication/reorderbuffer.h @@ -301,6 +301,7 @@ typedef struct ReorderBufferTXN { TimestampTz commit_time; TimestampTz prepare_time; + TimestampTz abort_time; } xact_time; /* @@ -647,9 +648,11 @@ extern void ReorderBufferFinishPrepared(ReorderBuffer *rb, TransactionId xid, extern void ReorderBufferAssignChild(ReorderBuffer *, TransactionId, TransactionId, XLogRecPtr commit_lsn); extern void ReorderBufferCommitChild(ReorderBuffer *, TransactionId, TransactionId, XLogRecPtr commit_lsn, XLogRecPtr end_lsn); -extern void ReorderBufferAbort(ReorderBuffer *, TransactionId, XLogRecPtr lsn); +extern void ReorderBufferAbort(ReorderBuffer *, TransactionId, XLogRecPtr lsn, + TimestampTz abort_time); extern void ReorderBufferAbortOld(ReorderBuffer *, TransactionId xid); -extern void ReorderBufferForget(ReorderBuffer *, TransactionId, XLogRecPtr lsn); +extern void ReorderBufferForget(ReorderBuffer *, TransactionId, XLogRecPtr lsn, + TimestampTz abort_time); extern void ReorderBufferInvalidate(ReorderBuffer *, TransactionId, XLogRecPtr lsn); extern void ReorderBufferSetBaseSnapshot(ReorderBuffer *, TransactionId, XLogRecPtr lsn, struct SnapshotData *snap); diff --git a/src/include/replication/worker_internal.h b/src/include/replication/worker_internal.h index 901845abc2..c2ff5d4144 100644 --- a/src/include/replication/worker_internal.h +++ b/src/include/replication/worker_internal.h @@ -17,8 +17,11 @@ #include "access/xlogdefs.h" #include "catalog/pg_subscription.h" #include "datatype/timestamp.h" +#include "replication/logicalrelation.h" #include "storage/fileset.h" #include "storage/lock.h" +#include "storage/shm_mq.h" +#include "storage/shm_toc.h" #include "storage/spin.h" @@ -60,6 +63,9 @@ typedef struct LogicalRepWorker */ FileSet *stream_fileset; + /* Indicates if this slot is used for an apply background worker. */ + bool subworker; + /* Stats. */ XLogRecPtr last_lsn; TimestampTz last_send_time; @@ -68,8 +74,68 @@ typedef struct LogicalRepWorker TimestampTz reply_time; } LogicalRepWorker; +/* Struct for saving and restoring apply errcontext information */ +typedef struct ApplyErrorCallbackArg +{ + LogicalRepMsgType command; /* 0 if invalid */ + LogicalRepRelMapEntry *rel; + + /* Remote node information */ + int remote_attnum; /* -1 if invalid */ + TransactionId remote_xid; + XLogRecPtr finish_lsn; + char *origin_name; +} ApplyErrorCallbackArg; + +/* + * Status of apply background worker. + */ +typedef enum ApplyBgworkerStatus +{ + APPLY_BGWORKER_BUSY = 0, /* assigned to a transaction */ + APPLY_BGWORKER_FINISHED, /* transaction is completed */ + APPLY_BGWORKER_EXIT /* exit */ +} ApplyBgworkerStatus; + +/* + * Struct for sharing information between apply main and apply background + * workers. + */ +typedef struct ApplyBgworkerShared +{ + slock_t mutex; + + /* Status of apply background worker. */ + ApplyBgworkerStatus status; + + /* server version of publisher. */ + int server_version; + + TransactionId stream_xid; + uint32 n; /* id of apply background worker */ +} ApplyBgworkerShared; + +/* + * Struct for maintaining an apply background worker. + */ +typedef struct ApplyBgworkerState +{ + shm_mq_handle *mq_handle; + dsm_segment *dsm_seg; + ApplyBgworkerShared volatile *pstate; +} ApplyBgworkerState; + /* Main memory context for apply worker. Permanent during worker lifetime. */ extern PGDLLIMPORT MemoryContext ApplyContext; +extern PGDLLIMPORT MemoryContext ApplyMessageContext; + +extern PGDLLIMPORT ApplyErrorCallbackArg apply_error_callback_arg; + +extern PGDLLIMPORT bool MySubscriptionValid; + +extern PGDLLIMPORT volatile ApplyBgworkerShared *MyParallelState; + +extern PGDLLIMPORT List *subxactlist; /* libpqreceiver connection */ extern PGDLLIMPORT struct WalReceiverConn *LogRepWorkerWalRcvConn; @@ -79,18 +145,22 @@ extern PGDLLIMPORT Subscription *MySubscription; extern PGDLLIMPORT LogicalRepWorker *MyLogicalRepWorker; extern PGDLLIMPORT bool in_remote_transaction; +extern PGDLLIMPORT bool in_streamed_transaction; +extern PGDLLIMPORT TransactionId stream_xid; extern void logicalrep_worker_attach(int slot); extern LogicalRepWorker *logicalrep_worker_find(Oid subid, Oid relid, bool only_running); extern List *logicalrep_workers_find(Oid subid, bool only_running); -extern void logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, - Oid userid, Oid relid); +extern bool logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, + Oid userid, Oid relid, + dsm_handle subworker_dsm); extern void logicalrep_worker_stop(Oid subid, Oid relid); extern void logicalrep_worker_wakeup(Oid subid, Oid relid); extern void logicalrep_worker_wakeup_ptr(LogicalRepWorker *worker); extern int logicalrep_sync_worker_count(Oid subid); +extern int logicalrep_apply_background_worker_count(Oid subid); extern void ReplicationOriginNameForTablesync(Oid suboid, Oid relid, char *originname, int szorgname); @@ -103,10 +173,38 @@ extern void process_syncing_tables(XLogRecPtr current_lsn); extern void invalidate_syncing_table_states(Datum arg, int cacheid, uint32 hashvalue); +extern void UpdateWorkerStats(XLogRecPtr last_lsn, TimestampTz send_time, + bool reply); + +extern void apply_dispatch(StringInfo s); + +/* Function for apply error callback */ +extern void apply_error_callback(void *arg); + +extern void subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue); + +/* Apply background worker setup and interactions */ +extern ApplyBgworkerState *apply_bgworker_start(TransactionId xid); +extern ApplyBgworkerState *apply_bgworker_find(TransactionId xid); +extern void apply_bgworker_wait_for(ApplyBgworkerState *wstate, + ApplyBgworkerStatus wait_for_status); +extern void apply_bgworker_send_data(ApplyBgworkerState *wstate, Size nbytes, + const void *data); +extern void apply_bgworker_free(ApplyBgworkerState *wstate); +extern void apply_bgworker_check_status(void); +extern void apply_bgworker_set_status(ApplyBgworkerStatus status); +extern void apply_bgworker_subxact_info_add(TransactionId current_xid); + static inline bool am_tablesync_worker(void) { return OidIsValid(MyLogicalRepWorker->relid); } +static inline bool +am_apply_bgworker(void) +{ + return MyLogicalRepWorker->subworker; +} + #endif /* WORKER_INTERNAL_H */ diff --git a/src/include/utils/wait_event.h b/src/include/utils/wait_event.h index b578e2ec75..c2d2a114d7 100644 --- a/src/include/utils/wait_event.h +++ b/src/include/utils/wait_event.h @@ -105,6 +105,7 @@ typedef enum WAIT_EVENT_HASH_GROW_BUCKETS_ALLOCATE, WAIT_EVENT_HASH_GROW_BUCKETS_ELECT, WAIT_EVENT_HASH_GROW_BUCKETS_REINSERT, + WAIT_EVENT_LOGICAL_APPLY_WORKER_STATE_CHANGE, WAIT_EVENT_LOGICAL_SYNC_DATA, WAIT_EVENT_LOGICAL_SYNC_STATE_CHANGE, WAIT_EVENT_MQ_INTERNAL, diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out index 5db7146e06..919266ae06 100644 --- a/src/test/regress/expected/subscription.out +++ b/src/test/regress/expected/subscription.out @@ -197,7 +197,7 @@ ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); DROP SUBSCRIPTION regress_testsub; -- fail - streaming must be boolean CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = foo); -ERROR: streaming requires a Boolean value +ERROR: streaming requires a Boolean value or "parallel" -- now it works CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = true); WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 4fb746930a..664d4b8b1a 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -116,6 +116,10 @@ Append AppendPath AppendRelInfo AppendState +ApplyBgworkerEntry +ApplyBgworkerShared +ApplyBgworkerState +ApplyBgworkerStatus ApplyErrorCallbackArg ApplyExecutionData ApplySubXactData @@ -1483,6 +1487,7 @@ LogicalRepRelId LogicalRepRelMapEntry LogicalRepRelation LogicalRepRollbackPreparedTxnData +LogicalRepStreamAbortData LogicalRepTupleData LogicalRepTyp LogicalRepWorker -- 2.23.0.windows.1 [application/octet-stream] v14-0002-Test-streaming-parallel-option-in-tap-test.patch (69.1K, ../../OS3PR01MB62758DBE8FA12BA72A43AC819EB89@OS3PR01MB6275.jpnprd01.prod.outlook.com/3-v14-0002-Test-streaming-parallel-option-in-tap-test.patch) download | inline diff: From b2f6169e8739a10d2293faa5bf2474a940e27d7e Mon Sep 17 00:00:00 2001 From: "shiy.fnst" <[email protected]> Date: Fri, 13 May 2022 14:50:30 +0800 Subject: [PATCH v14 2/4] Test streaming parallel option in tap test Change all TAP tests using the SUBSCRIPTION "streaming" option, so they now test both 'on' and 'parallel' values. --- src/test/subscription/t/015_stream.pl | 199 ++++--- src/test/subscription/t/016_stream_subxact.pl | 119 +++-- src/test/subscription/t/017_stream_ddl.pl | 188 ++++--- .../t/018_stream_subxact_abort.pl | 195 ++++--- .../t/019_stream_subxact_ddl_abort.pl | 110 +++- .../subscription/t/022_twophase_cascade.pl | 363 +++++++------ .../subscription/t/023_twophase_stream.pl | 498 ++++++++++-------- 7 files changed, 1035 insertions(+), 637 deletions(-) diff --git a/src/test/subscription/t/015_stream.pl b/src/test/subscription/t/015_stream.pl index 6561b189de..0bdd234935 100644 --- a/src/test/subscription/t/015_stream.pl +++ b/src/test/subscription/t/015_stream.pl @@ -8,6 +8,116 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" +# parameter so the same code can be run both for the streaming=on and +# streaming=parallel cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_parallel) = @_; + + # Interleave a pair of transactions, each exceeding the 64kB limit. + my $in = ''; + my $out = ''; + + my $timer = IPC::Run::timeout($PostgreSQL::Test::Utils::timeout_default); + + my $h = $node_publisher->background_psql('postgres', \$in, \$out, $timer, + on_error_stop => 0); + + # Check that a background worker starts if "streaming" parameter is + # specified as "parallel". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_parallel) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + $in .= q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + }; + $h->pump_nb; + + if ($is_parallel) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001, 9999) s(i); + DELETE FROM test_tab WHERE a > 5000; + COMMIT; + }); + + $in .= q{ + COMMIT; + \q + }; + $h->finish; # errors make the next test fail, so ignore them here + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), + 'check extra columns contain local defaults'); + + # Test the streaming in binary mode + $node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET (binary = on)"); + + # Insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001, 10000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + COMMIT; + }); + + $node_publisher->wait_for_catchup($appname); + + $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(6667|6667|6667), + 'check extra columns contain local defaults'); + + # Change the local values of the extra columns on the subscriber, + # update publisher, and check that subscriber retains the expected + # values. This is to ensure that non-streaming transactions behave + # properly after a streaming transaction. + $node_subscriber->safe_psql('postgres', + "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'" + ); + $node_publisher->safe_psql('postgres', + "UPDATE test_tab SET b = md5(a::text)"); + + $node_publisher->wait_for_catchup($appname); + + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab" + ); + is($result, qq(6667|6667|6667), + 'check extra columns contain locally changed data'); + + # Cleanup the test data + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE (a > 2)"); + $node_publisher->wait_for_catchup($appname); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -37,6 +147,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -54,82 +168,25 @@ my $result = "SELECT count(*), count(c), count(d = 999) FROM test_tab"); is($result, qq(2|2|2), 'check initial data was copied to subscriber'); -# Interleave a pair of transactions, each exceeding the 64kB limit. -my $in = ''; -my $out = ''; - -my $timer = IPC::Run::timeout($PostgreSQL::Test::Utils::timeout_default); - -my $h = $node_publisher->background_psql('postgres', \$in, \$out, $timer, - on_error_stop => 0); - -$in .= q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -}; -$h->pump_nb; - -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001, 9999) s(i); -DELETE FROM test_tab WHERE a > 5000; -COMMIT; -}); - -$in .= q{ -COMMIT; -\q -}; -$h->finish; # errors make the next test fail, so ignore them here - -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), 'check extra columns contain local defaults'); - -# Test the streaming in binary mode -$node_subscriber->safe_psql('postgres', - "ALTER SUBSCRIPTION tap_sub SET (binary = on)"); - -# Insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001, 10000) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -COMMIT; -}); - -$node_publisher->wait_for_catchup($appname); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(6667|6667|6667), 'check extra columns contain local defaults'); +###################################### +# Test using streaming mode 'parallel' +###################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" +); -# Change the local values of the extra columns on the subscriber, -# update publisher, and check that subscriber retains the expected -# values. This is to ensure that non-streaming transactions behave -# properly after a streaming transaction. $node_subscriber->safe_psql('postgres', - "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'" -); -$node_publisher->safe_psql('postgres', - "UPDATE test_tab SET b = md5(a::text)"); + "ALTER SUBSCRIPTION tap_sub SET(streaming = parallel, binary = off)"); -$node_publisher->wait_for_catchup($appname); +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab" -); -is($result, qq(6667|6667|6667), - 'check extra columns contain locally changed data'); +test_streaming($node_publisher, $node_subscriber, $appname, 1); $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/016_stream_subxact.pl b/src/test/subscription/t/016_stream_subxact.pl index f27f1694f2..45429dddba 100644 --- a/src/test/subscription/t/016_stream_subxact.pl +++ b/src/test/subscription/t/016_stream_subxact.pl @@ -8,6 +8,72 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" +# parameter so the same code can be run both for the streaming=on and +# streaming=parallel cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_parallel) = @_; + + # Check that a background worker starts if "streaming" parameter is + # specified as "parallel". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_parallel) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + # Insert, update and delete enough rows to exceed 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series( 3, 500) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501, 1000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001, 1500) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + SAVEPOINT s3; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501, 2000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + SAVEPOINT s4; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001, 2500) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + COMMIT; + }); + + if ($is_parallel) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(1667|1667|1667), + 'check data was copied to subscriber in streaming mode and extra columns contain local defaults' + ); + + # Cleanup the test data + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE (a > 2)"); + $node_publisher->wait_for_catchup($appname); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -37,6 +103,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -54,41 +124,26 @@ my $result = "SELECT count(*), count(c), count(d = 999) FROM test_tab"); is($result, qq(2|2|2), 'check initial data was copied to subscriber'); -# Insert, update and delete enough rows to exceed 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series( 3, 500) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501, 1000) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001, 1500) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -SAVEPOINT s3; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501, 2000) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -SAVEPOINT s4; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001, 2500) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -COMMIT; -}); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(1667|1667|1667), - 'check data was copied to subscriber in streaming mode and extra columns contain local defaults' +###################################### +# Test using streaming mode 'parallel' +###################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" ); +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = parallel)"); + +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; + +test_streaming($node_publisher, $node_subscriber, $appname, 1); + $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/017_stream_ddl.pl b/src/test/subscription/t/017_stream_ddl.pl index 0bce63b716..52dfef4780 100644 --- a/src/test/subscription/t/017_stream_ddl.pl +++ b/src/test/subscription/t/017_stream_ddl.pl @@ -8,6 +8,111 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" +# parameter so the same code can be run both for the streaming=on and +# streaming=parallel cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_parallel) = @_; + + # a small (non-streamed) transaction with DDL and DML + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab VALUES (3, md5(3::text)); + ALTER TABLE test_tab ADD COLUMN c INT; + SAVEPOINT s1; + INSERT INTO test_tab VALUES (4, md5(4::text), -4); + COMMIT; + }); + + # Check that a background worker starts if "streaming" parameter is + # specified as "parallel". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_parallel) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + # large (streamed) transaction with DDL and DML + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(5, 1000) s(i); + ALTER TABLE test_tab ADD COLUMN d INT; + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001, 2000) s(i); + COMMIT; + }); + + if ($is_parallel) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + # a small (non-streamed) transaction with DDL and DML + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab VALUES (2001, md5(2001::text), -2001, 2*2001); + ALTER TABLE test_tab ADD COLUMN e INT; + SAVEPOINT s1; + INSERT INTO test_tab VALUES (2002, md5(2002::text), -2002, 2*2002, -3*2002); + COMMIT; + }); + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d), count(e) FROM test_tab"); + is($result, qq(2002|1999|1002|1), + 'check data was copied to subscriber in streaming mode and extra columns contain local defaults' + ); + + # A large (streamed) transaction with DDL and DML. One of the DDL is performed + # after DML to ensure that we invalidate the schema sent for test_tab so that + # the next transaction has to send the schema again. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(2003,5000) s(i); + ALTER TABLE test_tab ADD COLUMN f INT; + COMMIT; + }); + + # A small transaction that won't get streamed. This is just to ensure that we + # send the schema again to reflect the last column added in the previous test. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i, 4*i FROM generate_series(5001,5005) s(i); + COMMIT; + }); + + $node_publisher->wait_for_catchup($appname); + + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d), count(e), count(f) FROM test_tab" + ); + is($result, qq(5005|5002|4005|3004|5), + 'check data was copied to subscriber for both streaming and non-streaming transactions' + ); + + # Cleanup the test data + $node_publisher->safe_psql( + 'postgres', q{ + DELETE FROM test_tab WHERE (a > 2); + ALTER TABLE test_tab DROP COLUMN c, DROP COLUMN d, DROP COLUMN e, DROP COLUMN f; + }); + $node_publisher->wait_for_catchup($appname); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -37,6 +142,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -54,76 +163,25 @@ my $result = "SELECT count(*), count(c), count(d = 999) FROM test_tab"); is($result, qq(2|0|0), 'check initial data was copied to subscriber'); -# a small (non-streamed) transaction with DDL and DML -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab VALUES (3, md5(3::text)); -ALTER TABLE test_tab ADD COLUMN c INT; -SAVEPOINT s1; -INSERT INTO test_tab VALUES (4, md5(4::text), -4); -COMMIT; -}); - -# large (streamed) transaction with DDL and DML -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(5, 1000) s(i); -ALTER TABLE test_tab ADD COLUMN d INT; -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001, 2000) s(i); -COMMIT; -}); - -# a small (non-streamed) transaction with DDL and DML -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab VALUES (2001, md5(2001::text), -2001, 2*2001); -ALTER TABLE test_tab ADD COLUMN e INT; -SAVEPOINT s1; -INSERT INTO test_tab VALUES (2002, md5(2002::text), -2002, 2*2002, -3*2002); -COMMIT; -}); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d), count(e) FROM test_tab"); -is($result, qq(2002|1999|1002|1), - 'check data was copied to subscriber in streaming mode and extra columns contain local defaults' +###################################### +# Test using streaming mode 'parallel' +###################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" ); -# A large (streamed) transaction with DDL and DML. One of the DDL is performed -# after DML to ensure that we invalidate the schema sent for test_tab so that -# the next transaction has to send the schema again. -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(2003,5000) s(i); -ALTER TABLE test_tab ADD COLUMN f INT; -COMMIT; -}); - -# A small transaction that won't get streamed. This is just to ensure that we -# send the schema again to reflect the last column added in the previous test. -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i, 4*i FROM generate_series(5001,5005) s(i); -COMMIT; -}); +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = parallel)"); -$node_publisher->wait_for_catchup($appname); +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d), count(e), count(f) FROM test_tab"); -is($result, qq(5005|5002|4005|3004|5), - 'check data was copied to subscriber for both streaming and non-streaming transactions' -); +test_streaming($node_publisher, $node_subscriber, $appname, 1); $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/018_stream_subxact_abort.pl b/src/test/subscription/t/018_stream_subxact_abort.pl index 7155442e76..68f0e4b0d1 100644 --- a/src/test/subscription/t/018_stream_subxact_abort.pl +++ b/src/test/subscription/t/018_stream_subxact_abort.pl @@ -8,6 +8,113 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" +# parameter so the same code can be run both for the streaming=on and +# streaming=parallel cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_parallel) = @_; + + # Check that a background worker starts if "streaming" parameter is + # specified as "parallel". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_parallel) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + # large (streamed) transaction with DDL, DML and ROLLBACKs + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i); + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501,1000) s(i); + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001,1500) s(i); + SAVEPOINT s3; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501,2000) s(i); + ROLLBACK TO s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001,2500) s(i); + ROLLBACK TO s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2501,3000) s(i); + SAVEPOINT s4; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3001,3500) s(i); + SAVEPOINT s5; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3501,4000) s(i); + COMMIT; + }); + + if ($is_parallel) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c) FROM test_tab"); + is($result, qq(2000|0), + 'check rollback to savepoint was reflected on subscriber and extra columns contain local defaults' + ); + + # large (streamed) transaction with subscriber receiving out of order + # subtransaction ROLLBACKs + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(4001,4500) s(i); + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001,5500) s(i); + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(6001,6500) s(i); + SAVEPOINT s3; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(7001,7500) s(i); + RELEASE s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(8001,8500) s(i); + ROLLBACK TO s1; + COMMIT; + }); + + $node_publisher->wait_for_catchup($appname); + + $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c) FROM test_tab"); + is($result, qq(2500|0), + 'check rollback to savepoint was reflected on subscriber'); + + # large (streamed) transaction with subscriber receiving rollback + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(8501,9000) s(i); + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(9001,9500) s(i); + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(9501,10000) s(i); + ROLLBACK; + }); + + $node_publisher->wait_for_catchup($appname); + + $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c) FROM test_tab"); + is($result, qq(2500|0), 'check rollback was reflected on subscriber'); + + # Cleanup the test data + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE (a > 2)"); + $node_publisher->wait_for_catchup($appname); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -36,6 +143,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -53,81 +164,25 @@ my $result = "SELECT count(*), count(c) FROM test_tab"); is($result, qq(2|0), 'check initial data was copied to subscriber'); -# large (streamed) transaction with DDL, DML and ROLLBACKs -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i); -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501,1000) s(i); -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001,1500) s(i); -SAVEPOINT s3; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501,2000) s(i); -ROLLBACK TO s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001,2500) s(i); -ROLLBACK TO s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2501,3000) s(i); -SAVEPOINT s4; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3001,3500) s(i); -SAVEPOINT s5; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3501,4000) s(i); -COMMIT; -}); - -$node_publisher->wait_for_catchup($appname); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c) FROM test_tab"); -is($result, qq(2000|0), - 'check rollback to savepoint was reflected on subscriber and extra columns contain local defaults' +###################################### +# Test using streaming mode 'parallel' +###################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" ); -# large (streamed) transaction with subscriber receiving out of order -# subtransaction ROLLBACKs -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(4001,4500) s(i); -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001,5500) s(i); -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(6001,6500) s(i); -SAVEPOINT s3; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(7001,7500) s(i); -RELEASE s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(8001,8500) s(i); -ROLLBACK TO s1; -COMMIT; -}); - -$node_publisher->wait_for_catchup($appname); +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = parallel)"); -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c) FROM test_tab"); -is($result, qq(2500|0), - 'check rollback to savepoint was reflected on subscriber'); - -# large (streamed) transaction with subscriber receiving rollback -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(8501,9000) s(i); -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(9001,9500) s(i); -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(9501,10000) s(i); -ROLLBACK; -}); +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c) FROM test_tab"); -is($result, qq(2500|0), 'check rollback was reflected on subscriber'); +test_streaming($node_publisher, $node_subscriber, $appname, 1); $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/019_stream_subxact_ddl_abort.pl b/src/test/subscription/t/019_stream_subxact_ddl_abort.pl index dbd0fca4d1..b276063721 100644 --- a/src/test/subscription/t/019_stream_subxact_ddl_abort.pl +++ b/src/test/subscription/t/019_stream_subxact_ddl_abort.pl @@ -9,6 +9,69 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" +# parameter so the same code can be run both for the streaming=on and +# streaming=parallel cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_parallel) = @_; + + # Check that a background worker starts if "streaming" parameter is + # specified as "parallel". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_parallel) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + # large (streamed) transaction with DDL, DML and ROLLBACKs + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i); + ALTER TABLE test_tab ADD COLUMN c INT; + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(501,1000) s(i); + ALTER TABLE test_tab ADD COLUMN d INT; + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001,1500) s(i); + ALTER TABLE test_tab ADD COLUMN e INT; + SAVEPOINT s3; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(1501,2000) s(i); + ALTER TABLE test_tab DROP COLUMN c; + ROLLBACK TO s1; + INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(501,1000) s(i); + COMMIT; + }); + + if ($is_parallel) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c) FROM test_tab"); + is($result, qq(1000|500), + 'check rollback to savepoint was reflected on subscriber and extra columns contain local defaults' + ); + + # Cleanup the test data + $node_publisher->safe_psql( + 'postgres', q{ + DELETE FROM test_tab WHERE (a > 2); + ALTER TABLE test_tab DROP COLUMN c; + }); + $node_publisher->wait_for_catchup($appname); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -37,6 +100,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -54,35 +121,26 @@ my $result = "SELECT count(*), count(c) FROM test_tab"); is($result, qq(2|0), 'check initial data was copied to subscriber'); -# large (streamed) transaction with DDL, DML and ROLLBACKs -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i); -ALTER TABLE test_tab ADD COLUMN c INT; -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(501,1000) s(i); -ALTER TABLE test_tab ADD COLUMN d INT; -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001,1500) s(i); -ALTER TABLE test_tab ADD COLUMN e INT; -SAVEPOINT s3; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(1501,2000) s(i); -ALTER TABLE test_tab DROP COLUMN c; -ROLLBACK TO s1; -INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(501,1000) s(i); -COMMIT; -}); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c) FROM test_tab"); -is($result, qq(1000|500), - 'check rollback to savepoint was reflected on subscriber and extra columns contain local defaults' +###################################### +# Test using streaming mode 'parallel' +###################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" ); +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = parallel)"); + +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; + +test_streaming($node_publisher, $node_subscriber, $appname, 1); + $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/022_twophase_cascade.pl b/src/test/subscription/t/022_twophase_cascade.pl index 7a797f37ba..0a4152d3be 100644 --- a/src/test/subscription/t/022_twophase_cascade.pl +++ b/src/test/subscription/t/022_twophase_cascade.pl @@ -11,6 +11,208 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" parameter +# so the same code can be run both for the streaming=on and streaming=parallel +# cases. +sub test_streaming +{ + my ($node_A, $node_B, $node_C, $appname_B, $appname_C, $streaming_mode) = + @_; + + my $oldpid_B = $node_A->safe_psql( + 'postgres', " + SELECT pid FROM pg_stat_replication + WHERE application_name = '$appname_B' AND state = 'streaming';"); + my $oldpid_C = $node_B->safe_psql( + 'postgres', " + SELECT pid FROM pg_stat_replication + WHERE application_name = '$appname_C' AND state = 'streaming';"); + + # Setup logical replication streaming mode + + $node_B->safe_psql( + 'postgres', " + ALTER SUBSCRIPTION tap_sub_B + SET (streaming = $streaming_mode);"); + $node_C->safe_psql( + 'postgres', " + ALTER SUBSCRIPTION tap_sub_C + SET (streaming = $streaming_mode)"); + + # Wait for subscribers to finish initialization + + $node_A->poll_query_until( + 'postgres', " + SELECT pid != $oldpid_B FROM pg_stat_replication + WHERE application_name = '$appname_B' AND state = 'streaming';" + ) or die "Timed out while waiting for apply to restart"; + $node_B->poll_query_until( + 'postgres', " + SELECT pid != $oldpid_C FROM pg_stat_replication + WHERE application_name = '$appname_C' AND state = 'streaming';" + ) or die "Timed out while waiting for apply to restart"; + + ############################### + # Test 2PC PREPARE / COMMIT PREPARED. + # 1. Data is streamed as a 2PC transaction. + # 2. Then do commit prepared. + # + # Expect all data is replicated on subscriber(s) after the commit. + ############################### + + # Check that a background worker starts if "streaming" optparameterion is + # specified as "parallel". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($streaming_mode eq 'parallel') + { + $node_B->append_conf('postgresql.conf', "log_min_messages = debug1"); + $node_B->reload; + + $node_C->append_conf('postgresql.conf', "log_min_messages = debug1"); + $node_C->reload; + } + + # Insert, update and delete enough rows to exceed the 64kB limit. + # Then 2PC PREPARE + $node_A->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + if ($streaming_mode eq 'parallel') + { + $node_B->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_B->append_conf('postgresql.conf', "log_min_messages = warning"); + $node_B->reload; + + $node_C->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_C->append_conf('postgresql.conf', "log_min_messages = warning"); + $node_C->reload; + } + + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); + + # check the transaction state is prepared on subscriber(s) + my $result = + $node_B->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber B'); + $result = + $node_C->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber C'); + + # 2PC COMMIT + $node_A->safe_psql('postgres', "COMMIT PREPARED 'test_prepared_tab';"); + + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); + + # check that transaction was committed on subscriber(s) + $result = $node_B->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), + 'Rows inserted by 2PC have committed on subscriber B, and extra columns have local defaults' + ); + $result = $node_C->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), + 'Rows inserted by 2PC have committed on subscriber C, and extra columns have local defaults' + ); + + # check the transaction state is ended on subscriber(s) + $result = + $node_B->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is committed on subscriber B'); + $result = + $node_C->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is committed on subscriber C'); + + ############################### + # Test 2PC PREPARE with a nested ROLLBACK TO SAVEPOINT. + # 0. Cleanup from previous test leaving only 2 rows. + # 1. Insert one more row. + # 2. Record a SAVEPOINT. + # 3. Data is streamed using 2PC. + # 4. Do rollback to SAVEPOINT prior to the streamed inserts. + # 5. Then COMMIT PREPARED. + # + # Expect data after the SAVEPOINT is aborted leaving only 3 rows (= 2 original + 1 from step 1). + ############################### + + # First, delete the data except for 2 rows (delete will be replicated) + $node_A->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); + + # 2PC PREPARE with a nested ROLLBACK TO SAVEPOINT + $node_A->safe_psql( + 'postgres', " + BEGIN; + INSERT INTO test_tab VALUES (9999, 'foobar'); + SAVEPOINT sp_inner; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + ROLLBACK TO SAVEPOINT sp_inner; + PREPARE TRANSACTION 'outer'; + "); + + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); + + # check the transaction state prepared on subscriber(s) + $result = + $node_B->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber B'); + $result = + $node_C->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber C'); + + # 2PC COMMIT + $node_A->safe_psql('postgres', "COMMIT PREPARED 'outer';"); + + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); + + # check the transaction state is ended on subscriber + $result = + $node_B->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is ended on subscriber B'); + $result = + $node_C->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is ended on subscriber C'); + + # check inserts are visible at subscriber(s). + # All the streamed data (prior to the SAVEPOINT) should be rolled back. + # (9999, 'foobar') should be committed. + $result = $node_B->safe_psql('postgres', + "SELECT count(*) FROM test_tab where b = 'foobar';"); + is($result, qq(1), 'Rows committed are present on subscriber B'); + $result = + $node_B->safe_psql('postgres', "SELECT count(*) FROM test_tab;"); + is($result, qq(3), 'Rows committed are present on subscriber B'); + $result = $node_C->safe_psql('postgres', + "SELECT count(*) FROM test_tab where b = 'foobar';"); + is($result, qq(1), 'Rows committed are present on subscriber C'); + $result = + $node_C->safe_psql('postgres', "SELECT count(*) FROM test_tab;"); + is($result, qq(3), 'Rows committed are present on subscriber C'); + + # Cleanup the test data + $node_A->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); +} + ############################### # Setup a cascade of pub/sub nodes. # node_A -> node_B -> node_C @@ -260,160 +462,15 @@ is($result, qq(21), 'Rows committed are present on subscriber C'); # 2PC + STREAMING TESTS # --------------------- -my $oldpid_B = $node_A->safe_psql( - 'postgres', " - SELECT pid FROM pg_stat_replication - WHERE application_name = '$appname_B' AND state = 'streaming';"); -my $oldpid_C = $node_B->safe_psql( - 'postgres', " - SELECT pid FROM pg_stat_replication - WHERE application_name = '$appname_C' AND state = 'streaming';"); - -# Setup logical replication (streaming = on) - -$node_B->safe_psql( - 'postgres', " - ALTER SUBSCRIPTION tap_sub_B - SET (streaming = on);"); -$node_C->safe_psql( - 'postgres', " - ALTER SUBSCRIPTION tap_sub_C - SET (streaming = on)"); - -# Wait for subscribers to finish initialization - -$node_A->poll_query_until( - 'postgres', " - SELECT pid != $oldpid_B FROM pg_stat_replication - WHERE application_name = '$appname_B' AND state = 'streaming';" -) or die "Timed out while waiting for apply to restart"; -$node_B->poll_query_until( - 'postgres', " - SELECT pid != $oldpid_C FROM pg_stat_replication - WHERE application_name = '$appname_C' AND state = 'streaming';" -) or die "Timed out while waiting for apply to restart"; - -############################### -# Test 2PC PREPARE / COMMIT PREPARED. -# 1. Data is streamed as a 2PC transaction. -# 2. Then do commit prepared. -# -# Expect all data is replicated on subscriber(s) after the commit. -############################### - -# Insert, update and delete enough rows to exceed the 64kB limit. -# Then 2PC PREPARE -$node_A->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_A->wait_for_catchup($appname_B); -$node_B->wait_for_catchup($appname_C); +################################ +# Test using streaming mode 'on' +################################ +test_streaming($node_A, $node_B, $node_C, $appname_B, $appname_C, 'on'); -# check the transaction state is prepared on subscriber(s) -$result = - $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber B'); -$result = - $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber C'); - -# 2PC COMMIT -$node_A->safe_psql('postgres', "COMMIT PREPARED 'test_prepared_tab';"); - -$node_A->wait_for_catchup($appname_B); -$node_B->wait_for_catchup($appname_C); - -# check that transaction was committed on subscriber(s) -$result = $node_B->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), - 'Rows inserted by 2PC have committed on subscriber B, and extra columns have local defaults' -); -$result = $node_C->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), - 'Rows inserted by 2PC have committed on subscriber C, and extra columns have local defaults' -); - -# check the transaction state is ended on subscriber(s) -$result = - $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is committed on subscriber B'); -$result = - $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is committed on subscriber C'); - -############################### -# Test 2PC PREPARE with a nested ROLLBACK TO SAVEPOINT. -# 0. Cleanup from previous test leaving only 2 rows. -# 1. Insert one more row. -# 2. Record a SAVEPOINT. -# 3. Data is streamed using 2PC. -# 4. Do rollback to SAVEPOINT prior to the streamed inserts. -# 5. Then COMMIT PREPARED. -# -# Expect data after the SAVEPOINT is aborted leaving only 3 rows (= 2 original + 1 from step 1). -############################### - -# First, delete the data except for 2 rows (delete will be replicated) -$node_A->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); - -# 2PC PREPARE with a nested ROLLBACK TO SAVEPOINT -$node_A->safe_psql( - 'postgres', " - BEGIN; - INSERT INTO test_tab VALUES (9999, 'foobar'); - SAVEPOINT sp_inner; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - ROLLBACK TO SAVEPOINT sp_inner; - PREPARE TRANSACTION 'outer'; - "); - -$node_A->wait_for_catchup($appname_B); -$node_B->wait_for_catchup($appname_C); - -# check the transaction state prepared on subscriber(s) -$result = - $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber B'); -$result = - $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber C'); - -# 2PC COMMIT -$node_A->safe_psql('postgres', "COMMIT PREPARED 'outer';"); - -$node_A->wait_for_catchup($appname_B); -$node_B->wait_for_catchup($appname_C); - -# check the transaction state is ended on subscriber -$result = - $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is ended on subscriber B'); -$result = - $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is ended on subscriber C'); - -# check inserts are visible at subscriber(s). -# All the streamed data (prior to the SAVEPOINT) should be rolled back. -# (9999, 'foobar') should be committed. -$result = $node_B->safe_psql('postgres', - "SELECT count(*) FROM test_tab where b = 'foobar';"); -is($result, qq(1), 'Rows committed are present on subscriber B'); -$result = $node_B->safe_psql('postgres', "SELECT count(*) FROM test_tab;"); -is($result, qq(3), 'Rows committed are present on subscriber B'); -$result = $node_C->safe_psql('postgres', - "SELECT count(*) FROM test_tab where b = 'foobar';"); -is($result, qq(1), 'Rows committed are present on subscriber C'); -$result = $node_C->safe_psql('postgres', "SELECT count(*) FROM test_tab;"); -is($result, qq(3), 'Rows committed are present on subscriber C'); +###################################### +# Test using streaming mode 'parallel' +###################################### +test_streaming($node_A, $node_B, $node_C, $appname_B, $appname_C, 'parallel'); ############################### # check all the cleanup diff --git a/src/test/subscription/t/023_twophase_stream.pl b/src/test/subscription/t/023_twophase_stream.pl index d8475d25a4..b89414ab74 100644 --- a/src/test/subscription/t/023_twophase_stream.pl +++ b/src/test/subscription/t/023_twophase_stream.pl @@ -8,6 +8,266 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" +# parameter so the same code can be run both for the streaming=on and +# streaming=parallel cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_parallel) = @_; + + ############################### + # Test 2PC PREPARE / COMMIT PREPARED. + # 1. Data is streamed as a 2PC transaction. + # 2. Then do commit prepared. + # + # Expect all data is replicated on subscriber side after the commit. + ############################### + + # Check that a background worker starts if "streaming" parameter is + # specified as "parallel". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_parallel) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + # check that 2PC gets replicated to subscriber + # Insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + if ($is_parallel) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is in prepared state on subscriber + my $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber'); + + # 2PC transaction gets committed + $node_publisher->safe_psql('postgres', + "COMMIT PREPARED 'test_prepared_tab';"); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is committed on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), + 'Rows inserted by 2PC have committed on subscriber, and extra columns contain local defaults' + ); + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is committed on subscriber'); + + ############################### + # Test 2PC PREPARE / ROLLBACK PREPARED. + # 1. Table is deleted back to 2 rows which are replicated on subscriber. + # 2. Data is streamed using 2PC. + # 3. Do rollback prepared. + # + # Expect data rolls back leaving only the original 2 rows. + ############################### + + # First, delete the data except for 2 rows (will be replicated) + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE a > 2;"); + + # Then insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is in prepared state on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber'); + + # 2PC transaction gets aborted + $node_publisher->safe_psql('postgres', + "ROLLBACK PREPARED 'test_prepared_tab';"); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is aborted on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(2|2|2), + 'Rows inserted by 2PC are rolled back, leaving only the original 2 rows' + ); + + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is aborted on subscriber'); + + ############################### + # Check that 2PC COMMIT PREPARED is decoded properly on crash restart. + # 1. insert, update and delete enough rows to exceed the 64kB limit. + # 2. Then server crashes before the 2PC transaction is committed. + # 3. After servers are restarted the pending transaction is committed. + # + # Expect all data is replicated on subscriber side after the commit. + # Note: both publisher and subscriber do crash/restart. + ############################### + + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + $node_subscriber->stop('immediate'); + $node_publisher->stop('immediate'); + + $node_publisher->start; + $node_subscriber->start; + + # commit post the restart + $node_publisher->safe_psql('postgres', + "COMMIT PREPARED 'test_prepared_tab';"); + $node_publisher->wait_for_catchup($appname); + + # check inserts are visible + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), + 'Rows inserted by 2PC have committed on subscriber, and extra columns contain local defaults' + ); + + ############################### + # Do INSERT after the PREPARE but before ROLLBACK PREPARED. + # 1. Table is deleted back to 2 rows which are replicated on subscriber. + # 2. Data is streamed using 2PC. + # 3. A single row INSERT is done which is after the PREPARE. + # 4. Then do a ROLLBACK PREPARED. + # + # Expect the 2PC data rolls back leaving only 3 rows on the subscriber + # (the original 2 + inserted 1). + ############################### + + # First, delete the data except for 2 rows (will be replicated) + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE a > 2;"); + + # Then insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is in prepared state on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber'); + + # Insert a different record (now we are outside of the 2PC transaction) + # Note: the 2PC transaction still holds row locks so make sure this insert is for a separate primary key + $node_publisher->safe_psql('postgres', + "INSERT INTO test_tab VALUES (99999, 'foobar')"); + + # 2PC transaction gets aborted + $node_publisher->safe_psql('postgres', + "ROLLBACK PREPARED 'test_prepared_tab';"); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is aborted on subscriber, + # but the extra INSERT outside of the 2PC still was replicated + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3|3|3), + 'check the outside insert was copied to subscriber'); + + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is aborted on subscriber'); + + ############################### + # Do INSERT after the PREPARE but before COMMIT PREPARED. + # 1. Table is deleted back to 2 rows which are replicated on subscriber. + # 2. Data is streamed using 2PC. + # 3. A single row INSERT is done which is after the PREPARE. + # 4. Then do a COMMIT PREPARED. + # + # Expect 2PC data + the extra row are on the subscriber + # (the 3334 + inserted 1 = 3335). + ############################### + + # First, delete the data except for 2 rows (will be replicated) + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE a > 2;"); + + # Then insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is in prepared state on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber'); + + # Insert a different record (now we are outside of the 2PC transaction) + # Note: the 2PC transaction still holds row locks so make sure this insert is for a separare primary key + $node_publisher->safe_psql('postgres', + "INSERT INTO test_tab VALUES (99999, 'foobar')"); + + # 2PC transaction gets committed + $node_publisher->safe_psql('postgres', + "COMMIT PREPARED 'test_prepared_tab';"); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is committed on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3335|3335|3335), + 'Rows inserted by 2PC (as well as outside insert) have committed on subscriber, and extra columns contain local defaults' + ); + + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is committed on subscriber'); + + # Cleanup the test data + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE a > 2;"); + $node_publisher->wait_for_catchup($appname); +} + ############################### # Setup ############################### @@ -48,6 +308,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql( 'postgres', " CREATE SUBSCRIPTION tap_sub @@ -70,236 +334,30 @@ my $twophase_query = $node_subscriber->poll_query_until('postgres', $twophase_query) or die "Timed out while waiting for subscriber to enable twophase"; -############################### # Check initial data was copied to subscriber -############################### my $result = $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab"); is($result, qq(2|2|2), 'check initial data was copied to subscriber'); -############################### -# Test 2PC PREPARE / COMMIT PREPARED. -# 1. Data is streamed as a 2PC transaction. -# 2. Then do commit prepared. -# -# Expect all data is replicated on subscriber side after the commit. -############################### - -# check that 2PC gets replicated to subscriber -# Insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is in prepared state on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber'); - -# 2PC transaction gets committed -$node_publisher->safe_psql('postgres', - "COMMIT PREPARED 'test_prepared_tab';"); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is committed on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), - 'Rows inserted by 2PC have committed on subscriber, and extra columns contain local defaults' -); -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is committed on subscriber'); - -############################### -# Test 2PC PREPARE / ROLLBACK PREPARED. -# 1. Table is deleted back to 2 rows which are replicated on subscriber. -# 2. Data is streamed using 2PC. -# 3. Do rollback prepared. -# -# Expect data rolls back leaving only the original 2 rows. -############################### - -# First, delete the data except for 2 rows (will be replicated) -$node_publisher->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); - -# Then insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is in prepared state on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber'); - -# 2PC transaction gets aborted -$node_publisher->safe_psql('postgres', - "ROLLBACK PREPARED 'test_prepared_tab';"); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is aborted on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(2|2|2), - 'Rows inserted by 2PC are rolled back, leaving only the original 2 rows'); - -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is aborted on subscriber'); - -############################### -# Check that 2PC COMMIT PREPARED is decoded properly on crash restart. -# 1. insert, update and delete enough rows to exceed the 64kB limit. -# 2. Then server crashes before the 2PC transaction is committed. -# 3. After servers are restarted the pending transaction is committed. -# -# Expect all data is replicated on subscriber side after the commit. -# Note: both publisher and subscriber do crash/restart. -############################### - -$node_publisher->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_subscriber->stop('immediate'); -$node_publisher->stop('immediate'); - -$node_publisher->start; -$node_subscriber->start; - -# commit post the restart -$node_publisher->safe_psql('postgres', - "COMMIT PREPARED 'test_prepared_tab';"); -$node_publisher->wait_for_catchup($appname); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -# check inserts are visible -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), - 'Rows inserted by 2PC have committed on subscriber, and extra columns contain local defaults' +###################################### +# Test using streaming mode 'parallel' +###################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" ); -############################### -# Do INSERT after the PREPARE but before ROLLBACK PREPARED. -# 1. Table is deleted back to 2 rows which are replicated on subscriber. -# 2. Data is streamed using 2PC. -# 3. A single row INSERT is done which is after the PREPARE. -# 4. Then do a ROLLBACK PREPARED. -# -# Expect the 2PC data rolls back leaving only 3 rows on the subscriber -# (the original 2 + inserted 1). -############################### - -# First, delete the data except for 2 rows (will be replicated) -$node_publisher->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); - -# Then insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is in prepared state on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber'); - -# Insert a different record (now we are outside of the 2PC transaction) -# Note: the 2PC transaction still holds row locks so make sure this insert is for a separate primary key -$node_publisher->safe_psql('postgres', - "INSERT INTO test_tab VALUES (99999, 'foobar')"); - -# 2PC transaction gets aborted -$node_publisher->safe_psql('postgres', - "ROLLBACK PREPARED 'test_prepared_tab';"); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is aborted on subscriber, -# but the extra INSERT outside of the 2PC still was replicated -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3|3|3), 'check the outside insert was copied to subscriber'); - -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is aborted on subscriber'); - -############################### -# Do INSERT after the PREPARE but before COMMIT PREPARED. -# 1. Table is deleted back to 2 rows which are replicated on subscriber. -# 2. Data is streamed using 2PC. -# 3. A single row INSERT is done which is after the PREPARE. -# 4. Then do a COMMIT PREPARED. -# -# Expect 2PC data + the extra row are on the subscriber -# (the 3334 + inserted 1 = 3335). -############################### - -# First, delete the data except for 2 rows (will be replicated) -$node_publisher->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); - -# Then insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is in prepared state on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber'); - -# Insert a different record (now we are outside of the 2PC transaction) -# Note: the 2PC transaction still holds row locks so make sure this insert is for a separare primary key -$node_publisher->safe_psql('postgres', - "INSERT INTO test_tab VALUES (99999, 'foobar')"); - -# 2PC transaction gets committed -$node_publisher->safe_psql('postgres', - "COMMIT PREPARED 'test_prepared_tab';"); - -$node_publisher->wait_for_catchup($appname); +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = parallel)"); -# check that transaction is committed on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3335|3335|3335), - 'Rows inserted by 2PC (as well as outside insert) have committed on subscriber, and extra columns contain local defaults' -); +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is committed on subscriber'); +test_streaming($node_publisher, $node_subscriber, $appname, 1); ############################### # check all the cleanup -- 2.23.0.windows.1 [application/octet-stream] v14-0003-Add-some-checks-before-using-apply-background-wo.patch (35.7K, ../../OS3PR01MB62758DBE8FA12BA72A43AC819EB89@OS3PR01MB6275.jpnprd01.prod.outlook.com/4-v14-0003-Add-some-checks-before-using-apply-background-wo.patch) download | inline diff: From e565cef6206d75b9183194c36b071ff18a221801 Mon Sep 17 00:00:00 2001 From: wangw <[email protected]> Date: Tue, 14 Jun 2022 11:23:52 +0800 Subject: [PATCH v14 3/4] Add some checks before using apply background worker to apply changes. If any of the following checks are violated, an error will be reported. 1. The unique columns between publisher and subscriber are difference. 2. There is any non-immutable function present in expression in subscriber's relation. Check from the following 4 items: a. The function in triggers; b. Column default value expressions and domain constraints; c. Constraint expressions. d. The foreign keys. --- doc/src/sgml/ref/create_subscription.sgml | 4 + .../replication/logical/applybgworker.c | 44 ++ src/backend/replication/logical/proto.c | 62 ++- src/backend/replication/logical/relation.c | 190 +++++++++ src/backend/replication/logical/tablesync.c | 1 + src/backend/replication/logical/worker.c | 23 +- src/backend/utils/cache/typcache.c | 17 + src/include/replication/logicalproto.h | 1 + src/include/replication/logicalrelation.h | 16 + src/include/replication/worker_internal.h | 1 + src/include/utils/typcache.h | 2 + .../subscription/t/022_twophase_cascade.pl | 7 + .../subscription/t/032_streaming_apply.pl | 391 ++++++++++++++++++ 13 files changed, 751 insertions(+), 8 deletions(-) create mode 100644 src/test/subscription/t/032_streaming_apply.pl diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml index 7cf0d2a977..e8be32503b 100644 --- a/doc/src/sgml/ref/create_subscription.sgml +++ b/doc/src/sgml/ref/create_subscription.sgml @@ -240,6 +240,10 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl committed. Note that if an error happens when applying changes in a background worker, the finish LSN of the remote transaction might not be reported in the server log. + To run in this mode, there are following two requirements. The first + is that the unique column should be the same between publisher and + subscriber; the second is that there should not be any non-immutable + function in subscriber-side replicated table. </para> </listitem> </varlistentry> diff --git a/src/backend/replication/logical/applybgworker.c b/src/backend/replication/logical/applybgworker.c index dd70d333e0..e6a596e87e 100644 --- a/src/backend/replication/logical/applybgworker.c +++ b/src/backend/replication/logical/applybgworker.c @@ -773,3 +773,47 @@ apply_bgworker_subxact_info_add(TransactionId current_xid) MemoryContextSwitchTo(oldctx); } } + +/* + * Check if changes on this logical replication relation can be applied by + * apply background worker. + * + * Although we maintains the commit order by allowing only one process to + * commit at a time, our access order to the relation has changed. + * This could cause unexpected problems if the unique column on the replicated + * table is inconsistent with the publisher-side or contains non-immutable + * functions when applying transactions in the apply background worker. + */ +void +apply_bgworker_relation_check(LogicalRepRelMapEntry *rel) +{ + /* Check only we are in apply bgworker. */ + if (!am_apply_bgworker()) + return; + + /* + * If it is a partitioned table, we do not check it, we will check its + * partition later. + */ + if (rel->localrel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + return; + + /* + * If any unique index exist, check that they are same as remoterel. + */ + if (!rel->sameunique) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot replicate relation with different unique index"), + errhint("Please change the streaming option to 'on' instead of 'parallel'."))); + + /* + * Check if there is any non-immutable function present in expression in + * this relation. + */ + if (rel->volatility == FUNCTION_NONIMMUTABLE) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot replicate relation. There is at least one non-immutable function"), + errhint("Please change the streaming option to 'on' instead of 'parallel'."))); +} diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c index b3718f0829..9b31dcec07 100644 --- a/src/backend/replication/logical/proto.c +++ b/src/backend/replication/logical/proto.c @@ -23,7 +23,8 @@ /* * Protocol message flags. */ -#define LOGICALREP_IS_REPLICA_IDENTITY 1 +#define LOGICALREP_IS_REPLICA_IDENTITY 0x0001 +#define LOGICALREP_IS_UNIQUE 0x0002 #define MESSAGE_TRANSACTIONAL (1<<0) #define TRUNCATE_CASCADE (1<<0) @@ -933,11 +934,55 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns) TupleDesc desc; int i; uint16 nliveatts = 0; - Bitmapset *idattrs = NULL; + Bitmapset *idattrs = NULL, + *attunique = NULL; bool replidentfull; desc = RelationGetDescr(rel); + if (rel->rd_rel->relhasindex) + { + List *indexoidlist = RelationGetIndexList(rel); + ListCell *indexoidscan; + + foreach(indexoidscan, indexoidlist) + { + Oid indexoid = lfirst_oid(indexoidscan); + Relation indexRel; + + /* Look up the description for index */ + indexRel = RelationIdGetRelation(indexoid); + + if (!RelationIsValid(indexRel)) + elog(ERROR, "could not open relation with OID %u", indexoid); + + if (indexRel->rd_index->indisunique) + { + int i; + + /* Add referenced attributes to idindexattrs */ + for (i = 0; i < indexRel->rd_index->indnatts; i++) + { + int attrnum = indexRel->rd_index->indkey.values[i]; + + /* + * We don't include non-key columns into idindexattrs + * bitmaps. See RelationGetIndexAttrBitmap. + */ + if (attrnum != 0) + { + if (i < indexRel->rd_index->indnkeyatts && + !bms_is_member(attrnum - FirstLowInvalidHeapAttributeNumber, attunique)) + attunique = bms_add_member(attunique, + attrnum - FirstLowInvalidHeapAttributeNumber); + } + } + } + RelationClose(indexRel); + } + list_free(indexoidlist); + } + /* send number of live attributes */ for (i = 0; i < desc->natts; i++) { @@ -976,6 +1021,10 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns) idattrs)) flags |= LOGICALREP_IS_REPLICA_IDENTITY; + if (bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber, + attunique)) + flags |= LOGICALREP_IS_UNIQUE; + pq_sendbyte(out, flags); /* attribute name */ @@ -1001,7 +1050,8 @@ logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel) int natts; char **attnames; Oid *atttyps; - Bitmapset *attkeys = NULL; + Bitmapset *attkeys = NULL, + *attunique = NULL; natts = pq_getmsgint(in, 2); attnames = palloc(natts * sizeof(char *)); @@ -1012,11 +1062,14 @@ logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel) { uint8 flags; - /* Check for replica identity column */ + /* Check for replica identity and unique column */ flags = pq_getmsgbyte(in); if (flags & LOGICALREP_IS_REPLICA_IDENTITY) attkeys = bms_add_member(attkeys, i); + if (flags & LOGICALREP_IS_UNIQUE) + attunique = bms_add_member(attunique, i); + /* attribute name */ attnames[i] = pstrdup(pq_getmsgstring(in)); @@ -1030,6 +1083,7 @@ logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel) rel->attnames = attnames; rel->atttyps = atttyps; rel->attkeys = attkeys; + rel->attunique = attunique; rel->natts = natts; } diff --git a/src/backend/replication/logical/relation.c b/src/backend/replication/logical/relation.c index e989047681..8fb34640da 100644 --- a/src/backend/replication/logical/relation.c +++ b/src/backend/replication/logical/relation.c @@ -19,12 +19,19 @@ #include "access/table.h" #include "catalog/namespace.h" +#include "catalog/pg_proc.h" #include "catalog/pg_subscription_rel.h" +#include "commands/trigger.h" #include "executor/executor.h" #include "nodes/makefuncs.h" +#include "optimizer/optimizer.h" #include "replication/logicalrelation.h" #include "replication/worker_internal.h" +#include "rewrite/rewriteHandler.h" #include "utils/inval.h" +#include "utils/lsyscache.h" +#include "utils/syscache.h" +#include "utils/typcache.h" static MemoryContext LogicalRepRelMapContext = NULL; @@ -91,6 +98,23 @@ logicalrep_relmap_invalidate_cb(Datum arg, Oid reloid) } } +/* + * Reset the flag volatility of all existing entry in the relation map cache. + */ +static void +logicalrep_relmap_reset_volatility_cb(Datum arg, int cacheid, uint32 hashvalue) +{ + HASH_SEQ_STATUS hash_seq; + LogicalRepRelMapEntry *entry; + + if (LogicalRepRelMap == NULL) + return; + + hash_seq_init(&hash_seq, LogicalRepRelMap); + while ((entry = hash_seq_search(&hash_seq)) != NULL) + entry->volatility = FUNCTION_UNKNOWN; +} + /* * Initialize the relation map cache. */ @@ -116,6 +140,9 @@ logicalrep_relmap_init(void) /* Watch for invalidation events. */ CacheRegisterRelcacheCallback(logicalrep_relmap_invalidate_cb, (Datum) 0); + CacheRegisterSyscacheCallback(PROCOID, + logicalrep_relmap_reset_volatility_cb, + (Datum) 0); } /* @@ -142,6 +169,7 @@ logicalrep_relmap_free_entry(LogicalRepRelMapEntry *entry) pfree(remoterel->atttyps); } bms_free(remoterel->attkeys); + bms_free(remoterel->attunique); if (entry->attrmap) free_attrmap(entry->attrmap); @@ -190,6 +218,7 @@ logicalrep_relmap_update(LogicalRepRelation *remoterel) } entry->remoterel.replident = remoterel->replident; entry->remoterel.attkeys = bms_copy(remoterel->attkeys); + entry->remoterel.attunique = bms_copy(remoterel->attunique); MemoryContextSwitchTo(oldctx); } @@ -310,6 +339,160 @@ logicalrep_rel_mark_updatable(LogicalRepRelMapEntry *entry) } } +/* + * Check if unique index/constraint matches and mark sameunique and volatility + * flag. + * + * Don't throw any error here just mark the relation entry as not sameunique or + * FUNCTION_NONIMMUTABLE as we only check these in apply background worker. + */ +static void +logicalrep_rel_mark_safe_in_apply_bgworker(LogicalRepRelMapEntry *entry) +{ + Bitmapset *ukey; + int i; + TupleDesc tupdesc; + int attnum; + List *fkeys = NIL; + + /* + * Check that the unique column in the relation on the subscriber-side is + * also the unique column on the publisher-side. + */ + entry->sameunique = true; + ukey = RelationGetIndexAttrBitmap(entry->localrel, + INDEX_ATTR_BITMAP_KEY); + + if (ukey) + { + i = -1; + while ((i = bms_next_member(ukey, i)) >= 0) + { + attnum = AttrNumberGetAttrOffset(i + FirstLowInvalidHeapAttributeNumber); + + if (entry->attrmap->attnums[attnum] < 0 || + !bms_is_member(entry->attrmap->attnums[attnum], entry->remoterel.attunique)) + { + entry->sameunique = false; + break; + } + } + } + + /* + * Check whether there is any non-immutable function in the local table. + * + * a. The function in triggers; + * b. Column default value expressions and domain constraints; + * c. Constraint expressions; + * d. Foreign keys. + */ + if (entry->volatility != FUNCTION_UNKNOWN) + return; + + /* Initialize the flag. */ + entry->volatility = FUNCTION_IMMUTABLE; + + /* Check the trigger functions. */ + if (entry->localrel->trigdesc != NULL) + { + for (i = 0; i < entry->localrel->trigdesc->numtriggers; i++) + { + Trigger *trig = entry->localrel->trigdesc->triggers + i; + + if (trig->tgenabled != TRIGGER_FIRES_ALWAYS && + trig->tgenabled != TRIGGER_FIRES_ON_REPLICA) + continue; + + if (func_volatile(trig->tgfoid) != PROVOLATILE_IMMUTABLE) + { + entry->volatility = FUNCTION_NONIMMUTABLE; + return; + } + } + } + + /* Check the columns. */ + tupdesc = RelationGetDescr(entry->localrel); + for (attnum = 0; attnum < tupdesc->natts; attnum++) + { + Form_pg_attribute att = TupleDescAttr(tupdesc, attnum); + + /* We don't need info for dropped or generated attributes */ + if (att->attisdropped || att->attgenerated) + continue; + + /* + * We don't need to check columns that only exist on the + * subscriber + */ + if (entry->attrmap->attnums[attnum] < 0) + continue; + + if (att->atthasdef) + { + Node *defaultexpr; + + defaultexpr = build_column_default(entry->localrel, attnum + 1); + if (contain_mutable_functions(defaultexpr)) + { + entry->volatility = FUNCTION_NONIMMUTABLE; + return; + } + } + + /* + * If the column is of a DOMAIN type, determine whether + * that domain has any CHECK expressions that are not + * immutable. + */ + if (get_typtype(att->atttypid) == TYPTYPE_DOMAIN) + { + List *domain_constraints; + ListCell *lc; + + domain_constraints = GetDomainConstraints(att->atttypid); + + foreach(lc, domain_constraints) + { + DomainConstraintState *con = (DomainConstraintState *) lfirst(lc); + + if (con->check_expr && contain_mutable_functions((Node *) con->check_expr)) + { + entry->volatility = FUNCTION_NONIMMUTABLE; + return; + } + } + } + } + + /* Check the constraints. */ + if (tupdesc->constr) + { + ConstrCheck *check = tupdesc->constr->check; + + /* + * Determine if there are any CHECK constraints which + * contains non-immutable function. + */ + for (i = 0; i < tupdesc->constr->num_check; i++) + { + Expr *check_expr = stringToNode(check[i].ccbin); + + if (contain_mutable_functions((Node *) check_expr)) + { + entry->volatility = FUNCTION_NONIMMUTABLE; + return; + } + } + } + + /* Check the foreign keys. */ + fkeys = RelationGetFKeyList(entry->localrel); + if (fkeys) + entry->volatility = FUNCTION_NONIMMUTABLE; +} + /* * Open the local relation associated with the remote one. * @@ -438,6 +621,9 @@ logicalrep_rel_open(LogicalRepRelId remoteid, LOCKMODE lockmode) */ logicalrep_rel_mark_updatable(entry); + /* Set if changes could be applied in the apply background worker. */ + logicalrep_rel_mark_safe_in_apply_bgworker(entry); + entry->localrelvalid = true; } @@ -653,6 +839,7 @@ logicalrep_partition_open(LogicalRepRelMapEntry *root, } entry->remoterel.replident = remoterel->replident; entry->remoterel.attkeys = bms_copy(remoterel->attkeys); + entry->remoterel.attunique = bms_copy(remoterel->attunique); } entry->localrel = partrel; @@ -696,6 +883,9 @@ logicalrep_partition_open(LogicalRepRelMapEntry *root, /* Set if the table's replica identity is enough to apply update/delete. */ logicalrep_rel_mark_updatable(entry); + /* Set if changes could be applied in the apply background worker. */ + logicalrep_rel_mark_safe_in_apply_bgworker(entry); + entry->localrelvalid = true; /* state and statelsn are left set to 0. */ diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c index 8ffba7e2e5..3cdbf8b457 100644 --- a/src/backend/replication/logical/tablesync.c +++ b/src/backend/replication/logical/tablesync.c @@ -884,6 +884,7 @@ fetch_remote_table_info(char *nspname, char *relname, lrel->attnames = palloc0(MaxTupleAttributeNumber * sizeof(char *)); lrel->atttyps = palloc0(MaxTupleAttributeNumber * sizeof(Oid)); lrel->attkeys = NULL; + lrel->attunique = NULL; /* * Store the columns as a list of names. Ignore those that are not diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c index 84938a87be..8db43246e4 100644 --- a/src/backend/replication/logical/worker.c +++ b/src/backend/replication/logical/worker.c @@ -1390,6 +1390,14 @@ apply_handle_stream_stop(StringInfo s) { char action = LOGICAL_REP_MSG_STREAM_STOP; + /* + * Unlike stream_commit, we don't need to wait here for stream_stop to + * finish. Allowing the other transaction to be applied before stream_stop + * is finished can only lead to failures if the unique index/constraint is + * different between publisher and subscriber. But for such cases, we don't + * allow streamed transactions to be applied in parallel. See + * apply_bgworker_relation_check. + */ apply_bgworker_send_data(stream_apply_worker, 1, &action); elog(DEBUG1, "stopped streaming of xid %u, %u changes streamed", stream_xid, nchanges); @@ -2038,6 +2046,8 @@ apply_handle_insert(StringInfo s) /* Set relation for error callback */ apply_error_callback_arg.rel = rel; + apply_bgworker_relation_check(rel); + /* Initialize the executor state. */ edata = create_edata_for_relation(rel); estate = edata->estate; @@ -2181,6 +2191,8 @@ apply_handle_update(StringInfo s) /* Check if we can do the update. */ check_relation_updatable(rel); + apply_bgworker_relation_check(rel); + /* Initialize the executor state. */ edata = create_edata_for_relation(rel); estate = edata->estate; @@ -2349,6 +2361,8 @@ apply_handle_delete(StringInfo s) /* Check if we can do the delete. */ check_relation_updatable(rel); + apply_bgworker_relation_check(rel); + /* Initialize the executor state. */ edata = create_edata_for_relation(rel); estate = edata->estate; @@ -2534,13 +2548,14 @@ apply_handle_tuple_routing(ApplyExecutionData *edata, } MemoryContextSwitchTo(oldctx); + part_entry = logicalrep_partition_open(relmapentry, partrel, + attrmap); + + apply_bgworker_relation_check(part_entry); + /* Check if we can do the update or delete on the leaf partition. */ if (operation == CMD_UPDATE || operation == CMD_DELETE) - { - part_entry = logicalrep_partition_open(relmapentry, partrel, - attrmap); check_relation_updatable(part_entry); - } switch (operation) { diff --git a/src/backend/utils/cache/typcache.c b/src/backend/utils/cache/typcache.c index 808f9ebd0d..b248899d82 100644 --- a/src/backend/utils/cache/typcache.c +++ b/src/backend/utils/cache/typcache.c @@ -2540,6 +2540,23 @@ compare_values_of_enum(TypeCacheEntry *tcache, Oid arg1, Oid arg2) return 0; } +/* + * GetDomainConstraints --- get DomainConstraintState list of specified domain type + */ +List * +GetDomainConstraints(Oid type_id) +{ + TypeCacheEntry *typentry; + List *constraints = NIL; + + typentry = lookup_type_cache(type_id, TYPECACHE_DOMAIN_CONSTR_INFO); + + if(typentry->domainData != NULL) + constraints = typentry->domainData->constraints; + + return constraints; +} + /* * Load (or re-load) the enumData member of the typcache entry. */ diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h index 7bba77c9e7..a503eb62c2 100644 --- a/src/include/replication/logicalproto.h +++ b/src/include/replication/logicalproto.h @@ -108,6 +108,7 @@ typedef struct LogicalRepRelation char replident; /* replica identity */ char relkind; /* remote relation kind */ Bitmapset *attkeys; /* Bitmap of key columns */ + Bitmapset *attunique; /* Bitmap of unique columns */ } LogicalRepRelation; /* Type mapping info */ diff --git a/src/include/replication/logicalrelation.h b/src/include/replication/logicalrelation.h index 78cd7e77f5..4476cf7cec 100644 --- a/src/include/replication/logicalrelation.h +++ b/src/include/replication/logicalrelation.h @@ -15,6 +15,17 @@ #include "access/attmap.h" #include "replication/logicalproto.h" +/* + * States to determine volatility of the function in expressions in one + * relation. + */ +typedef enum RelFuncVolatility +{ + FUNCTION_UNKNOWN = 0, /* initializing */ + FUNCTION_IMMUTABLE, /* all functions are immutable function */ + FUNCTION_NONIMMUTABLE /* at least one non-immutable function */ +} RelFuncVolatility; + typedef struct LogicalRepRelMapEntry { LogicalRepRelation remoterel; /* key is remoterel.remoteid */ @@ -31,6 +42,11 @@ typedef struct LogicalRepRelMapEntry Relation localrel; /* relcache entry (NULL when closed) */ AttrMap *attrmap; /* map of local attributes to remote ones */ bool updatable; /* Can apply updates/deletes? */ + bool sameunique; /* Are all unique columns of the local + relation contained by the unique columns in + remote? */ + RelFuncVolatility volatility; /* all functions in localrel are + immutable function? */ /* Sync state. */ char state; diff --git a/src/include/replication/worker_internal.h b/src/include/replication/worker_internal.h index c2ff5d4144..d79065d70f 100644 --- a/src/include/replication/worker_internal.h +++ b/src/include/replication/worker_internal.h @@ -194,6 +194,7 @@ extern void apply_bgworker_free(ApplyBgworkerState *wstate); extern void apply_bgworker_check_status(void); extern void apply_bgworker_set_status(ApplyBgworkerStatus status); extern void apply_bgworker_subxact_info_add(TransactionId current_xid); +extern void apply_bgworker_relation_check(LogicalRepRelMapEntry *rel); static inline bool am_tablesync_worker(void) diff --git a/src/include/utils/typcache.h b/src/include/utils/typcache.h index 431ad7f1b3..ed7c2e7f48 100644 --- a/src/include/utils/typcache.h +++ b/src/include/utils/typcache.h @@ -199,6 +199,8 @@ extern uint64 assign_record_type_identifier(Oid type_id, int32 typmod); extern int compare_values_of_enum(TypeCacheEntry *tcache, Oid arg1, Oid arg2); +extern List *GetDomainConstraints(Oid type_id); + extern size_t SharedRecordTypmodRegistryEstimate(void); extern void SharedRecordTypmodRegistryInit(SharedRecordTypmodRegistry *, diff --git a/src/test/subscription/t/022_twophase_cascade.pl b/src/test/subscription/t/022_twophase_cascade.pl index 0a4152d3be..4d2e352f94 100644 --- a/src/test/subscription/t/022_twophase_cascade.pl +++ b/src/test/subscription/t/022_twophase_cascade.pl @@ -39,6 +39,13 @@ sub test_streaming ALTER SUBSCRIPTION tap_sub_C SET (streaming = $streaming_mode)"); + if ($streaming_mode eq 'parallel') + { + $node_C->safe_psql( + 'postgres', " + ALTER TABLE test_tab ALTER c DROP DEFAULT"); + } + # Wait for subscribers to finish initialization $node_A->poll_query_until( diff --git a/src/test/subscription/t/032_streaming_apply.pl b/src/test/subscription/t/032_streaming_apply.pl new file mode 100644 index 0000000000..c06f87c255 --- /dev/null +++ b/src/test/subscription/t/032_streaming_apply.pl @@ -0,0 +1,391 @@ +# Copyright (c) 2022, PostgreSQL Global Development Group + +# Test the restrictions of streaming mode "parallel" in logical replication + +use strict; +use warnings; +use PostgreSQL::Test::Cluster; +use PostgreSQL::Test::Utils; +use Test::More; + +my $offset = 0; + +# Create publisher node +my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); +$node_publisher->init(allows_streaming => 'logical'); +$node_publisher->append_conf('postgresql.conf', + 'logical_decoding_work_mem = 64kB'); +$node_publisher->start; + +# Create subscriber node +my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber'); +$node_subscriber->init; +$node_subscriber->start; + +# Setup structure on publisher +$node_publisher->safe_psql('postgres', + "CREATE TABLE test_tab (a int primary key, b varchar)"); +$node_publisher->safe_psql('postgres', + "CREATE TABLE test_tab_partitioned (a int primary key, b varchar)"); + +# Setup structure on subscriber +# We need to test normal table and partition table. +$node_subscriber->safe_psql('postgres', + "CREATE TABLE test_tab (a int primary key, b varchar)"); +$node_subscriber->safe_psql('postgres', + "CREATE TABLE test_tab_partitioned (a int primary key, b varchar) PARTITION BY RANGE(a)"); +$node_subscriber->safe_psql('postgres', + "CREATE TABLE test_tab_partition (LIKE test_tab_partitioned)"); +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partitioned ATTACH PARTITION test_tab_partition DEFAULT" +); + +# Setup logical replication +my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres'; +$node_publisher->safe_psql('postgres', + "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); +$node_publisher->safe_psql('postgres', + "CREATE PUBLICATION tap_pub_partitioned FOR TABLE test_tab_partitioned"); + +my $appname = 'tap_sub'; +$node_subscriber->safe_psql( + 'postgres', " + CREATE SUBSCRIPTION tap_sub + CONNECTION '$publisher_connstr application_name=$appname' + PUBLICATION tap_pub, tap_pub_partitioned + WITH (streaming = parallel, copy_data = false)"); + +$node_publisher->wait_for_catchup($appname); + +# It is not allowed that the unique index on the publisher and the subscriber +# is different. Check the error reported by background worker in this case. +# First we check the unique index on normal table. +$node_subscriber->safe_psql('postgres', + "CREATE UNIQUE INDEX test_tab_b_idx ON test_tab (b)"); + +# Check that a background worker starts if "streaming" option is specified as +# "parallel". We have to look for the DEBUG1 log messages about that, so +# temporarily bump up the log verbosity. +$node_subscriber->append_conf('postgresql.conf', "log_min_messages = debug1"); +$node_subscriber->reload; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); +$node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); +$node_subscriber->reload; + +$node_subscriber->wait_for_log( + qr/ERROR: cannot replicate relation with different unique index/, + $offset); + +# Drop the unique index on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', "DROP INDEX test_tab_b_idx"); + +$node_publisher->wait_for_catchup($appname); + +my $result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); +is($result, qq(5000), 'data replicated to subscriber after dropping index'); + +# Then we check the unique index on partition table. +$node_subscriber->safe_psql('postgres', + "CREATE UNIQUE INDEX test_tab_b_partition_idx ON test_tab_partition (b)"); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab_partitioned SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log( + qr/ERROR: cannot replicate relation with different unique index/, + $offset); + +# Drop the unique index on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', "DROP INDEX test_tab_b_partition_idx"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); +is($result, qq(5000), 'data replicated to subscriber after dropping index'); + +# Triggers which execute non-immutable function are not allowed on the +# subscriber side. Check the error reported by background worker in this case. +# First we check the trigger function on normal table. +$node_subscriber->safe_psql( + 'postgres', qq{ +CREATE FUNCTION trigger_func() RETURNS TRIGGER AS \$\$ + BEGIN + RETURN NULL; + END +\$\$ language plpgsql; +CREATE TRIGGER insert_trig +BEFORE INSERT ON test_tab +FOR EACH ROW EXECUTE PROCEDURE trigger_func(); +ALTER TABLE test_tab ENABLE REPLICA TRIGGER insert_trig; +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', "DELETE FROM test_tab"); + +$node_subscriber->wait_for_log( + qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, + $offset); + +# Drop the trigger on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "DROP TRIGGER insert_trig ON test_tab"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); +is($result, qq(0), 'data replicated to subscriber after dropping trigger'); + +# Then we check the unique index on partition table. +$node_subscriber->safe_psql( + 'postgres', qq{ +CREATE TRIGGER insert_trig +BEFORE INSERT ON test_tab_partition +FOR EACH ROW EXECUTE PROCEDURE trigger_func(); +ALTER TABLE test_tab_partition ENABLE REPLICA TRIGGER insert_trig; +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', "DELETE FROM test_tab_partitioned"); + +$node_subscriber->wait_for_log( + qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, + $offset); + +# Drop the trigger on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "DROP TRIGGER insert_trig ON test_tab_partition"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); +is($result, qq(0), 'data replicated to subscriber after dropping trigger'); + +# It is not allowed that column default value expression contains a +# non-immutable function on the subscriber side. Check the error reported by +# background worker in this case. +# First we check the column default value expression on normal table. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab ALTER COLUMN b SET DEFAULT random()"); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log( + qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, + $offset); + +# Drop default value on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab ALTER COLUMN b DROP DEFAULT"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); +is($result, qq(5000), + 'data replicated to subscriber after dropping default value expression'); + +# Then we check the column default value expression on partition table. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partition ALTER COLUMN b SET DEFAULT random()"); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab_partitioned SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log( + qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, + $offset); + +# Drop default value on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partition ALTER COLUMN b DROP DEFAULT"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); +is($result, qq(5000), + 'data replicated to subscriber after dropping default value expression'); + +# It is not allowed that domain constraint expression contains a non-immutable +# function on the subscriber side. Check the error reported by background +# worker in this case. +# Because the column type of the partition table must be the same as its parent +# table, only test normal table here. +$node_subscriber->safe_psql( + 'postgres', qq{ +CREATE DOMAIN test_domain AS int CHECK(VALUE > random()); +ALTER TABLE test_tab ALTER COLUMN a TYPE test_domain; +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', "DELETE FROM test_tab"); + +$node_subscriber->wait_for_log( + qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, + $offset); + +# Drop domain constraint expression on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab ALTER COLUMN a TYPE int"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); +is($result, qq(0), + 'data replicated to subscriber after dropping domain constraint expression' +); + +# It is not allowed that constraint expression contains a non-immutable function +# on the subscriber side. Check the error reported by background worker in this +# case. +# First we check the constraint expression on normal table. +$node_subscriber->safe_psql( + 'postgres', qq{ +ALTER TABLE test_tab ADD CONSTRAINT test_tab_con check (a > random()); +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log( + qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, + $offset); + +# Drop constraint on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab DROP CONSTRAINT test_tab_con"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); +is($result, qq(5000), + 'data replicated to subscriber after dropping constraint expression'); + +# Then we check the constraint expression on partition table. +$node_subscriber->safe_psql( + 'postgres', qq{ +ALTER TABLE test_tab_partition ADD CONSTRAINT test_tab_con check (a > random()); +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', "DELETE FROM test_tab_partitioned"); + +$node_subscriber->wait_for_log( + qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, + $offset); + +# Drop constraint on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partition DROP CONSTRAINT test_tab_con"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); +is($result, qq(0), + 'data replicated to subscriber after dropping constraint expression'); + +# It is not allowed that foreign key on the subscriber side. Check the error +# reported by background worker in this case. +# First we check the foreign key on normal table. +$node_publisher->safe_psql('postgres', "DELETE FROM test_tab"); +$node_publisher->wait_for_catchup($appname); +$node_subscriber->safe_psql( + 'postgres', qq{ +CREATE TABLE test_tab_f (a int primary key); +ALTER TABLE test_tab ADD CONSTRAINT test_tabfk FOREIGN KEY(a) REFERENCES test_tab_f(a); +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log( + qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, + $offset); + +# Drop the foreign key constraint on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab DROP CONSTRAINT test_tabfk"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); +is($result, qq(5000), + 'data replicated to subscriber after dropping the foreign key'); + +# Then we check the foreign key on partition table. +$node_publisher->wait_for_catchup($appname); +$node_subscriber->safe_psql( + 'postgres', qq{ +CREATE TABLE test_tab_partition_f (a int primary key); +ALTER TABLE test_tab_partition ADD CONSTRAINT test_tab_patition_fk FOREIGN KEY(a) REFERENCES test_tab_partition_f(a); +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab_partitioned SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log( + qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, + $offset); + +# Drop the foreign key constraint on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partition DROP CONSTRAINT test_tab_patition_fk"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); +is($result, qq(5000), + 'data replicated to subscriber after dropping the foreign key'); + +$node_subscriber->stop; +$node_publisher->stop; + +done_testing(); -- 2.23.0.windows.1 [application/octet-stream] v14-0004-Retry-to-apply-streaming-xact-only-in-apply-work.patch (25.5K, ../../OS3PR01MB62758DBE8FA12BA72A43AC819EB89@OS3PR01MB6275.jpnprd01.prod.outlook.com/5-v14-0004-Retry-to-apply-streaming-xact-only-in-apply-work.patch) download | inline diff: From 0c2c0564945d8c8e38fff04dc3080988f0c8f8cd Mon Sep 17 00:00:00 2001 From: wangw <[email protected]> Date: Wed, 15 Jun 2022 11:02:08 +0800 Subject: [PATCH v14 4/4] Retry to apply streaming xact only in apply worker. --- doc/src/sgml/catalogs.sgml | 11 ++ doc/src/sgml/ref/create_subscription.sgml | 4 + src/backend/catalog/pg_subscription.c | 1 + src/backend/catalog/system_views.sql | 4 +- src/backend/commands/subscriptioncmds.c | 1 + .../replication/logical/applybgworker.c | 19 ++- src/backend/replication/logical/worker.c | 89 +++++++++++ src/bin/pg_dump/pg_dump.c | 5 +- src/include/catalog/pg_subscription.h | 4 + .../subscription/t/032_streaming_apply.pl | 139 ++++++++++-------- 10 files changed, 209 insertions(+), 68 deletions(-) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index f73fae39b6..3cb3d41a30 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -7907,6 +7907,17 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>subretry</structfield> <type>bool</type> + </para> + <para> + If true, the subscription will not try to apply streaming transaction + in <literal>parallel</literal> mode. See + <xref linkend="sql-createsubscription"/> for more information. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>subconninfo</structfield> <type>text</type> diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml index e8be32503b..af660ffb0f 100644 --- a/doc/src/sgml/ref/create_subscription.sgml +++ b/doc/src/sgml/ref/create_subscription.sgml @@ -244,6 +244,10 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl is that the unique column should be the same between publisher and subscriber; the second is that there should not be any non-immutable function in subscriber-side replicated table. + When applying a streaming transaction, if either requirement is not + met, the background worker will exit with an error. And when retrying + later, we will try to apply this transaction in <literal>on</literal> + mode. </para> </listitem> </varlistentry> diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c index add51caadf..1824390d7d 100644 --- a/src/backend/catalog/pg_subscription.c +++ b/src/backend/catalog/pg_subscription.c @@ -71,6 +71,7 @@ GetSubscription(Oid subid, bool missing_ok) sub->stream = subform->substream; sub->twophasestate = subform->subtwophasestate; sub->disableonerr = subform->subdisableonerr; + sub->retry = subform->subretry; /* Get conninfo */ datum = SysCacheGetAttr(SUBSCRIPTIONOID, diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql index fedaed533b..10f4dd6785 100644 --- a/src/backend/catalog/system_views.sql +++ b/src/backend/catalog/system_views.sql @@ -1298,8 +1298,8 @@ REVOKE ALL ON pg_replication_origin_status FROM public; -- All columns of pg_subscription except subconninfo are publicly readable. REVOKE ALL ON pg_subscription FROM public; GRANT SELECT (oid, subdbid, subskiplsn, subname, subowner, subenabled, - subbinary, substream, subtwophasestate, subdisableonerr, subslotname, - subsynccommit, subpublications) + subbinary, substream, subtwophasestate, subdisableonerr, + subretry, subslotname, subsynccommit, subpublications) ON pg_subscription TO public; CREATE VIEW pg_stat_subscription_stats AS diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c index 467ff11e39..01c52a0fb8 100644 --- a/src/backend/commands/subscriptioncmds.c +++ b/src/backend/commands/subscriptioncmds.c @@ -663,6 +663,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt, LOGICALREP_TWOPHASE_STATE_PENDING : LOGICALREP_TWOPHASE_STATE_DISABLED); values[Anum_pg_subscription_subdisableonerr - 1] = BoolGetDatum(opts.disableonerr); + values[Anum_pg_subscription_subretry - 1] = BoolGetDatum(false); values[Anum_pg_subscription_subconninfo - 1] = CStringGetTextDatum(conninfo); if (opts.slot_name) diff --git a/src/backend/replication/logical/applybgworker.c b/src/backend/replication/logical/applybgworker.c index e6a596e87e..859a169917 100644 --- a/src/backend/replication/logical/applybgworker.c +++ b/src/backend/replication/logical/applybgworker.c @@ -107,6 +107,19 @@ apply_bgworker_can_start(TransactionId xid) if (!XLogRecPtrIsInvalid(MySubscription->skiplsn)) return false; + /* + * We don't start new background worker if retry was set as it's possible + * that the last time we tried to apply a transaction in background worker + * and the check failed (see function apply_bgworker_relation_check). So + * we will try to apply this transaction in apply worker. + */ + if (MySubscription->retry) + { + elog(DEBUG1, "retry to apply an streaming transaction in apply " + "background worker"); + return false; + } + /* * For streaming transactions that are being applied in apply background * worker, we cannot decide whether to apply the change for a relation @@ -804,8 +817,7 @@ apply_bgworker_relation_check(LogicalRepRelMapEntry *rel) if (!rel->sameunique) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("cannot replicate relation with different unique index"), - errhint("Please change the streaming option to 'on' instead of 'parallel'."))); + errmsg("cannot replicate relation with different unique index"))); /* * Check if there is any non-immutable function present in expression in @@ -814,6 +826,5 @@ apply_bgworker_relation_check(LogicalRepRelMapEntry *rel) if (rel->volatility == FUNCTION_NONIMMUTABLE) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("cannot replicate relation. There is at least one non-immutable function"), - errhint("Please change the streaming option to 'on' instead of 'parallel'."))); + errmsg("cannot replicate relation. There is at least one non-immutable function"))); } diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c index 8db43246e4..87de1449c1 100644 --- a/src/backend/replication/logical/worker.c +++ b/src/backend/replication/logical/worker.c @@ -377,6 +377,8 @@ static void clear_subscription_skip_lsn(XLogRecPtr finish_lsn); static inline void set_apply_error_context_xact(TransactionId xid, XLogRecPtr lsn); static inline void reset_apply_error_context_info(void); +static void set_subscription_retry(bool retry); + /* * Should this worker apply changes for given relation. * @@ -904,6 +906,9 @@ apply_handle_commit(StringInfo s) apply_handle_commit_internal(&commit_data); + /* Set the flag that we will not retry later. */ + set_subscription_retry(false); + /* Check the status of apply background worker if any. */ apply_bgworker_check_status(); @@ -1015,6 +1020,9 @@ apply_handle_prepare(StringInfo s) in_remote_transaction = false; + /* Set the flag that we will not retry later. */ + set_subscription_retry(false); + /* Check the status of apply background worker if any. */ apply_bgworker_check_status(); @@ -1068,6 +1076,9 @@ apply_handle_commit_prepared(StringInfo s) store_flush_position(prepare_data.end_lsn); in_remote_transaction = false; + /* Set the flag that we will not retry later. */ + set_subscription_retry(false); + /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(prepare_data.end_lsn); @@ -1123,6 +1134,9 @@ apply_handle_rollback_prepared(StringInfo s) store_flush_position(rollback_data.rollback_end_lsn); in_remote_transaction = false; + /* Set the flag that we will not retry later. */ + set_subscription_retry(false); + /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(rollback_data.rollback_end_lsn); @@ -1215,6 +1229,9 @@ apply_handle_stream_prepare(StringInfo s) /* Unlink the files with serialized changes and subxact info. */ stream_cleanup_files(MyLogicalRepWorker->subid, prepare_data.xid); } + + /* Set the flag that we will not retry later. */ + set_subscription_retry(false); } in_remote_transaction = false; @@ -1641,6 +1658,9 @@ apply_handle_stream_abort(StringInfo s) */ else serialize_stream_abort(xid, subxid); + + /* Set the flag that we will not retry later. */ + set_subscription_retry(false); } reset_apply_error_context_info(); @@ -1852,6 +1872,9 @@ apply_handle_stream_commit(StringInfo s) /* Unlink the files with serialized changes and subxact info. */ stream_cleanup_files(MyLogicalRepWorker->subid, xid); } + + /* Set the flag that we will not retry later. */ + set_subscription_retry(false); } /* Check the status of apply background worker if any. */ @@ -3894,6 +3917,9 @@ start_table_sync(XLogRecPtr *origin_startpos, char **myslotname) } PG_CATCH(); { + /* Set the flag that we will retry later. */ + set_subscription_retry(true); + if (MySubscription->disableonerr) DisableSubscriptionAndExit(); else @@ -3932,6 +3958,9 @@ start_apply(XLogRecPtr origin_startpos) } PG_CATCH(); { + /* Set the flag that we will retry later. */ + set_subscription_retry(true); + if (MySubscription->disableonerr) DisableSubscriptionAndExit(); else @@ -4457,3 +4486,63 @@ reset_apply_error_context_info(void) apply_error_callback_arg.remote_attnum = -1; set_apply_error_context_xact(InvalidTransactionId, InvalidXLogRecPtr); } + +/* + * Set subretry of pg_subscription catalog. + * + * If retry is true, subscriber is about to exit with an error. Otherwise, it + * means that the changes was applied successfully. + */ +static void +set_subscription_retry(bool retry) +{ + Relation rel; + HeapTuple tup; + bool started_tx = false; + bool nulls[Natts_pg_subscription]; + bool replaces[Natts_pg_subscription]; + Datum values[Natts_pg_subscription]; + + if (MySubscription->retry == retry || + am_apply_bgworker()) + return; + + if (!IsTransactionState()) + { + StartTransactionCommand(); + started_tx = true; + } + + /* Look up the subscription in the catalog */ + rel = table_open(SubscriptionRelationId, RowExclusiveLock); + tup = SearchSysCacheCopy1(SUBSCRIPTIONOID, + ObjectIdGetDatum(MySubscription->oid)); + + if (!HeapTupleIsValid(tup)) + elog(ERROR, "subscription \"%s\" does not exist", MySubscription->name); + + LockSharedObject(SubscriptionRelationId, MySubscription->oid, 0, + AccessShareLock); + + /* Form a new tuple. */ + memset(values, 0, sizeof(values)); + memset(nulls, false, sizeof(nulls)); + memset(replaces, false, sizeof(replaces)); + + /* reset subretry */ + values[Anum_pg_subscription_subretry - 1] = BoolGetDatum(retry); + replaces[Anum_pg_subscription_subretry - 1] = true; + + tup = heap_modify_tuple(tup, RelationGetDescr(rel), values, nulls, + replaces); + + /* Update the catalog. */ + CatalogTupleUpdate(rel, &tup->t_self, tup); + + /* Cleanup. */ + heap_freetuple(tup); + table_close(rel, NoLock); + + if (started_tx) + CommitTransactionCommand(); +} diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 53a41b4e3f..fb7ace74d7 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -4472,8 +4472,9 @@ getSubscriptions(Archive *fout) ntups = PQntuples(res); /* - * Get subscription fields. We don't include subskiplsn in the dump as - * after restoring the dump this value may no longer be relevant. + * Get subscription fields. We don't include subskiplsn and subretry in + * the dump as after restoring the dump this value may no longer be + * relevant. */ i_tableoid = PQfnumber(res, "tableoid"); i_oid = PQfnumber(res, "oid"); diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h index 2995e019a8..ce20a46b68 100644 --- a/src/include/catalog/pg_subscription.h +++ b/src/include/catalog/pg_subscription.h @@ -76,6 +76,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW bool subdisableonerr; /* True if a worker error should cause the * subscription to be disabled */ + bool subretry; /* True if the previous apply change failed. */ + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* Connection string to the publisher */ text subconninfo BKI_FORCE_NOT_NULL; @@ -116,6 +118,8 @@ typedef struct Subscription bool disableonerr; /* Indicates if the subscription should be * automatically disabled if a worker error * occurs */ + bool retry; /* Indicates if the previous apply change + * failed. */ char *conninfo; /* Connection string to the publisher */ char *slotname; /* Name of the replication slot */ char *synccommit; /* Synchronous commit setting for worker */ diff --git a/src/test/subscription/t/032_streaming_apply.pl b/src/test/subscription/t/032_streaming_apply.pl index c06f87c255..f338f6d2b3 100644 --- a/src/test/subscription/t/032_streaming_apply.pl +++ b/src/test/subscription/t/032_streaming_apply.pl @@ -58,7 +58,9 @@ $node_subscriber->safe_psql( $node_publisher->wait_for_catchup($appname); # It is not allowed that the unique index on the publisher and the subscriber -# is different. Check the error reported by background worker in this case. +# is different. Check the error reported by background worker in this case. And +# after retrying in apply worker, we check if the data is replicated +# successfully. # First we check the unique index on normal table. $node_subscriber->safe_psql('postgres', "CREATE UNIQUE INDEX test_tab_b_idx ON test_tab (b)"); @@ -82,14 +84,15 @@ $node_subscriber->wait_for_log( qr/ERROR: cannot replicate relation with different unique index/, $offset); -# Drop the unique index on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', "DROP INDEX test_tab_b_idx"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); my $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); -is($result, qq(5000), 'data replicated to subscriber after dropping index'); +is($result, qq(5000), 'data replicated to subscribers after retrying because of unique index'); + +# Drop the unique index on the subscriber. +$node_subscriber->safe_psql('postgres', "DROP INDEX test_tab_b_idx"); # Then we check the unique index on partition table. $node_subscriber->safe_psql('postgres', @@ -106,17 +109,20 @@ $node_subscriber->wait_for_log( qr/ERROR: cannot replicate relation with different unique index/, $offset); -# Drop the unique index on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', "DROP INDEX test_tab_b_partition_idx"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); -is($result, qq(5000), 'data replicated to subscriber after dropping index'); +is($result, qq(5000), 'data replicated to subscribers after retrying because of unique index'); + +# Drop the unique index on the subscriber. +$node_subscriber->safe_psql('postgres', "DROP INDEX test_tab_b_partition_idx"); # Triggers which execute non-immutable function are not allowed on the # subscriber side. Check the error reported by background worker in this case. +# And after retrying in apply worker, we check if the data is replicated +# successfully. # First we check the trigger function on normal table. $node_subscriber->safe_psql( 'postgres', qq{ @@ -140,15 +146,16 @@ $node_subscriber->wait_for_log( qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, $offset); -# Drop the trigger on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "DROP TRIGGER insert_trig ON test_tab"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); -is($result, qq(0), 'data replicated to subscriber after dropping trigger'); +is($result, qq(0), 'data replicated to subscribers after retrying because of trigger'); + +# Drop the trigger on the subscriber. +$node_subscriber->safe_psql('postgres', + "DROP TRIGGER insert_trig ON test_tab"); # Then we check the unique index on partition table. $node_subscriber->safe_psql( @@ -168,19 +175,21 @@ $node_subscriber->wait_for_log( qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, $offset); -# Drop the trigger on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "DROP TRIGGER insert_trig ON test_tab_partition"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); -is($result, qq(0), 'data replicated to subscriber after dropping trigger'); +is($result, qq(0), 'data replicated to subscribers after retrying because of trigger'); + +# Drop the trigger on the subscriber. +$node_subscriber->safe_psql('postgres', + "DROP TRIGGER insert_trig ON test_tab_partition"); # It is not allowed that column default value expression contains a # non-immutable function on the subscriber side. Check the error reported by -# background worker in this case. +# background worker in this case. And after retrying in apply worker, we check +# if the data is replicated successfully. # First we check the column default value expression on normal table. $node_subscriber->safe_psql('postgres', "ALTER TABLE test_tab ALTER COLUMN b SET DEFAULT random()"); @@ -196,16 +205,17 @@ $node_subscriber->wait_for_log( qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, $offset); -# Drop default value on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "ALTER TABLE test_tab ALTER COLUMN b DROP DEFAULT"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); is($result, qq(5000), - 'data replicated to subscriber after dropping default value expression'); + 'data replicated to subscribers after retrying because of column default value'); + +# Drop default value on the subscriber. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab ALTER COLUMN b DROP DEFAULT"); # Then we check the column default value expression on partition table. $node_subscriber->safe_psql('postgres', @@ -222,20 +232,22 @@ $node_subscriber->wait_for_log( qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, $offset); -# Drop default value on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "ALTER TABLE test_tab_partition ALTER COLUMN b DROP DEFAULT"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); is($result, qq(5000), - 'data replicated to subscriber after dropping default value expression'); + 'data replicated to subscribers after retrying because of column default value'); + +# Drop default value on the subscriber. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partition ALTER COLUMN b DROP DEFAULT"); # It is not allowed that domain constraint expression contains a non-immutable # function on the subscriber side. Check the error reported by background -# worker in this case. +# worker in this case. And after retrying in apply worker, we check if the data +# is replicated successfully. # Because the column type of the partition table must be the same as its parent # table, only test normal table here. $node_subscriber->safe_psql( @@ -253,21 +265,23 @@ $node_subscriber->wait_for_log( qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, $offset); -# Drop domain constraint expression on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "ALTER TABLE test_tab ALTER COLUMN a TYPE int"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); is($result, qq(0), - 'data replicated to subscriber after dropping domain constraint expression' + 'data replicated to subscribers after retrying because of domain' ); -# It is not allowed that constraint expression contains a non-immutable function -# on the subscriber side. Check the error reported by background worker in this -# case. +# Drop domain constraint expression on the subscriber. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab ALTER COLUMN a TYPE int"); + +# It is not allowed that constraint expression contains a non-immutable +# function on the subscriber side. Check the error reported by background +# worker in this case. And after retrying in apply worker, we check if the data +# is replicated successfully. # First we check the constraint expression on normal table. $node_subscriber->safe_psql( 'postgres', qq{ @@ -285,16 +299,17 @@ $node_subscriber->wait_for_log( qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, $offset); -# Drop constraint on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "ALTER TABLE test_tab DROP CONSTRAINT test_tab_con"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); is($result, qq(5000), - 'data replicated to subscriber after dropping constraint expression'); + 'data replicated to subscribers after retrying because of constraint'); + +# Drop constraint on the subscriber. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab DROP CONSTRAINT test_tab_con"); # Then we check the constraint expression on partition table. $node_subscriber->safe_psql( @@ -311,19 +326,21 @@ $node_subscriber->wait_for_log( qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, $offset); -# Drop constraint on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "ALTER TABLE test_tab_partition DROP CONSTRAINT test_tab_con"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); is($result, qq(0), - 'data replicated to subscriber after dropping constraint expression'); + 'data replicated to subscribers after retrying because of constraint'); + +# Drop constraint on the subscriber. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partition DROP CONSTRAINT test_tab_con"); # It is not allowed that foreign key on the subscriber side. Check the error -# reported by background worker in this case. +# reported by background worker in this case. And after retrying in apply +# worker, we check if the data is replicated successfully. # First we check the foreign key on normal table. $node_publisher->safe_psql('postgres', "DELETE FROM test_tab"); $node_publisher->wait_for_catchup($appname); @@ -344,16 +361,17 @@ $node_subscriber->wait_for_log( qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, $offset); -# Drop the foreign key constraint on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "ALTER TABLE test_tab DROP CONSTRAINT test_tabfk"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); is($result, qq(5000), - 'data replicated to subscriber after dropping the foreign key'); + 'data replicated to subscribers after retrying because of foreign key'); + +# Drop the foreign key constraint on the subscriber. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab DROP CONSTRAINT test_tabfk"); # Then we check the foreign key on partition table. $node_publisher->wait_for_catchup($appname); @@ -374,16 +392,17 @@ $node_subscriber->wait_for_log( qr/ERROR: cannot replicate relation. There is at least one non-immutable function/, $offset); -# Drop the foreign key constraint on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "ALTER TABLE test_tab_partition DROP CONSTRAINT test_tab_patition_fk"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); is($result, qq(5000), - 'data replicated to subscriber after dropping the foreign key'); + 'data replicated to subscribers after retrying because of foreign key'); + +# Drop the foreign key constraint on the subscriber. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partition DROP CONSTRAINT test_tab_patition_fk"); $node_subscriber->stop; $node_publisher->stop; -- 2.23.0.windows.1 ^ permalink raw reply [nested|flat] 66+ messages in thread
* RE: Perform streaming logical transactions by background workers and parallel apply @ 2022-06-28 03:23 [email protected] <[email protected]> parent: Peter Smith <[email protected]> 1 sibling, 0 replies; 66+ messages in thread From: [email protected] @ 2022-06-28 03:23 UTC (permalink / raw) To: Peter Smith <[email protected]>; +Cc: Amit Kapila <[email protected]>; Masahiko Sawada <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; PostgreSQL Hackers <[email protected]> On Mon, Jun 21, 2022 at 9:41 AM Peter Smith <[email protected]> wrote: > Here are some review comments for the v11-0001 patch. > > (I will review the remaining patches 0002-0005 and post any comments later) > Thanks for your comments. > 6. doc/src/sgml/protocol.sgml > > Since there are protocol changes made here, shouldn’t there also be > some corresponding LOGICALREP_PROTO_XXX constants and special checking > added in the worker.c? I think it is okay not to add new macro. Because we just expanded the existing options ("streaming"). And we added a check for version in function apply_handle_stream_abort. > 8. doc/src/sgml/ref/create_subscription.sgml > > + <para> > + If set to <literal>on</literal>, the changes of transaction are > + written to temporary files and then applied at once after the > + transaction is committed on the publisher. > + </para> > > SUGGESTION > If set to on, the incoming changes are written to a temporary file and > then applied only after the transaction is committed on the publisher. In "on" mode, there may be more than one temporary file for one streaming transaction. (see the invocation of function BufFileCreateFileSet in function stream_open_file and function subxact_info_write) So I think the existing description might be better. If you feel this sentence is not clear, I will try to improve it later. > 10. src/backend/access/transam/xact.c > > @@ -1741,6 +1742,13 @@ RecordTransactionAbort(bool isSubXact) > elog(PANIC, "cannot abort transaction %u, it was already committed", > xid); > > + /* > + * Are we using the replication origins feature? Or, in other words, > + * are we replaying remote actions? > + */ > + replorigin = (replorigin_session_origin != InvalidRepOriginId && > + replorigin_session_origin != DoNotReplicateId); > + > /* Fetch the data we need for the abort record */ > nrels = smgrGetPendingDeletes(false, &rels); > nchildren = xactGetCommittedChildren(&children); > @@ -1765,6 +1773,11 @@ RecordTransactionAbort(bool isSubXact) > MyXactFlags, InvalidTransactionId, > NULL); > > + if (replorigin) > + /* Move LSNs forward for this replication origin */ > + replorigin_session_advance(replorigin_session_origin_lsn, > + XactLastRecEnd); > + > > I did not see any reason why the code assigning the 'replorigin' and > the code checking the 'replorigin' are separated like they are. I > thought these 2 new code fragments should be kept together. Perhaps it > was decided this assignment must be outside the critical section? But > if that’s the case maybe a comment explaining so would be good. > > ~~~ > > 11. src/backend/access/transam/xact.c > > + if (replorigin) > + /* Move LSNs forward for this replication origin */ > + replorigin_session_advance(replorigin_session_origin_lsn, > + > > The positioning of that comment is unusual. Maybe better before the check? As Amit-san said in [1], this is just for consistency with the code in the function RecordTransactionCommit. > 12. src/backend/commands/subscriptioncmds.c - defGetStreamingMode > > + /* > + * If no parameter given, assume "true" is meant. > + */ > + if (def->arg == NULL) > + return SUBSTREAM_ON; > > SUGGESTION for comment > If the streaming parameter is given but no parameter value is > specified, then assume "true" is meant. I think it might be better to be consistent with the function defGetBoolean here. > 24. .../replication/logical/applybgwroker.c - LogicalApplyBgwLoop > > +/* Apply Background Worker main loop */ > +static void > +LogicalApplyBgwLoop(shm_mq_handle *mqh, volatile ApplyBgworkerShared > *pst) > > Why is the name incosistent with other function names in the file? > Should it be apply_bgworker_loop? I think this function name would be better to be consistent with the function LogicalRepApplyLoop. > 28. .../replication/logical/applybgwroker.c - LogicalApplyBgwMain > > For consistency should it be called apply_bgworker_main? I think this function name would be better to be consistent with the function ApplyWorkerMain. > 30. src/backend/replication/logical/decode.c > > @@ -651,9 +651,10 @@ DecodeCommit(LogicalDecodingContext *ctx, > XLogRecordBuffer *buf, > { > for (i = 0; i < parsed->nsubxacts; i++) > { > - ReorderBufferForget(ctx->reorder, parsed->subxacts[i], buf->origptr); > + ReorderBufferForget(ctx->reorder, parsed->subxacts[i], buf->origptr, > + commit_time); > } > - ReorderBufferForget(ctx->reorder, xid, buf->origptr); > + ReorderBufferForget(ctx->reorder, xid, buf->origptr, commit_time); > > ReorderBufferForget was declared with 'abort_time' param. So it makes > these calls a bit confusing looking to be passing 'commit_time' > > Maybe better to do like below and pass 'forget_time' (inside that > 'if') along with an explanatory comment: > > TimestampTz forget_time = commit_time; I did not change this. I am just not sure how much this will help. > 36. src/backend/replication/logical/launcher.c - > logicalrep_apply_background_worker_count > > + int res = 0; > + > > A better variable name here would be 'count', or even 'n'. I think this variable name would be better to be consistent with the function logicalrep_sync_worker_count. > 38. src/backend/replication/logical/proto.c - logicalrep_read_stream_abort > > + /* > + * If the version of the publisher is lower than the version of the > + * subscriber, it may not support sending these two fields, so only take > + * these fields when include_abort_lsn is true. > + */ > + if (include_abort_lsn) > + { > + abort_data->abort_lsn = pq_getmsgint64(in); > + abort_data->abort_time = pq_getmsgint64(in); > + } > + else > + { > + abort_data->abort_lsn = InvalidXLogRecPtr; > + abort_data->abort_time = 0; > + } > > This comment is documenting a decision that was made elsewhere. > > But it somehow feels wrong to me that the decision to read or not read > the abort time/lsn is made by the caller of this function. IMO it > might make more sense if the server version was simply passed as a > param and then this function can be in control of its own destiny and > make the decision does it need to read those extra fields or not. An > extra member flag can be added to LogicalRepStreamAbortData to > indicate if abort_data read these values or not. I understand what you mean. But I am not sure if it is appropriate to introduce version information in the file proto.c just for the STREAM_ABORT message. And I think it might complicate the file proto.c if introducing version information. Also, I think it might not be a good idea to add a flag to LogicalRepStreamAbortData (There is no similar flag in structure LogicalRep.*Data). So, I just introduce a flag to decide whether we should read these fields from the STREAM_ABORT message. > 41. src/backend/replication/logical/worker.c > > -static ApplyErrorCallbackArg apply_error_callback_arg = > +ApplyErrorCallbackArg apply_error_callback_arg = > { > .command = 0, > .rel = NULL, > @@ -242,7 +246,7 @@ static ApplyErrorCallbackArg apply_error_callback_arg = > .origin_name = NULL, > }; > > Maybe it is still a good idea to at least keep the old comment here: > /* Struct for saving and restoring apply errcontext information */ I think the old comment looks like it was for the structure ApplyErrorCallbackArg, not the variable apply_error_callback_arg. So I did not add new comments here for variable apply_error_callback_arg. > 42. src/backend/replication/logical/worker.c > > +/* check if we are applying the transaction in apply background worker */ > +#define apply_bgworker_active() (in_streamed_transaction && > stream_apply_worker != NULL) > > 42a. > Uppercase comment. > > 42b. > "in apply background worker" -> "in apply background worker" => 42a. improved as suggested. => 42b. Sorry, I am not sure what you mean. > 43. src/backend/replication/logical/worker.c - handle_streamed_transaction > > @@ -426,41 +437,76 @@ end_replication_step(void) > } > > /* > - * Handle streamed transactions. > + * Handle streamed transactions for both main apply worker and apply > background > + * worker. > * > - * If in streaming mode (receiving a block of streamed transaction), we > - * simply redirect it to a file for the proper toplevel transaction. > + * In streaming case (receiving a block of streamed transaction), for > + * SUBSTREAM_ON mode, we simply redirect it to a file for the proper toplevel > + * transaction, and for SUBSTREAM_APPLY mode, we send the changes to > background > + * apply worker (LOGICAL_REP_MSG_RELATION or LOGICAL_REP_MSG_TYPE > changes will > + * also be applied in main apply worker). > * > - * Returns true for streamed transactions, false otherwise (regular mode). > + * For non-streamed transactions, returns false; > + * For streamed transactions, returns true if in main apply worker (except we > + * apply streamed transaction in "apply" mode and address > + * LOGICAL_REP_MSG_RELATION or LOGICAL_REP_MSG_TYPE changes), false > otherwise. > */ > > Maybe it is accurate (I don’t know), but this header comment seems > excessively complicated with so many quirks about when to return > true/false. Can it be reworded into plainer language? Improved the comments like below: ``` * For non-streamed transactions, returns false; * For streamed transactions, returns true if in main apply worker, false * otherwise. * * But there are two exceptions: If we apply streamed transaction in main apply * worker with parallel mode, it will return false when we address * LOGICAL_REP_MSG_RELATION or LOGICAL_REP_MSG_TYPE changes. ``` > 46. src/backend/replication/logical/worker.c - apply_handle_stream_prepare > > + /* > + * This is the main apply worker and the transaction has been > + * serialized to file, replay all the spooled operations. > + */ > > SUGGESTION > The transaction has been serialized to file. Replay all the spooled operations. Both #46 and #54 seem to try to improve on the same comment. Personally I prefer the improvement in #54. So improved this as suggested in #54. > 50. src/backend/replication/logical/worker.c - apply_handle_stream_abort > > + /* Check whether the publisher sends abort_lsn and abort_time. */ > + if (am_apply_bgworker()) > + include_abort_lsn = MyParallelState->server_version >= 150000; > + > + logicalrep_read_stream_abort(s, &abort_data, include_abort_lsn); > > Here is where I felt maybe just the server version could be passed so > the logicalrep_read_stream_abort could decide itself what message > parts needed to be read. Basically it seems strange that the message > contain parts which might not be read. I felt it is better to always > read the whole message then later you can choose what parts you are > interested in. Please refer to the reply to #38. In addition, we do not always read these two new fields from STREAM_ABORT message. Because if the subscriber's version is higher than the publisher's version, it may try to read data that in the invalid area. I think this is not a correct behaviour. > 63. > > I also did a quick check of all the new debug logging added. Here is > everyhing from patch v11-0001. > > apply_bgworker_free: > + elog(DEBUG1, "adding finished apply worker #%u for xid %u to the idle list", > + wstate->pstate->n, wstate->pstate->stream_xid); > > LogicalApplyBgwLoop: > + elog(DEBUG1, "[Apply BGW #%u] ended processing streaming chunk," > + "waiting on shm_mq_receive", pst->n); > > + elog(DEBUG1, "[Apply BGW #%u] exiting", pst->n); > > ApplyBgworkerMain: > + elog(DEBUG1, "[Apply BGW #%u] started", pst->n); > > apply_bgworker_setup: > + elog(DEBUG1, "setting up apply worker #%u", > list_length(ApplyWorkersList) + 1); > > apply_bgworker_set_status: > + elog(DEBUG1, "[Apply BGW #%u] set status to %d", MyParallelState->n, > status); > > apply_bgworker_subxact_info_add: > + elog(DEBUG1, "[Apply BGW #%u] defining savepoint %s", > + MyParallelState->n, spname); > > apply_handle_stream_prepare: > + elog(DEBUG1, "received prepare for streamed transaction %u", > + prepare_data.xid); > > apply_handle_stream_start: > + elog(DEBUG1, "starting streaming of xid %u", stream_xid); > > apply_handle_stream_stop: > + elog(DEBUG1, "stopped streaming of xid %u, %u changes streamed", > stream_xid, nchanges); > > apply_handle_stream_abort: > + elog(DEBUG1, "[Apply BGW #%u] aborting current transaction xid=%u, > subxid=%u", > + MyParallelState->n, GetCurrentTransactionIdIfAny(), > + GetCurrentSubTransactionId()); > > + elog(DEBUG1, "[Apply BGW #%u] rolling back to savepoint %s", > + MyParallelState->n, spname); > > apply_handle_stream_commit: > + elog(DEBUG1, "received commit for streamed transaction %u", xid); > > > Observations: > > 63a. > Every new introduced message is at level DEBUG1 (not DEBUG). AFAIK > this is OK, because the messages are all protocol related and every > other existing debug message of the current replication worker.c was > also at the same DEBUG1 level. > > 63b. > The prefix "[Apply BGW #%u]" is used to indicate the bgworker is > executing the code, but it does not seem to be used 100% consistently > - e.g. there are some apply_bgworker_XXX functions not using this > prefix. Is that OK or a mistake? Thanks for your check. I confirm this point in v13. And there are 5 functions do not use the prefix "[Apply BGW #%u]": ``` apply_bgworker_free apply_bgworker_setup apply_bgworker_send_data apply_bgworker_wait_for apply_bgworker_check_status ``` These 5 functions do not use this prefix because they only output logs in apply worker. So I think it is okay. The rest of the comments are improved as suggested. The new patches were attached in [2]. [1] - https://www.postgresql.org/message-id/CAA4eK1J9_jcLNVqmxt_d28uGi6hAV31wjYdgmg1p8BGuEctNpw%40mail.gma... [2] - https://www.postgresql.org/message-id/OS3PR01MB62758DBE8FA12BA72A43AC819EB89%40OS3PR01MB6275.jpnprd0... Regards, Wang wei ^ permalink raw reply [nested|flat] 66+ messages in thread
* RE: Perform streaming logical transactions by background workers and parallel apply @ 2022-06-28 03:23 [email protected] <[email protected]> parent: Peter Smith <[email protected]> 0 siblings, 0 replies; 66+ messages in thread From: [email protected] @ 2022-06-28 03:23 UTC (permalink / raw) To: Peter Smith <[email protected]>; +Cc: Amit Kapila <[email protected]>; Masahiko Sawada <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; PostgreSQL Hackers <[email protected]> On Thu, Jun 23, 2022 at 9:41 AM Peter Smith <[email protected]> wrote: > Here are some review comments for v12-0002 Thanks for your comments. > 3. .../subscription/t/022_twophase_cascade.pl > > For every test file in this patch the new function is passed $is_apply > = 0/1 to indicate to use 'on' or 'apply' parameter value. But in this > test file the parameter is passed as $streaming_mode = 'on'/'apply'. > > I was wondering if (for the sake of consistency) it might be better to > use the same parameter kind for all of the test files. Actually, I > don't care if you choose to do nothing and leave this as-is; I am just > posting this review comment in case it was not a deliberate decision > to implement them differently. > > e.g. > + my ($node_publisher, $node_subscriber, $appname, $is_apply) = @_; > > versus > + my ($node_A, $node_B, $node_C, $appname_B, $appname_C, > $streaming_mode) = > + @_; This is because in 022_twophase_cascade.pl, altering subscription streaming mode is inside test_streaming(), it would be more convenient to pass which streaming mode we use (on or apply), we can directly use that in alter subscription command. In other files, we need to get the option because we only check the log in apply mode, so I think it is sufficient to pass 'is_apply' (whose value is 0 or 1). Because of these differences, I did not change it. The rest of the comments are improved as suggested. The new patches were attached in [1]. [1] - https://www.postgresql.org/message-id/OS3PR01MB62758DBE8FA12BA72A43AC819EB89%40OS3PR01MB6275.jpnprd0... Regards, Wang wei ^ permalink raw reply [nested|flat] 66+ messages in thread
* Re: Perform streaming logical transactions by background workers and parallel apply @ 2022-06-28 04:15 Amit Kapila <[email protected]> parent: [email protected] <[email protected]> 4 siblings, 1 reply; 66+ messages in thread From: Amit Kapila @ 2022-06-28 04:15 UTC (permalink / raw) To: [email protected] <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Peter Smith <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; PostgreSQL Hackers <[email protected]> On Tue, Jun 28, 2022 at 8:51 AM [email protected] <[email protected]> wrote: > > On Thu, Jun 23, 2022 at 16:44 PM Amit Kapila <[email protected]> wrote: > > On Thu, Jun 23, 2022 at 12:51 PM [email protected] > > <[email protected]> wrote: > > > > > > On Mon, Jun 20, 2022 at 11:00 AM Amit Kapila <[email protected]> > > wrote: > > > > I have improved the comments in this and other related sections of the > > > > patch. See attached. > > > Thanks for your comments and patch! > > > Improved the comments as you suggested. > > > > > > > > > 3. > > > > > > + > > > > > > + <para> > > > > > > + Setting streaming mode to <literal>apply</literal> could export invalid > > > > LSN > > > > > > + as finish LSN of failed transaction. Changing the streaming mode and > > > > making > > > > > > + the same conflict writes the finish LSN of the failed transaction in the > > > > > > + server log if required. > > > > > > + </para> > > > > > > > > > > > > How will the user identify that this is an invalid LSN value and she > > > > > > shouldn't use it to SKIP the transaction? Can we change the second > > > > > > sentence to: "User should change the streaming mode to 'on' if they > > > > > > would instead wish to see the finish LSN on error. Users can use > > > > > > finish LSN to SKIP applying the transaction." I think we can give > > > > > > reference to docs where the SKIP feature is explained. > > > > > Improved the sentence as suggested. > > > > > > > > > > > > > You haven't answered first part of the comment: "How will the user > > > > identify that this is an invalid LSN value and she shouldn't use it to > > > > SKIP the transaction?". Have you checked what value it displays? For > > > > example, in one of the case in apply_error_callback as shown in below > > > > code, we don't even display finish LSN if it is invalid. > > > > else if (XLogRecPtrIsInvalid(errarg->finish_lsn)) > > > > errcontext("processing remote data for replication origin \"%s\" > > > > during \"%s\" in transaction %u", > > > > errarg->origin_name, > > > > logicalrep_message_type(errarg->command), > > > > errarg->remote_xid); > > > I am sorry that I missed something in my previous reply. > > > The invalid LSN value here is to say InvalidXLogRecPtr (0/0). > > > Here is an example : > > > ``` > > > 2022-06-23 14:30:11.343 CST [822333] logical replication worker CONTEXT: > > processing remote data for replication origin "pg_16389" during "INSERT" for > > replication target relation "public.tab" in transaction 727 finished at 0/0 > > > ``` > > > > > > > I don't think it is a good idea to display invalid values. We can mask > > this as we are doing in other cases in function apply_error_callback. > > The ideal way is that we provide a view/system table for users to > > check these errors but that is a matter of another patch. So users > > probably need to check Logs to see if the error is from a background > > apply worker to decide whether or not to switch streaming mode. > > Thanks for your comments. > I improved it as you suggested. I mask the LSN if it is invalid LSN(0/0). > Also, I improved the related pg-doc as following: > ``` > When the streaming mode is <literal>parallel</literal>, the finish LSN of > failed transactions may not be logged. In that case, it may be necessary to > change the streaming mode to <literal>on</literal> and cause the same > conflicts again so the finish LSN of the failed transaction will be written > to the server log. For the usage of finish LSN, please refer to <link > linkend="sql-altersubscription"><command>ALTER SUBSCRIPTION ... > SKIP</command></link>. > ``` > After improving this (mask invalid LSN), I found that this improvement and > parallel apply patch do not seem to have a strong correlation. Would it be > better to improve and commit in another separate patch? > Is there any other case where we can hit this code path (mask invalidLSN) without this patch? -- With Regards, Amit Kapila. ^ permalink raw reply [nested|flat] 66+ messages in thread
* RE: Perform streaming logical transactions by background workers and parallel apply @ 2022-06-28 07:20 [email protected] <[email protected]> parent: Amit Kapila <[email protected]> 0 siblings, 0 replies; 66+ messages in thread From: [email protected] @ 2022-06-28 07:20 UTC (permalink / raw) To: Amit Kapila <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Peter Smith <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; PostgreSQL Hackers <[email protected]> On Tues, Jun 28, 2022 at 12:15 PM Amit Kapila <[email protected]> wrote: > On Tue, Jun 28, 2022 at 8:51 AM [email protected] > <[email protected]> wrote: > > > > On Thu, Jun 23, 2022 at 16:44 PM Amit Kapila <[email protected]> > wrote: > > > On Thu, Jun 23, 2022 at 12:51 PM [email protected] > > > <[email protected]> wrote: > > > > > > > > On Mon, Jun 20, 2022 at 11:00 AM Amit Kapila <[email protected]> > > > wrote: > > > > > I have improved the comments in this and other related sections of the > > > > > patch. See attached. > > > > Thanks for your comments and patch! > > > > Improved the comments as you suggested. > > > > > > > > > > > 3. > > > > > > > + > > > > > > > + <para> > > > > > > > + Setting streaming mode to <literal>apply</literal> could export > invalid > > > > > LSN > > > > > > > + as finish LSN of failed transaction. Changing the streaming mode > and > > > > > making > > > > > > > + the same conflict writes the finish LSN of the failed transaction in > the > > > > > > > + server log if required. > > > > > > > + </para> > > > > > > > > > > > > > > How will the user identify that this is an invalid LSN value and she > > > > > > > shouldn't use it to SKIP the transaction? Can we change the second > > > > > > > sentence to: "User should change the streaming mode to 'on' if they > > > > > > > would instead wish to see the finish LSN on error. Users can use > > > > > > > finish LSN to SKIP applying the transaction." I think we can give > > > > > > > reference to docs where the SKIP feature is explained. > > > > > > Improved the sentence as suggested. > > > > > > > > > > > > > > > > You haven't answered first part of the comment: "How will the user > > > > > identify that this is an invalid LSN value and she shouldn't use it to > > > > > SKIP the transaction?". Have you checked what value it displays? For > > > > > example, in one of the case in apply_error_callback as shown in below > > > > > code, we don't even display finish LSN if it is invalid. > > > > > else if (XLogRecPtrIsInvalid(errarg->finish_lsn)) > > > > > errcontext("processing remote data for replication origin \"%s\" > > > > > during \"%s\" in transaction %u", > > > > > errarg->origin_name, > > > > > logicalrep_message_type(errarg->command), > > > > > errarg->remote_xid); > > > > I am sorry that I missed something in my previous reply. > > > > The invalid LSN value here is to say InvalidXLogRecPtr (0/0). > > > > Here is an example : > > > > ``` > > > > 2022-06-23 14:30:11.343 CST [822333] logical replication worker CONTEXT: > > > processing remote data for replication origin "pg_16389" during "INSERT" for > > > replication target relation "public.tab" in transaction 727 finished at 0/0 > > > > ``` > > > > > > > > > > I don't think it is a good idea to display invalid values. We can mask > > > this as we are doing in other cases in function apply_error_callback. > > > The ideal way is that we provide a view/system table for users to > > > check these errors but that is a matter of another patch. So users > > > probably need to check Logs to see if the error is from a background > > > apply worker to decide whether or not to switch streaming mode. > > > > Thanks for your comments. > > I improved it as you suggested. I mask the LSN if it is invalid LSN(0/0). > > Also, I improved the related pg-doc as following: > > ``` > > When the streaming mode is <literal>parallel</literal>, the finish LSN of > > failed transactions may not be logged. In that case, it may be necessary to > > change the streaming mode to <literal>on</literal> and cause the same > > conflicts again so the finish LSN of the failed transaction will be written > > to the server log. For the usage of finish LSN, please refer to <link > > linkend="sql-altersubscription"><command>ALTER SUBSCRIPTION ... > > SKIP</command></link>. > > ``` > > After improving this (mask invalid LSN), I found that this improvement and > > parallel apply patch do not seem to have a strong correlation. Would it be > > better to improve and commit in another separate patch? > > > > Is there any other case where we can hit this code path (mask > invalidLSN) without this patch? I realized that there is no normal case that could hit this code path in HEAD. If we want to hit this code path, we must set apply_error_callback_arg.rel to valid relation and set finish LSN to InvalidXLogRecPtr. But now in HEAD, we only set apply_error_callback_arg.rel to valid relation after setting finish LSN to valid LSN. So it seems fine change this along with the parallel apply patch. Regards, Wang wei ^ permalink raw reply [nested|flat] 66+ messages in thread
* Re: Perform streaming logical transactions by background workers and parallel apply @ 2022-07-01 06:43 Peter Smith <[email protected]> parent: [email protected] <[email protected]> 4 siblings, 2 replies; 66+ messages in thread From: Peter Smith @ 2022-07-01 06:43 UTC (permalink / raw) To: [email protected] <[email protected]>; +Cc: Amit Kapila <[email protected]>; Masahiko Sawada <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; PostgreSQL Hackers <[email protected]> Below are some review comments for patches v14-0001, and v14-0002: ======== v14-0001 ======== 1.1 Commit message For now, 'parallel' means the streaming will be applied via a apply background worker if available. 'on' means the streaming transaction will be spilled to disk. By the way, we do not change the default behaviour. SUGGESTION (minor tweaks) The parameter value 'parallel' means the streaming will be applied via an apply background worker, if available. The parameter value 'on' means the streaming transaction will be spilled to disk. The default value is 'off' (same as current behaviour). ====== 1.2 doc/src/sgml/protocol.sgml - Protocol constants Previously I wrote that since there are protocol changes here, shouldn’t there also be some corresponding LOGICALREP_PROTO_XXX constants and special checking added in the worker.c? But you said [1 comment #6] you think it is OK because... IMO, I still disagree with the reply. The fact is that the protocol *has* been changed, so IIUC that is precisely the reason for having those protocol constants. e.g I am guessing you might assign the new one somewhere here: -- server_version = walrcv_server_version(LogRepWorkerWalRcvConn); options.proto.logical.proto_version = server_version >= 150000 ? LOGICALREP_PROTO_TWOPHASE_VERSION_NUM : server_version >= 140000 ? LOGICALREP_PROTO_STREAM_VERSION_NUM : LOGICALREP_PROTO_VERSION_NUM; -- And then later you would refer to this new protocol version (instead of the server version) when calling to the apply_handle_stream_abort function. ====== 1.3 doc/src/sgml/ref/create_subscription.sgml + <para> + If set to <literal>on</literal>, the changes of transaction are + written to temporary files and then applied at once after the + transaction is committed on the publisher. + </para> Previously I suggested changing some text but it was rejected [1 comment #8] because you said there may be *multiple* files, not just one. That is fair enough, but there were some other changes to that suggested text unrelated to the number of files. SUGGESTION #2 If set to on, the incoming changes are written to temporary files and then applied only after the transaction is committed on the publisher. ~~~ 1.4 + <para> + If set to <literal>parallel</literal>, incoming changes are directly + applied via one of the apply background workers, if available. If no + background worker is free to handle streaming transaction then the + changes are written to a file and applied after the transaction is + committed. Note that if an error happens when applying changes in a + background worker, the finish LSN of the remote transaction might + not be reported in the server log. </para> Should this also say "written to temporary files" instead of "written to a file"? ====== 1.5 src/backend/commands/subscriptioncmds.c + /* + * If no parameter given, assume "true" is meant. + */ Previously I suggested an update for this comment, but it was rejected [1 comment #12] saying you wanted consistency with defGetBoolean. Sure, that is one point of view. Another one is that "two wrongs don't make a right". IIUC that comment as it currently stands is incorrect because in this case there *is* a parameter given - it is just the parameter *value* that is missing. Maybe see what other people think? ====== 1.6. src/backend/replication/logical/Makefile It seems to me like these files were intended to be listed in alphabetical order, so you should move this new file accordingly. ====== 1.7 .../replication/logical/applybgworker.c +/* queue size of DSM, 16 MB for now. */ +#define DSM_QUEUE_SIZE 160000000 The comment should start uppercase. ~~~ 1.8 .../replication/logical/applybgworker.c - apply_bgworker_can_start Maybe this is just my opinion but it sounds a bit strange to over-use "we" in all the comments. 1.8.a +/* + * Confirm if we can try to start a new apply background worker. + */ +static bool +apply_bgworker_can_start(TransactionId xid) SUGGESTION Check if starting a new apply background worker is allowed. 1.8.b + /* + * We don't start new background worker if we are not in streaming parallel + * mode. + */ SUGGESTION Don't start a new background worker if not in streaming parallel mode. 1.8.c + /* + * We don't start new background worker if user has set skiplsn as it's + * possible that user want to skip the streaming transaction. For + * streaming transaction, we need to spill the transaction to disk so that + * we can get the last LSN of the transaction to judge whether to skip + * before starting to apply the change. + */ SUGGESTION Don't start a new background worker if... ~~~ 1.9 .../replication/logical/applybgworker.c - apply_bgworker_start +/* + * Try to start worker inside ApplyWorkersHash for requested xid. + */ +ApplyBgworkerState * +apply_bgworker_start(TransactionId xid) The comment seems not quite right. SUGGESTION Try to start an apply background worker and, if successful, cache it in ApplyWorkersHash keyed by the specified xid. ~~~ 1.10 .../replication/logical/applybgworker.c - apply_bgworker_find + /* + * Find entry for requested transaction. + */ + entry = hash_search(ApplyWorkersHash, &xid, HASH_FIND, &found); + if (found) + { + entry->wstate->pstate->status = APPLY_BGWORKER_BUSY; + return entry->wstate; + } + else + return NULL; +} IMO it is an unexpected side-effect for the function called "find" to be also modifying the thing that it found. IMO this setting BUSY should either be done by the caller, or else this function name should be renamed to make it obvious that this is doing more than just "finding" something. ~~~ 1.11 .../replication/logical/applybgworker.c - LogicalApplyBgwLoop + /* + * Push apply error context callback. Fields will be filled applying + * applying a change. + */ Typo: "applying applying" ~~~ 1.12 .../replication/logical/applybgworker.c - apply_bgworker_setup + if (launched) + ApplyWorkersList = lappend(ApplyWorkersList, wstate); + else + { + shm_mq_detach(wstate->mq_handle); + dsm_detach(wstate->dsm_seg); + pfree(wstate); + + wstate->mq_handle = NULL; + wstate->dsm_seg = NULL; + wstate = NULL; + } I am not sure what those first 2 NULL assignments are trying to achieve. Nothing AFAICT. In any case, it looks like a bug to deference the 'wstate' after you already pfree-d it in the line above. ~~~ 1.13 .../replication/logical/applybgworker.c - apply_bgworker_check_status + * Exit if any relation is not in the READY state and if any worker is handling + * the streaming transaction at the same time. Because for streaming + * transactions that is being applied in apply background worker, we cannot + * decide whether to apply the change for a relation that is not in the READY + * state (see should_apply_changes_for_rel) as we won't know remote_final_lsn + * by that time. + */ +void +apply_bgworker_check_status(void) Somehow, I felt that this "Exit if..." comment really belonged at the appropriate place in the function body, instead of in the function header. ====== 1.14 src/backend/replication/logical/launcher.c - WaitForReplicationWorkerAttach @@ -151,8 +153,10 @@ get_subscription_list(void) * * This is only needed for cleaning up the shared memory in case the worker * fails to attach. + * + * Returns false if the attach fails. Otherwise return true. */ -static void +static bool WaitForReplicationWorkerAttach(LogicalRepWorker *worker, Comment should say either "Return" or "returns"; not one of each. ~~~ 1.15. src/backend/replication/logical/launcher.c - WaitForReplicationWorkerAttach + return worker->in_use ? true : false; Same as just: return worker->in_use; ~~~ 1.16. src/backend/replication/logical/launcher.c - logicalrep_worker_launch + bool is_subworker = (subworker_dsm != DSM_HANDLE_INVALID); + + /* We don't support table sync in subworker */ + Assert(!(is_subworker && OidIsValid(relid))); I'm not sure the comment is good. It sounds like it is something that might be possible but is just current "not supported". In fact, I thought this is really just a sanity check because the combination of those params is just plain wrong isn't it? Maybe a better comment is just: /* Sanity check */ ====== 1.17 src/backend/replication/logical/proto.c + /* + * If the version of the publisher is lower than the version of the + * subscriber, it may not support sending these two fields. So these + * fields are only taken if they are included. + */ + if (include_abort_lsn) 1.17a I thought that the comment about "versions of publishers lower than version of subscribers..." is bogus. Perhaps you have in mind just thinking about versions prior to PG16 but that is not what the comment is saying. E.g. sometime in the future, the publisher may be PG18 and the subscriber might be PG25. So that might work fine (even though the publisher is a lower version), but this comment will be completely misleading. BTW this is another reason I think code needs to be using protocol versions (not server versions). [See other comment #1.2] 1.17b. Anyway, I felt that any comment describing the meaning of the the 'include_abort_lsn' param would be better in the function header comment, instead of in the function body. ====== 1.18 src/backend/replication/logical/worker.c - file header comment + * 1) Separate background workers + * + * Assign a new apply background worker (if available) as soon as the xact's... Somehow this long comment did not ever mention that this mode is selected by the user using the 'streaming=parallel'. I thought probably it should say that somewhere here. ~~~ 1.19. src/backend/replication/logical/worker.c - ApplyErrorCallbackArg apply_error_callback_arg = { .command = 0, .rel = NULL, .remote_attnum = -1, .remote_xid = InvalidTransactionId, .finish_lsn = InvalidXLogRecPtr, .origin_name = NULL, }; I still thought that the above initialization deserves some sort of comment, even if you don't want to use the comment text previously suggested [1 comment #41] ~~~ 1.20 src/backend/replication/logical/worker.c - @@ -251,27 +258,38 @@ static MemoryContext LogicalStreamingContext = NULL; WalReceiverConn *LogRepWorkerWalRcvConn = NULL; Subscription *MySubscription = NULL; -static bool MySubscriptionValid = false; +bool MySubscriptionValid = false; bool in_remote_transaction = false; static XLogRecPtr remote_final_lsn = InvalidXLogRecPtr; /* fields valid only when processing streamed transaction */ -static bool in_streamed_transaction = false; +bool in_streamed_transaction = false; The tab alignment here looks wrong. IMO it's not worth trying to align these at all. I think the tabs are leftover from before when the vars used to be static. ~~~ 1.21 src/backend/replication/logical/worker.c - apply_bgworker_active +/* Check if we are applying the transaction in apply background worker */ +#define apply_bgworker_active() (in_streamed_transaction && stream_apply_worker != NULL) Sorry [1 comment #42b], I had meant to write "in apply background worker" -> "in an apply background worker". ~~~ 1.22 src/backend/replication/logical/worker.c - skip_xact_finish_lsn /* * We enable skipping all data modification changes (INSERT, UPDATE, etc.) for * the subscription if the remote transaction's finish LSN matches the subskiplsn. * Once we start skipping changes, we don't stop it until we skip all changes of * the transaction even if pg_subscription is updated and MySubscription->skiplsn - * gets changed or reset during that. Also, in streaming transaction cases, we - * don't skip receiving and spooling the changes since we decide whether or not + * gets changed or reset during that. Also, in streaming transaction cases (streaming = on), + * we don't skip receiving and spooling the changes since we decide whether or not * to skip applying the changes when starting to apply changes. The subskiplsn is * cleared after successfully skipping the transaction or applying non-empty * transaction. The latter prevents the mistakenly specified subskiplsn from - * being left. + * being left. Note that we cannot skip the streaming transaction in parallel + * mode, because we cannot get the finish LSN before applying the changes. */ "in parallel mode, because" -> "in 'streaming = parallel' mode, because" ~~~ 1.23 src/backend/replication/logical/worker.c - handle_streamed_transaction 1.23a /* - * Handle streamed transactions. + * Handle streamed transactions for both main apply worker and apply background + * worker. SUGGESTION Handle streamed transactions for both the main apply worker and the apply background workers. 1.23b + * In streaming case (receiving a block of streamed transaction), for + * SUBSTREAM_ON mode, we simply redirect it to a file for the proper toplevel + * transaction, and for SUBSTREAM_PARALLEL mode, we send the changes to + * background apply worker (LOGICAL_REP_MSG_RELATION or LOGICAL_REP_MSG_TYPE + * changes will also be applied in main apply worker). "background apply worker" -> "apply background workers" Also, I think you don't need to say "we" everywhere: "we simply redirect it" -> "simply redirect it" "we send the changes" -> "send the changes" 1.23c. + * But there are two exceptions: If we apply streamed transaction in main apply + * worker with parallel mode, it will return false when we address + * LOGICAL_REP_MSG_RELATION or LOGICAL_REP_MSG_TYPE changes. SUGGESTION Exception: When parallel mode is applying streamed transaction in the main apply worker, (e.g. when addressing LOGICAL_REP_MSG_RELATION or LOGICAL_REP_MSG_TYPE changes), then return false. ~~~ 1.24 src/backend/replication/logical/worker.c - handle_streamed_transaction 1.24a. /* not in streaming mode */ - if (!in_streamed_transaction) + if (!(in_streamed_transaction || am_apply_bgworker())) return false; Uppercase comment 1.24b + /* define a savepoint for a subxact if needed. */ + apply_bgworker_subxact_info_add(current_xid); Uppercase comment ~~~ 1.25 src/backend/replication/logical/worker.c - handle_streamed_transaction + /* + * This is the main apply worker, and there is an apply background + * worker. So we apply the changes of this transaction in an apply + * background worker. Pass the data to the worker. + */ SUGGESTION (to be more consistent with the next comment) This is the main apply worker, but there is an apply background worker, so apply the changes of this transaction in that background worker. Pass the data to the worker. ~~~ 1.26 src/backend/replication/logical/worker.c - handle_streamed_transaction + /* + * This is the main apply worker, but there is no apply background + * worker. So we write to temporary files and apply when the final + * commit arrives. SUGGESTION This is the main apply worker, but there is no apply background worker, so write to temporary files and apply when the final commit arrives. ~~~ 1.27 src/backend/replication/logical/worker.c - apply_handle_stream_prepare + /* + * Check if we are processing this transaction in an apply background + * worker. + */ SUGGESTION: Check if we are processing this transaction in an apply background worker and if so, send the changes to that worker. ~~~ 1.28 src/backend/replication/logical/worker.c - apply_handle_stream_prepare + if (wstate) + { + apply_bgworker_send_data(wstate, s->len, s->data); + + /* + * Wait for apply background worker to finish. This is required to + * maintain commit order which avoids failures due to transaction + * dependencies and deadlocks. + */ + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_FINISHED); + apply_bgworker_free(wstate); I think maybe the comment can be changed slightly, and then it can move up one line to the top of this code block (above the 3 statements). I think it will become more readable. SUGGESTION After sending the data to the apply background worker, wait for that worker to finish. This is necessary to maintain commit order which avoids failures due to transaction dependencies and deadlocks. ~~~ 1.29 src/backend/replication/logical/worker.c - apply_handle_stream_start + /* + * If no worker is available for the first stream start, we start to + * serialize all the changes of the transaction. + */ + else + { 1.29a. I felt that this comment should be INSIDE the else { block to be more readable. 1.29b. The comment can also be simplified a bit SUGGESTION: Since no apply background worker is available for the first stream start, serialize all the changes of the transaction. ~~~ 1.30 src/backend/replication/logical/worker.c - apply_handle_stream_start + /* if this is not the first segment, open existing subxact file */ + if (!first_segment) + subxact_info_read(MyLogicalRepWorker->subid, stream_xid); Uppercase comment ~~~ 1.31. src/backend/replication/logical/worker.c - apply_handle_stream_stop + if (apply_bgworker_active()) + { + char action = LOGICAL_REP_MSG_STREAM_STOP; Are all the tabs before the variable needed? ~~~ 1.32. src/backend/replication/logical/worker.c - apply_handle_stream_abort + /* Check whether the publisher sends abort_lsn and abort_time. */ + if (am_apply_bgworker()) + include_abort_lsn = MyParallelState->server_version >= 150000; Previously I already reported about this [1 comment #50] I just do not trust this code to do the correct thing. E.g. what if streaming=parallel but all bgworkers are exhausted then IIUC the am_apply_bgworker() will not be true. But then with both PG15 servers for pub/sub you will WRITE something but then you will not READ it. Won't the stream IO will get out of step and everything will fall apart? Perhaps the include_abort_lsn assignment should be unconditionally set, and I think this should be a protocol version check instead of a server version check shouldn’t it (see my earlier comment 1.2) ~~~ 1.32 src/backend/replication/logical/worker.c - apply_handle_stream_abort BTW, I think the PG16devel is now stamped in the GitHub HEAD so perhaps all of your 150000 checks should be now changed to say 160000? ~~~ 1.33 src/backend/replication/logical/worker.c - apply_handle_stream_abort + /* + * We are in main apply worker and the transaction has been serialized + * to file. + */ + else + serialize_stream_abort(xid, subxid); I think this will be more readable if written like: else { /* put comment here... */ serialize_stream_abort(xid, subxid); } ~~~ 1.34 src/backend/replication/logical/worker.c - apply_dispatch - /* * Logical replication protocol message dispatcher. */ -static void +void apply_dispatch(StringInfo s) Maybe removing the whitespace is not really needed as part of this patch? ====== 1.35 src/include/catalog/pg_subscription.h +/* Disallow streaming in-progress transactions */ +#define SUBSTREAM_OFF 'f' + +/* + * Streaming transactions are written to a temporary file and applied only + * after the transaction is committed on upstream. + */ +#define SUBSTREAM_ON 't' + +/* Streaming transactions are applied immediately via a background worker */ +#define SUBSTREAM_PARALLEL 'p' + 1.35a Should all these "Streaming transactions" be called "Streaming in-progress transactions"? 1.35b. Either align the values or don’t. Currently, they seem half-aligned. 1.35c. SUGGESTION (modify the 1st comment to be more consistent with the others) Streaming in-progress transactions are disallowed. ====== 1.36 src/include/replication/worker_internal.h extern int logicalrep_sync_worker_count(Oid subid); +extern int logicalrep_apply_background_worker_count(Oid subid); Just wondering if this should be called "logicalrep_apply_bgworker_count(Oid subid);" for consistency with the other function naming. ======== v14-0002 ======== 2.1 Commit message Change all TAP tests using the SUBSCRIPTION "streaming" option, so they now test both 'on' and 'parallel' values. "option" -> "parameter" ------ [1] https://www.postgresql.org/message-id/OS3PR01MB6275DCCDF35B3BBD52CA02CC9EB89%40OS3PR01MB6275.jpnprd0... Kind Regards, Peter Smith. Fujitsu Australia ^ permalink raw reply [nested|flat] 66+ messages in thread
* Re: Perform streaming logical transactions by background workers and parallel apply @ 2022-07-01 09:43 Amit Kapila <[email protected]> parent: Peter Smith <[email protected]> 1 sibling, 1 reply; 66+ messages in thread From: Amit Kapila @ 2022-07-01 09:43 UTC (permalink / raw) To: Peter Smith <[email protected]>; +Cc: [email protected] <[email protected]>; Masahiko Sawada <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; PostgreSQL Hackers <[email protected]> On Fri, Jul 1, 2022 at 12:13 PM Peter Smith <[email protected]> wrote: > > ====== > > 1.2 doc/src/sgml/protocol.sgml - Protocol constants > > Previously I wrote that since there are protocol changes here, > shouldn’t there also be some corresponding LOGICALREP_PROTO_XXX > constants and special checking added in the worker.c? > > But you said [1 comment #6] you think it is OK because... > > IMO, I still disagree with the reply. The fact is that the protocol > *has* been changed, so IIUC that is precisely the reason for having > those protocol constants. > > e.g I am guessing you might assign the new one somewhere here: > -- > server_version = walrcv_server_version(LogRepWorkerWalRcvConn); > options.proto.logical.proto_version = > server_version >= 150000 ? LOGICALREP_PROTO_TWOPHASE_VERSION_NUM : > server_version >= 140000 ? LOGICALREP_PROTO_STREAM_VERSION_NUM : > LOGICALREP_PROTO_VERSION_NUM; > -- > > And then later you would refer to this new protocol version (instead > of the server version) when calling to the apply_handle_stream_abort > function. > > ====== > One point related to this that occurred to me is how it will behave if the publisher is of version >=16 whereas the subscriber is of versions <=15? Won't in that case publisher sends the new fields but subscribers won't be reading those which may cause some problems. > ====== > > 1.5 src/backend/commands/subscriptioncmds.c > > + /* > + * If no parameter given, assume "true" is meant. > + */ > > Previously I suggested an update for this comment, but it was rejected > [1 comment #12] saying you wanted consistency with defGetBoolean. > > Sure, that is one point of view. Another one is that "two wrongs don't > make a right". IIUC that comment as it currently stands is incorrect > because in this case there *is* a parameter given - it is just the > parameter *value* that is missing. > You have a point but if we see this function in the vicinity then the proposed comment also makes sense. -- With Regards, Amit Kapila. ^ permalink raw reply [nested|flat] 66+ messages in thread
* Re: Perform streaming logical transactions by background workers and parallel apply @ 2022-07-04 04:11 Peter Smith <[email protected]> parent: [email protected] <[email protected]> 4 siblings, 1 reply; 66+ messages in thread From: Peter Smith @ 2022-07-04 04:11 UTC (permalink / raw) To: [email protected] <[email protected]>; +Cc: Amit Kapila <[email protected]>; Masahiko Sawada <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; PostgreSQL Hackers <[email protected]> Below are some review comments for patch v14-0003: ======== v14-0003 ======== 3.1 Commit message If any of the following checks are violated, an error will be reported. 1. The unique columns between publisher and subscriber are difference. 2. There is any non-immutable function present in expression in subscriber's relation. Check from the following 4 items: a. The function in triggers; b. Column default value expressions and domain constraints; c. Constraint expressions. d. The foreign keys. SUGGESTION (rewording to match the docs and the code). Add some checks before using apply background worker to apply changes. streaming=parallel mode has two requirements: 1) The unique columns must be the same between publisher and subscriber 2) There cannot be any non-immutable functions in the subscriber-side replicated table. Look for functions in the following places: * a. Trigger functions * b. Column default value expressions and domain constraints * c. Constraint expressions * d. Foreign keys ====== 3.2 doc/src/sgml/ref/create_subscription.sgml + To run in this mode, there are following two requirements. The first + is that the unique column should be the same between publisher and + subscriber; the second is that there should not be any non-immutable + function in subscriber-side replicated table. SUGGESTION Parallel mode has two requirements: 1) the unique columns must be the same between publisher and subscriber; 2) there cannot be any non-immutable functions in the subscriber-side replicated table. ====== 3.3 .../replication/logical/applybgworker.c - apply_bgworker_relation_check + * Check if changes on this logical replication relation can be applied by + * apply background worker. SUGGESTION Check if changes on this relation can be applied by an apply background worker. ~~~ 3.4 + * Although we maintains the commit order by allowing only one process to + * commit at a time, our access order to the relation has changed. SUGGESTION Although the commit order is maintained only allowing one process to commit at a time, the access order to the relation has changed. ~~~ 3.5 + /* Check only we are in apply bgworker. */ + if (!am_apply_bgworker()) + return; SUGGESTION /* Skip check if not an apply background worker. */ ~~~ 3.6 + /* + * If it is a partitioned table, we do not check it, we will check its + * partition later. + */ This comment is lacking useful details: /* Partition table checks are done later in (?????) */ ~~~ 3.7 + if (!rel->sameunique) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot replicate relation with different unique index"), + errhint("Please change the streaming option to 'on' instead of 'parallel'."))); Maybe the first message should change slightly so it is worded consistently with the other one. SUGGESTION errmsg("cannot replicate relation. Unique indexes must be the same"), ====== 3.8 src/backend/replication/logical/proto.c -#define LOGICALREP_IS_REPLICA_IDENTITY 1 +#define LOGICALREP_IS_REPLICA_IDENTITY 0x0001 +#define LOGICALREP_IS_UNIQUE 0x0002 I think these constants should named differently to reflect that they are just attribute flags. They should should use similar bitset styles to the other nearby constants. SUGGESTION #define ATTR_IS_REPLICA_IDENTITY (1 << 0) #define ATTR_IS_UNIQUE (1 << 1) ~~~ 3.9 src/backend/replication/logical/proto.c - logicalrep_write_attrs This big slab of new code to get the BMS looks very similar to RelationGetIdentityKeyBitmap. So perhaps this code should be encapsulated in another function like that one (in relcache.c?) and then just called from logicalrep_write_attrs ====== 3.10 src/backend/replication/logical/relation.c - logicalrep_relmap_reset_volatility_cb +/* + * Reset the flag volatility of all existing entry in the relation map cache. + */ +static void +logicalrep_relmap_reset_volatility_cb(Datum arg, int cacheid, uint32 hashvalue) SUGGESTION Reset the volatility flag of all entries in the relation map cache. ~~~ 3.11 src/backend/replication/logical/relation.c - logicalrep_rel_mark_safe_in_apply_bgworker +/* + * Check if unique index/constraint matches and mark sameunique and volatility + * flag. + * + * Don't throw any error here just mark the relation entry as not sameunique or + * FUNCTION_NONIMMUTABLE as we only check these in apply background worker. + */ +static void +logicalrep_rel_mark_safe_in_apply_bgworker(LogicalRepRelMapEntry *entry) SUGGESTION Check if unique index/constraint matches and assign 'sameunique' flag. Check if there are any non-immutable functions and assign the 'volatility' flag. Note: Don't throw any error here - these flags will be checked in the apply background worker. ~~~ 3.12 src/backend/replication/logical/relation.c - logicalrep_rel_mark_safe_in_apply_bgworker I did not really understand why you used an enum for one flag (volatility) but not the other one (sameunique); shouldn’t both of these be tri-values: unknown/yes/no? For E.g. there is a quick exit from this function if the FUNCTION_UNKNOWN, but there is no equivalent quick exit for the sameunique? It seems inconsistent. ~~~ 3.13 src/backend/replication/logical/relation.c - logicalrep_rel_mark_safe_in_apply_bgworker + /* + * Check whether there is any non-immutable function in the local table. + * + * a. The function in triggers; + * b. Column default value expressions and domain constraints; + * c. Constraint expressions; + * d. Foreign keys. + */ SUGGESTION * Check if there is any non-immutable function in the local table. * Look for functions in the following places: * a. trigger functions * b. Column default value expressions and domain constraints * c. Constraint expressions * d. Foreign keys ~~~ 3.14 src/backend/replication/logical/relation.c - logicalrep_rel_mark_safe_in_apply_bgworker There are lots of places setting FUNCTION_NONIMMUTABLE, so I think this code might be tidier if you just have a single return at the end of this function and 'goto' it. e.g. if (...) goto function_not_immutable; ... return; function_not_immutable: entry->volatility = FUNCTION_NONIMMUTABLE; ====== 3.15 src/backend/replication/logical/worker.c - apply_handle_stream_stop + /* + * Unlike stream_commit, we don't need to wait here for stream_stop to + * finish. Allowing the other transaction to be applied before stream_stop + * is finished can only lead to failures if the unique index/constraint is + * different between publisher and subscriber. But for such cases, we don't + * allow streamed transactions to be applied in parallel. See + * apply_bgworker_relation_check. + */ "can only lead to failures" -> "can lead to failures" ~~~ 3.16 src/backend/replication/logical/worker.c - apply_handle_tuple_routing @@ -2534,13 +2548,14 @@ apply_handle_tuple_routing(ApplyExecutionData *edata, } MemoryContextSwitchTo(oldctx); + part_entry = logicalrep_partition_open(relmapentry, partrel, + attrmap); + + apply_bgworker_relation_check(part_entry); + /* Check if we can do the update or delete on the leaf partition. */ if (operation == CMD_UPDATE || operation == CMD_DELETE) - { - part_entry = logicalrep_partition_open(relmapentry, partrel, - attrmap); check_relation_updatable(part_entry); - } Perhaps the apply_bgworker_relation_check(part_entry); should be done AFTER the CMD_UPDATE/CMD_DELETE check because then it will not change the existing errors for those cases. ====== 3.17 src/backend/utils/cache/typcache.c +/* + * GetDomainConstraints --- get DomainConstraintState list of specified domain type + */ +List * +GetDomainConstraints(Oid type_id) This is an unusual-looking function header comment, with the function name and the "---". ====== 3.18 src/include/replication/logicalrelation.h +/* + * States to determine volatility of the function in expressions in one + * relation. + */ +typedef enum RelFuncVolatility +{ + FUNCTION_UNKNOWN = 0, /* initializing */ + FUNCTION_IMMUTABLE, /* all functions are immutable function */ + FUNCTION_NONIMMUTABLE /* at least one non-immutable function */ +} RelFuncVolatility; + I think the comments can be improved, and also the values can be more self-explanatory. e.g. typedef enum RelFuncVolatility { FUNCTION_UNKNOWN_IMMUATABLE, /* unknown */ FUNCTION_ALL_MUTABLE, /* all functions are immutable */ FUNCTION_NOT_ALL_IMMUTABLE /* not all functions are immuatble */ } RelFuncVolatility; ~~~ 3.18 RelFuncVolatility should be added to typedefs.list ~~~ 3.19 @@ -31,6 +42,11 @@ typedef struct LogicalRepRelMapEntry Relation localrel; /* relcache entry (NULL when closed) */ AttrMap *attrmap; /* map of local attributes to remote ones */ bool updatable; /* Can apply updates/deletes? */ + bool sameunique; /* Are all unique columns of the local + relation contained by the unique columns in + remote? */ (This is similar to review comment 3.12) I felt it was inconsistent for this to be a boolean but for the 'volatility' member to be an enum. AFAIK these 2 flags are similar kinds – e.g. essentially tri-state flags unknown/true/false so I thought they should be treated the same. E.g. both enums? ~~~ 3.20 + RelFuncVolatility volatility; /* all functions in localrel are + immutable function? */ SUGGESTION /* Indicator of local relation function volatility */ ====== 3.21 .../subscription/t/022_twophase_cascade.pl + if ($streaming_mode eq 'parallel') + { + $node_C->safe_psql( + 'postgres', " + ALTER TABLE test_tab ALTER c DROP DEFAULT"); + } + Indentation of the ALTER does not seem right. ====== 3.22 .../subscription/t/032_streaming_apply.pl 3.22.a +# Setup structure on publisher "structure"? 3.22.b +# Setup structure on subscriber "structure"? ~~~ 3.23 +# Check that a background worker starts if "streaming" option is specified as +# "parallel". We have to look for the DEBUG1 log messages about that, so +# temporarily bump up the log verbosity. +$node_subscriber->append_conf('postgresql.conf', "log_min_messages = debug1"); +$node_subscriber->reload; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); +$node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); +$node_subscriber->reload; I didn't really think it was necessary to bump this log level, and to verify that the bgworker is started, because this test is anyway going to ensure that the ERROR "cannot replicate relation with different unique index" happens, so that is already implicitly ensuring the bgworker was used. ~~~ 3.24 +# Then we check the unique index on partition table. +$node_subscriber->safe_psql( + 'postgres', qq{ +CREATE TRIGGER insert_trig +BEFORE INSERT ON test_tab_partition +FOR EACH ROW EXECUTE PROCEDURE trigger_func(); +ALTER TABLE test_tab_partition ENABLE REPLICA TRIGGER insert_trig; +}); Looks like the wrong comment. I think it should say something like "Check the trigger on the partition table." ------ Kind Regards, Peter Smith. Fujitsu Australia ^ permalink raw reply [nested|flat] 66+ messages in thread
* Re: Perform streaming logical transactions by background workers and parallel apply @ 2022-07-04 06:47 Peter Smith <[email protected]> parent: [email protected] <[email protected]> 4 siblings, 1 reply; 66+ messages in thread From: Peter Smith @ 2022-07-04 06:47 UTC (permalink / raw) To: [email protected] <[email protected]>; +Cc: Amit Kapila <[email protected]>; Masahiko Sawada <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; PostgreSQL Hackers <[email protected]> Below are some review comments for patch v14-0004: ======== v14-0004 ======== 4.0 General. This comment is an after-thought but as I write this mail I am wondering if most of this 0004 patch is even necessary at all? Instead of introducing a new column and all the baggage that goes with it, can't the same functionality be achieved just by toggling the streaming mode 'substream' value from 'p' (parallel) to 't' (on) whenever an error occurs causing a retry? Anyway, if you do change it this way then most of the following comments can be disregarded. ====== 4.1 Commit message Patch needs an explanatory commit message. Currently, there is nothing. ====== 4.2 doc/src/sgml/catalogs.sgml + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>subretry</structfield> <type>bool</type> + </para> + <para> + If true, the subscription will not try to apply streaming transaction + in <literal>parallel</literal> mode. See + <xref linkend="sql-createsubscription"/> for more information. + </para></entry> + </row> I think it is overkill to mention anything about the streaming=parallel here because IIUC it is nothing to do with this field at all. I thought you really only need to say something brief like: SUGGESTION: True if the previous apply change failed and a retry was required. ====== 4.3 doc/src/sgml/ref/create_subscription.sgml @@ -244,6 +244,10 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl is that the unique column should be the same between publisher and subscriber; the second is that there should not be any non-immutable function in subscriber-side replicated table. + When applying a streaming transaction, if either requirement is not + met, the background worker will exit with an error. And when retrying + later, we will try to apply this transaction in <literal>on</literal> + mode. </para> I did not think it is good to say "we" in the docs. SUGGESTION When applying a streaming transaction, if either requirement is not met, the background worker will exit with an error. Parallel mode is disregarded when retrying; instead the transaction will be applied using <literal>streaming = on</literal>. ====== 4.4 .../replication/logical/applybgworker.c + /* + * We don't start new background worker if retry was set as it's possible + * that the last time we tried to apply a transaction in background worker + * and the check failed (see function apply_bgworker_relation_check). So + * we will try to apply this transaction in apply worker. + */ SUGGESTION (simplified, and remove "we") Don't use apply background workers for retries, because it is possible that the last time we tried to apply a transaction using an apply background worker the checks failed (see function apply_bgworker_relation_check). ~~~ 4.5 + elog(DEBUG1, "retry to apply an streaming transaction in apply " + "background worker"); IMO the log message is too confusing SUGGESTION "apply background workers are not used for retries" ====== 4.6 src/backend/replication/logical/worker.c 4.6.a - apply_handle_commit + /* Set the flag that we will not retry later. */ + set_subscription_retry(false); But the comment is wrong, isn't it? Shouldn’t it just say that we are not *currently* retrying? And can’t this just anyway be redundant if only the catalog column has a DEFAULT value of false? 4.6.b - apply_handle_prepare Ditto 4.6.c - apply_handle_commit_prepared Ditto 4.6.d - apply_handle_rollback_prepared Ditto 4.6.e - apply_handle_stream_prepare Ditto 4.6.f - apply_handle_stream_abort Ditto 4.6.g - apply_handle_stream_commit Ditto ~~~ 4.7 src/backend/replication/logical/worker.c 4.7.a - start_table_sync @@ -3894,6 +3917,9 @@ start_table_sync(XLogRecPtr *origin_startpos, char **myslotname) } PG_CATCH(); { + /* Set the flag that we will retry later. */ + set_subscription_retry(true); Maybe this should say more like "Flag that the next apply will be the result of a retry" 4.7.b - start_apply Ditto ~~~ 4.8 src/backend/replication/logical/worker.c - set_subscription_retry + +/* + * Set subretry of pg_subscription catalog. + * + * If retry is true, subscriber is about to exit with an error. Otherwise, it + * means that the changes was applied successfully. + */ +static void +set_subscription_retry(bool retry) "changes" -> "change" ? ~~~ 4.8 src/backend/replication/logical/worker.c - set_subscription_retry Isn't this flag only every used when streaming=parallel? But it does not seem ot be checking that anywhere before potentiall executing all this code when maybe will never be used. ====== 4.9 src/include/catalog/pg_subscription.h @@ -76,6 +76,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW bool subdisableonerr; /* True if a worker error should cause the * subscription to be disabled */ + bool subretry; /* True if the previous apply change failed. */ I was wondering if you can give this column a DEFAULT value of false, because then perhaps most of the patch code from worker.c may be able to be eliminated. ====== 4.10 .../subscription/t/032_streaming_apply.pl I felt that the test cases all seem to blend together. IMO it will be more readable if the main text parts are visually separated e.g using a comment like: # ================================================= ------ Kind Regards, Peter Smith. Fujitsu Australia ^ permalink raw reply [nested|flat] 66+ messages in thread
* RE: Perform streaming logical transactions by background workers and parallel apply @ 2022-07-07 03:31 [email protected] <[email protected]> parent: [email protected] <[email protected]> 4 siblings, 1 reply; 66+ messages in thread From: [email protected] @ 2022-07-07 03:31 UTC (permalink / raw) To: [email protected] <[email protected]>; Amit Kapila <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Peter Smith <[email protected]>; [email protected] <[email protected]>; PostgreSQL Hackers <[email protected]> On Tue, Jun 28, 2022 11:22 AM Wang, Wei/王 威 <[email protected]> wrote: > > I also improved patches as suggested by Peter-san in [1] and [2]. > Thanks for Shi Yu to improve the patches by addressing the comments in [2]. > > Attach the new patches. > Thanks for updating the patch. Here are some comments. 0001 patch ============== 1. + /* Check If there are free worker slot(s) */ + LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); I think "Check If" should be "Check if". 0003 patch ============== 1. Should we call apply_bgworker_relation_check() in apply_handle_truncate()? 0004 patch ============== 1. @@ -3932,6 +3958,9 @@ start_apply(XLogRecPtr origin_startpos) } PG_CATCH(); { + /* Set the flag that we will retry later. */ + set_subscription_retry(true); + if (MySubscription->disableonerr) DisableSubscriptionAndExit(); Else I think we need to emit the error and recover from the error state before setting the retry flag, like what we do in DisableSubscriptionAndExit(). Otherwise if an error is detected when setting the retry flag, we won't get the error message reported by the apply worker. Regards, Shi yu ^ permalink raw reply [nested|flat] 66+ messages in thread
* RE: Perform streaming logical transactions by background workers and parallel apply @ 2022-07-07 03:44 [email protected] <[email protected]> parent: Peter Smith <[email protected]> 1 sibling, 1 reply; 66+ messages in thread From: [email protected] @ 2022-07-07 03:44 UTC (permalink / raw) To: Peter Smith <[email protected]>; +Cc: Amit Kapila <[email protected]>; Masahiko Sawada <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; PostgreSQL Hackers <[email protected]> On Fri, Jul 1, 2022 at 14:43 PM Peter Smith <[email protected]> wrote: > Below are some review comments for patches v14-0001, and v14-0002: Thanks for your comments. > 1.10 .../replication/logical/applybgworker.c - apply_bgworker_find > > + /* > + * Find entry for requested transaction. > + */ > + entry = hash_search(ApplyWorkersHash, &xid, HASH_FIND, &found); > + if (found) > + { > + entry->wstate->pstate->status = APPLY_BGWORKER_BUSY; > + return entry->wstate; > + } > + else > + return NULL; > +} > > IMO it is an unexpected side-effect for the function called "find" to > be also modifying the thing that it found. IMO this setting BUSY > should either be done by the caller, or else this function name should > be renamed to make it obvious that this is doing more than just > "finding" something. Since we set the state to BUSY in the function apply_bgworker_start and the state is not modified (set to FINISHED) until the transaction completes, I think we do not need to set this state to BUSY again in the function apply_bgworker_find during applying the transaction. So I removed it and invoked function Assert. I also invoked function Assert in function apply_bgworker_start. > 1.16. src/backend/replication/logical/launcher.c - logicalrep_worker_launch > > + bool is_subworker = (subworker_dsm != DSM_HANDLE_INVALID); > + > + /* We don't support table sync in subworker */ > + Assert(!(is_subworker && OidIsValid(relid))); > > I'm not sure the comment is good. It sounds like it is something that > might be possible but is just current "not supported". In fact, I > thought this is really just a sanity check because the combination of > those params is just plain wrong isn't it? Maybe a better comment is > just: > /* Sanity check */ Improved this comment as following: ``` /* Sanity check : we don't support table sync in subworker. */ ``` > 1.22 src/backend/replication/logical/worker.c - skip_xact_finish_lsn > > /* > * We enable skipping all data modification changes (INSERT, UPDATE, etc.) for > * the subscription if the remote transaction's finish LSN matches > the subskiplsn. > * Once we start skipping changes, we don't stop it until we skip all > changes of > * the transaction even if pg_subscription is updated and > MySubscription->skiplsn > - * gets changed or reset during that. Also, in streaming transaction cases, we > - * don't skip receiving and spooling the changes since we decide whether or not > + * gets changed or reset during that. Also, in streaming transaction > cases (streaming = on), > + * we don't skip receiving and spooling the changes since we decide > whether or not > * to skip applying the changes when starting to apply changes. The > subskiplsn is > * cleared after successfully skipping the transaction or applying non-empty > * transaction. The latter prevents the mistakenly specified subskiplsn from > - * being left. > + * being left. Note that we cannot skip the streaming transaction in parallel > + * mode, because we cannot get the finish LSN before applying the changes. > */ > > "in parallel mode, because" -> "in 'streaming = parallel' mode, because" Not sure about this. > 1.28 src/backend/replication/logical/worker.c - apply_handle_stream_prepare > > + if (wstate) > + { > + apply_bgworker_send_data(wstate, s->len, s->data); > + > + /* > + * Wait for apply background worker to finish. This is required to > + * maintain commit order which avoids failures due to transaction > + * dependencies and deadlocks. > + */ > + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_FINISHED); > + apply_bgworker_free(wstate); > > I think maybe the comment can be changed slightly, and then it can > move up one line to the top of this code block (above the 3 > statements). I think it will become more readable. > > SUGGESTION > After sending the data to the apply background worker, wait for that > worker to finish. This is necessary to maintain commit order which > avoids failures due to transaction dependencies and deadlocks. I think it might be better to add a new comment before invoking function apply_bgworker_send_data. Improve the comments as you suggested. I improved this point in function apply_handle_stream_prepare, apply_handle_stream_abort and apply_handle_stream_commit. What do you think about changing it like this: ``` /* Send STREAM PREPARE message to the apply background worker. */ apply_bgworker_send_data(wstate, s->len, s->data); /* * After sending the data to the apply background worker, wait for * that worker to finish. This is necessary to maintain commit * order which avoids failures due to transaction dependencies and * deadlocks. */ apply_bgworker_wait_for(wstate, APPLY_BGWORKER_FINISHED); ``` > 1.34 src/backend/replication/logical/worker.c - apply_dispatch > > - > /* > * Logical replication protocol message dispatcher. > */ > -static void > +void > apply_dispatch(StringInfo s) > > Maybe removing the whitespace is not really needed as part of this patch? Yes, this change is not necessary for this patch. But since this change does not involve the modification of comments and actual code, it just adjusts the blank line between the function modified by this patch and the previous function, so I think it is okay in this patch. > 2.1 Commit message > > Change all TAP tests using the SUBSCRIPTION "streaming" option, so they > now test both 'on' and 'parallel' values. > > "option" -> "parameter" Sorry I missed this point when I was merging the patches. I merged this change in v15. Attach the new patches. Also improved the patches as suggested in [1], [2] and [3]. [1] - https://www.postgresql.org/message-id/CAA4eK1KgovaRcbSuzzWki1HVso6oLAdZ2aPr1nWxX1x%3DVDBQJg%40mail.g... [2] - https://www.postgresql.org/message-id/CAHut%2BPtRNAOwFtBp_TnDWdC7UpcTxPJzQnrm%3DNytN7cVBt5zRQ%40mail... [3] - https://www.postgresql.org/message-id/CAHut%2BPvrw%2BtgCEYGxv%2BnKrqg-zbJdYEXee6o4irPAsYoXcuUcw%40ma... Regards, Wang wei Attachments: [application/octet-stream] v15-0001-Perform-streaming-logical-transactions-by-backgr.patch (105.6K, ../../OS3PR01MB62755C6C9A75EB09F7218B589E839@OS3PR01MB6275.jpnprd01.prod.outlook.com/2-v15-0001-Perform-streaming-logical-transactions-by-backgr.patch) download | inline diff: From 10ac62669db230d73f5e0f65fb52d01649b1ed6c Mon Sep 17 00:00:00 2001 From: "houzj.fnst" <[email protected]> Date: Wed, 20 Apr 2022 16:45:07 +0800 Subject: [PATCH v15 1/4] Perform streaming logical transactions by background workers Currently, for large transactions, the publisher sends the data in multiple streams (changes divided into chunks depending upon logical_decoding_work_mem), and then on the subscriber-side, the apply worker writes the changes into temporary files and once it receives the commit, it reads from the file and applies the entire transaction. To improve the performance of such transactions, we can instead allow them to be applied via background workers. In this approach, we assign a new apply background worker (if available) as soon as the xact's first stream is received and the main apply worker will send changes to this new worker via shared memory. The apply background worker will directly apply the change instead of writing it to temporary files. We keep this worker assigned till the transaction commit is received and also wait for the worker to finish at commit. This preserves commit ordering and avoids writing to and reading from file in most cases. We still need to spill if there is no worker available. This patch also extends the SUBSCRIPTION 'streaming' parameter so that the user can control whether to apply the streaming transaction in an apply background worker or spill the change to disk. The user can set the streaming parameter to 'on/off', 'parallel'. The parameter value 'parallel' means the streaming will be applied via an apply background worker, if available. The parameter value 'on' means the streaming transaction will be spilled to disk. The default value is 'off' (same as current behaviour). --- doc/src/sgml/catalogs.sgml | 10 +- doc/src/sgml/config.sgml | 25 + doc/src/sgml/logical-replication.sgml | 10 + doc/src/sgml/protocol.sgml | 19 + doc/src/sgml/ref/create_subscription.sgml | 24 +- src/backend/access/transam/xact.c | 13 + src/backend/commands/subscriptioncmds.c | 66 +- src/backend/postmaster/bgworker.c | 3 + src/backend/replication/logical/Makefile | 1 + .../replication/logical/applybgworker.c | 775 ++++++++++++++++++ src/backend/replication/logical/decode.c | 10 +- src/backend/replication/logical/launcher.c | 130 ++- src/backend/replication/logical/origin.c | 26 +- src/backend/replication/logical/proto.c | 41 +- .../replication/logical/reorderbuffer.c | 10 +- src/backend/replication/logical/tablesync.c | 10 +- src/backend/replication/logical/worker.c | 688 ++++++++++++---- src/backend/replication/pgoutput/pgoutput.c | 9 +- src/backend/utils/activity/wait_event.c | 3 + src/backend/utils/misc/guc.c | 12 + src/backend/utils/misc/postgresql.conf.sample | 1 + src/bin/pg_dump/pg_dump.c | 6 +- src/include/catalog/pg_subscription.h | 21 +- src/include/replication/logicallauncher.h | 1 + src/include/replication/logicalproto.h | 27 +- src/include/replication/logicalworker.h | 1 + src/include/replication/origin.h | 2 +- src/include/replication/reorderbuffer.h | 7 +- src/include/replication/worker_internal.h | 102 ++- src/include/utils/wait_event.h | 1 + src/test/regress/expected/subscription.out | 2 +- src/tools/pgindent/typedefs.list | 5 + 32 files changed, 1841 insertions(+), 220 deletions(-) create mode 100644 src/backend/replication/logical/applybgworker.c diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index 4f3f375a84..815cae6082 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -7873,11 +7873,15 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l <row> <entry role="catalog_table_entry"><para role="column_definition"> - <structfield>substream</structfield> <type>bool</type> + <structfield>substream</structfield> <type>char</type> </para> <para> - If true, the subscription will allow streaming of in-progress - transactions + Controls how to handle the streaming of in-progress transactions: + <literal>f</literal> = disallow streaming of in-progress transactions, + <literal>t</literal> = spill the changes of in-progress transactions to + disk and apply at once after the transaction is committed on the + publisher, + <literal>p</literal> = apply changes directly using a background worker </para></entry> </row> diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml index 37fd80388c..6bbd986195 100644 --- a/doc/src/sgml/config.sgml +++ b/doc/src/sgml/config.sgml @@ -4970,6 +4970,31 @@ ANY <replaceable class="parameter">num_sync</replaceable> ( <replaceable class=" </listitem> </varlistentry> + <varlistentry id="guc-max-apply-bgworkers-per-subscription" xreflabel="max_apply_bgworkers_per_subscription"> + <term><varname>max_apply_bgworkers_per_subscription</varname> (<type>integer</type>) + <indexterm> + <primary><varname>max_apply_bgworkers_per_subscription</varname> configuration parameter</primary> + </indexterm> + </term> + <listitem> + <para> + Maximum number of apply background workers per subscription. This + parameter controls the amount of parallelism of the streaming of + in-progress transactions when subscription parameter + <literal>streaming = parallel</literal>. + </para> + <para> + The apply background workers are taken from the pool defined by + <varname>max_logical_replication_workers</varname>. + </para> + <para> + The default value is 2. This parameter can only be set in the + <filename>postgresql.conf</filename> file or on the server command + line. + </para> + </listitem> + </varlistentry> + </variablelist> </sect2> diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml index bdf1e7b727..92997f9299 100644 --- a/doc/src/sgml/logical-replication.sgml +++ b/doc/src/sgml/logical-replication.sgml @@ -1153,6 +1153,16 @@ CONTEXT: processing remote data for replication origin "pg_16395" during "INSER might not violate any constraint. This can easily make the subscriber inconsistent. </para> + + <para> + When the streaming mode is <literal>parallel</literal>, the finish LSN of + failed transactions may not be logged. In that case, it may be necessary to + change the streaming mode to <literal>on</literal> and cause the same + conflicts again so the finish LSN of the failed transaction will be written + to the server log. For the usage of finish LSN, please refer to <link + linkend="sql-altersubscription"><command>ALTER SUBSCRIPTION ... + SKIP</command></link>. + </para> </sect1> <sect1 id="logical-replication-restrictions"> diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml index c0b89a3c01..7e88ba9631 100644 --- a/doc/src/sgml/protocol.sgml +++ b/doc/src/sgml/protocol.sgml @@ -6809,6 +6809,25 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;" </listitem> </varlistentry> + <varlistentry> + <term>Int64 (XLogRecPtr)</term> + <listitem> + <para> + The LSN of the abort. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Int64 (TimestampTz)</term> + <listitem> + <para> + Abort timestamp of the transaction. The value is in number + of microseconds since PostgreSQL epoch (2000-01-01). + </para> + </listitem> + </varlistentry> + <varlistentry> <term>Int32 (TransactionId)</term> <listitem> diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml index 34b3264b26..71dd4aca81 100644 --- a/doc/src/sgml/ref/create_subscription.sgml +++ b/doc/src/sgml/ref/create_subscription.sgml @@ -217,13 +217,29 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl </varlistentry> <varlistentry> - <term><literal>streaming</literal> (<type>boolean</type>)</term> + <term><literal>streaming</literal> (<type>enum</type>)</term> <listitem> <para> Specifies whether to enable streaming of in-progress transactions - for this subscription. By default, all transactions - are fully decoded on the publisher and only then sent to the - subscriber as a whole. + for this subscription. The default value is <literal>off</literal>, + meaning all transactions are fully decoded on the publisher and only + then sent to the subscriber as a whole. + </para> + + <para> + If set to <literal>on</literal>, the incoming changes are written to + temporary files and then applied only after the transaction is + committed on the publisher. + </para> + + <para> + If set to <literal>parallel</literal>, incoming changes are directly + applied via one of the apply background workers, if available. If no + background worker is free to handle streaming transaction then the + changes are written to temporary files and applied after the + transaction is committed. Note that if an error happens when + applying changes in a background worker, the finish LSN of the + remote transaction might not be reported in the server log. </para> </listitem> </varlistentry> diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c index 116de1175b..3e61a57b50 100644 --- a/src/backend/access/transam/xact.c +++ b/src/backend/access/transam/xact.c @@ -1711,6 +1711,7 @@ RecordTransactionAbort(bool isSubXact) int nchildren; TransactionId *children; TimestampTz xact_time; + bool replorigin; /* * If we haven't been assigned an XID, nobody will care whether we aborted @@ -1741,6 +1742,13 @@ RecordTransactionAbort(bool isSubXact) elog(PANIC, "cannot abort transaction %u, it was already committed", xid); + /* + * Are we using the replication origins feature? Or, in other words, + * are we replaying remote actions? + */ + replorigin = (replorigin_session_origin != InvalidRepOriginId && + replorigin_session_origin != DoNotReplicateId); + /* Fetch the data we need for the abort record */ nrels = smgrGetPendingDeletes(false, &rels); nchildren = xactGetCommittedChildren(&children); @@ -1765,6 +1773,11 @@ RecordTransactionAbort(bool isSubXact) MyXactFlags, InvalidTransactionId, NULL); + if (replorigin) + /* Move LSNs forward for this replication origin */ + replorigin_session_advance(replorigin_session_origin_lsn, + XactLastRecEnd); + /* * Report the latest async abort LSN, so that the WAL writer knows to * flush this abort. There's nothing to be gained by delaying this, since diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c index bdc1208724..5f349067cc 100644 --- a/src/backend/commands/subscriptioncmds.c +++ b/src/backend/commands/subscriptioncmds.c @@ -83,7 +83,7 @@ typedef struct SubOpts bool copy_data; bool refresh; bool binary; - bool streaming; + char streaming; bool twophase; bool disableonerr; XLogRecPtr lsn; @@ -95,6 +95,62 @@ static List *merge_publications(List *oldpublist, List *newpublist, bool addpub, static void ReportSlotConnectionError(List *rstates, Oid subid, char *slotname, char *err); +/* + * Extract the streaming mode value from a DefElem. This is like + * defGetBoolean() but also accepts the special value of "parallel". + */ +static char +defGetStreamingMode(DefElem *def) +{ + /* + * If no value given, assume "true" is meant. + */ + if (def->arg == NULL) + return SUBSTREAM_ON; + + /* + * Allow 0, 1, "false", "true", "off", "on" or "parallel". + */ + switch (nodeTag(def->arg)) + { + case T_Integer: + switch (intVal(def->arg)) + { + case 0: + return SUBSTREAM_OFF; + case 1: + return SUBSTREAM_ON; + default: + /* otherwise, error out below */ + break; + } + break; + default: + { + char *sval = defGetString(def); + + /* + * The set of strings accepted here should match up with the + * grammar's opt_boolean_or_string production. + */ + if (pg_strcasecmp(sval, "false") == 0 || + pg_strcasecmp(sval, "off") == 0) + return SUBSTREAM_OFF; + if (pg_strcasecmp(sval, "true") == 0 || + pg_strcasecmp(sval, "on") == 0) + return SUBSTREAM_ON; + if (pg_strcasecmp(sval, "parallel") == 0) + return SUBSTREAM_PARALLEL; + } + break; + } + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s requires a Boolean value or \"parallel\"", + def->defname))); + return SUBSTREAM_OFF; /* keep compiler quiet */ +} + /* * Common option parsing function for CREATE and ALTER SUBSCRIPTION commands. * @@ -132,7 +188,7 @@ parse_subscription_options(ParseState *pstate, List *stmt_options, if (IsSet(supported_opts, SUBOPT_BINARY)) opts->binary = false; if (IsSet(supported_opts, SUBOPT_STREAMING)) - opts->streaming = false; + opts->streaming = SUBSTREAM_OFF; if (IsSet(supported_opts, SUBOPT_TWOPHASE_COMMIT)) opts->twophase = false; if (IsSet(supported_opts, SUBOPT_DISABLE_ON_ERR)) @@ -233,7 +289,7 @@ parse_subscription_options(ParseState *pstate, List *stmt_options, errorConflictingDefElem(defel, pstate); opts->specified_opts |= SUBOPT_STREAMING; - opts->streaming = defGetBoolean(defel); + opts->streaming = defGetStreamingMode(defel); } else if (strcmp(defel->defname, "two_phase") == 0) { @@ -600,7 +656,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt, values[Anum_pg_subscription_subowner - 1] = ObjectIdGetDatum(owner); values[Anum_pg_subscription_subenabled - 1] = BoolGetDatum(opts.enabled); values[Anum_pg_subscription_subbinary - 1] = BoolGetDatum(opts.binary); - values[Anum_pg_subscription_substream - 1] = BoolGetDatum(opts.streaming); + values[Anum_pg_subscription_substream - 1] = CharGetDatum(opts.streaming); values[Anum_pg_subscription_subtwophasestate - 1] = CharGetDatum(opts.twophase ? LOGICALREP_TWOPHASE_STATE_PENDING : @@ -1059,7 +1115,7 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt, if (IsSet(opts.specified_opts, SUBOPT_STREAMING)) { values[Anum_pg_subscription_substream - 1] = - BoolGetDatum(opts.streaming); + CharGetDatum(opts.streaming); replaces[Anum_pg_subscription_substream - 1] = true; } diff --git a/src/backend/postmaster/bgworker.c b/src/backend/postmaster/bgworker.c index 40601aefd9..40ccb8993c 100644 --- a/src/backend/postmaster/bgworker.c +++ b/src/backend/postmaster/bgworker.c @@ -128,6 +128,9 @@ static const struct }, { "ApplyWorkerMain", ApplyWorkerMain + }, + { + "ApplyBgworkerMain", ApplyBgworkerMain } }; diff --git a/src/backend/replication/logical/Makefile b/src/backend/replication/logical/Makefile index c4e2fdeb71..cbfb5d794e 100644 --- a/src/backend/replication/logical/Makefile +++ b/src/backend/replication/logical/Makefile @@ -15,6 +15,7 @@ include $(top_builddir)/src/Makefile.global override CPPFLAGS := -I$(srcdir) $(CPPFLAGS) OBJS = \ + applybgworker.o \ decode.o \ launcher.o \ logical.o \ diff --git a/src/backend/replication/logical/applybgworker.c b/src/backend/replication/logical/applybgworker.c new file mode 100644 index 0000000000..dfa49be98c --- /dev/null +++ b/src/backend/replication/logical/applybgworker.c @@ -0,0 +1,775 @@ +/*------------------------------------------------------------------------- + * applybgworker.c + * Support routines for applying xact by apply background worker + * + * Copyright (c) 2016-2022, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/replication/logical/applybgworker.c + * + * This file contains routines that are intended to support setting up, using, + * and tearing down a ApplyBgworkerState. + * + * Refer to the comments in file header of logical/worker.c to see more + * information about apply background worker. + * + *------------------------------------------------------------------------- + */ + +#include "postgres.h" + +#include "libpq/pqformat.h" +#include "mb/pg_wchar.h" +#include "pgstat.h" +#include "postmaster/interrupt.h" +#include "replication/logicallauncher.h" +#include "replication/logicalworker.h" +#include "replication/origin.h" +#include "replication/walreceiver.h" +#include "replication/worker_internal.h" +#include "storage/ipc.h" +#include "storage/procarray.h" +#include "tcop/tcopprot.h" +#include "utils/inval.h" +#include "utils/memutils.h" +#include "utils/resowner.h" +#include "utils/syscache.h" + +#define PG_LOGICAL_APPLY_SHM_MAGIC 0x79fb2447 // TODO Consider change + +/* + * DSM keys for apply background worker. Unlike other parallel execution code, + * since we don't need to worry about DSM keys conflicting with plan_node_id we + * can use small integers. + */ +#define APPLY_BGWORKER_KEY_SHARED 1 +#define APPLY_BGWORKER_KEY_MQ 2 + +/* Queue size of DSM, 16 MB for now. */ +#define DSM_QUEUE_SIZE 160000000 + +/* + * There are three fields in message: start_lsn, end_lsn and send_time. Because + * we have updated these statistics in apply worker, we could ignore these + * fields in apply background worker. (see function LogicalRepApplyLoop) + */ +#define IGNORE_SIZE_IN_MESSAGE (3 * sizeof(uint64)) + +/* + * Entry for a hash table we use to map from xid to our apply background worker + * state. + */ +typedef struct ApplyBgworkerEntry +{ + TransactionId xid; + ApplyBgworkerState *wstate; +} ApplyBgworkerEntry; + +/* Apply background workers hash table (initialized on first use) */ +static HTAB *ApplyWorkersHash = NULL; +static List *ApplyWorkersFreeList = NIL; +static List *ApplyWorkersList = NIL; + +/* + * Information shared between main apply worker and apply background worker. + */ +volatile ApplyBgworkerShared *MyParallelState = NULL; + +List *subxactlist = NIL; + +static bool apply_bgworker_can_start(TransactionId xid); +static ApplyBgworkerState *apply_bgworker_setup(void); +static void apply_bgworker_setup_dsm(ApplyBgworkerState *wstate); + +/* + * Check if starting a new apply background worker is allowed. + */ +static bool +apply_bgworker_can_start(TransactionId xid) +{ + if (!TransactionIdIsValid(xid)) + return false; + + /* + * Don't start a new background worker if not in streaming parallel mode. + */ + if (MySubscription->stream != SUBSTREAM_PARALLEL) + return false; + + /* + * Don't start a new background worker if user has set skiplsn as it's + * possible that user want to skip the streaming transaction. For + * streaming transaction, we need to spill the transaction to disk so that + * we can get the last LSN of the transaction to judge whether to skip + * before starting to apply the change. + */ + if (!XLogRecPtrIsInvalid(MySubscription->skiplsn)) + return false; + + /* + * For streaming transactions that are being applied in apply background + * worker, we cannot decide whether to apply the change for a relation + * that is not in the READY state (see should_apply_changes_for_rel) as we + * won't know remote_final_lsn by that time. So, we don't start new apply + * background worker in this case. + */ + if (!AllTablesyncsReady()) + return false; + + return true; +} + +/* + * Try to start an apply background worker and, if successful, cache it in + * ApplyWorkersHash keyed by the specified xid. + */ +ApplyBgworkerState * +apply_bgworker_start(TransactionId xid) +{ + bool found; + ApplyBgworkerState *wstate; + ApplyBgworkerEntry *entry = NULL; + + if (!apply_bgworker_can_start(xid)) + return NULL; + + /* First time through, initialize apply workers hashtable */ + if (ApplyWorkersHash == NULL) + { + HASHCTL ctl; + + MemSet(&ctl, 0, sizeof(ctl)); + ctl.keysize = sizeof(TransactionId); + ctl.entrysize = sizeof(ApplyBgworkerEntry); + ctl.hcxt = ApplyContext; + + ApplyWorkersHash = hash_create("logical apply workers hash", 8, &ctl, + HASH_ELEM | HASH_BLOBS | HASH_CONTEXT); + } + + /* + * Now, we try to get an apply background worker. If there is at least one + * worker in the free list, then take one. Otherwise, we try to start a + * new apply background worker. + */ + if (list_length(ApplyWorkersFreeList) > 0) + { + wstate = (ApplyBgworkerState *) llast(ApplyWorkersFreeList); + ApplyWorkersFreeList = list_delete_last(ApplyWorkersFreeList); + Assert(wstate->pstate->status == APPLY_BGWORKER_FINISHED); + } + else + { + wstate = apply_bgworker_setup(); + + if (wstate == NULL) + return NULL; + } + + /* + * Create entry for requested transaction. + */ + entry = hash_search(ApplyWorkersHash, &xid, HASH_ENTER, &found); + if (found) + elog(ERROR, "hash table corrupted"); + + /* Fill up the hash entry */ + wstate->pstate->status = APPLY_BGWORKER_BUSY; + wstate->pstate->server_version = walrcv_server_version(LogRepWorkerWalRcvConn); + wstate->pstate->stream_xid = xid; + entry->wstate = wstate; + entry->xid = xid; + + return wstate; +} + +/* + * Try to look up worker inside ApplyWorkersHash for requested xid. + */ +ApplyBgworkerState * +apply_bgworker_find(TransactionId xid) +{ + bool found; + ApplyBgworkerEntry *entry = NULL; + + if (!TransactionIdIsValid(xid)) + return NULL; + + if (ApplyWorkersHash == NULL) + return NULL; + + /* + * Find entry for requested transaction. + */ + entry = hash_search(ApplyWorkersHash, &xid, HASH_FIND, &found); + if (found) + { + Assert(entry->wstate->pstate->status == APPLY_BGWORKER_BUSY); + return entry->wstate; + } + else + return NULL; +} + +/* + * Add the worker to the free list and remove the entry from the hash table. + */ +void +apply_bgworker_free(ApplyBgworkerState *wstate) +{ + MemoryContext oldctx; + TransactionId xid = wstate->pstate->stream_xid; + + Assert(wstate->pstate->status == APPLY_BGWORKER_FINISHED); + + oldctx = MemoryContextSwitchTo(ApplyContext); + + hash_search(ApplyWorkersHash, &xid, HASH_REMOVE, NULL); + + elog(DEBUG1, "adding finished apply worker #%u for xid %u to the free list", + wstate->pstate->n, wstate->pstate->stream_xid); + + ApplyWorkersFreeList = lappend(ApplyWorkersFreeList, wstate); + + MemoryContextSwitchTo(oldctx); +} + +/* Apply Background Worker main loop */ +static void +LogicalApplyBgwLoop(shm_mq_handle *mqh, volatile ApplyBgworkerShared *pst) +{ + shm_mq_result shmq_res; + PGPROC *registrant; + ErrorContextCallback errcallback; + + registrant = BackendPidGetProc(MyBgworkerEntry->bgw_notify_pid); + SetLatch(®istrant->procLatch); + + /* + * Push apply error context callback. Fields will be filled applying a + * change. + */ + errcallback.callback = apply_error_callback; + errcallback.previous = error_context_stack; + error_context_stack = &errcallback; + + for (;;) + { + void *data; + Size len; + int c; + StringInfoData s; + MemoryContext oldctx; + + CHECK_FOR_INTERRUPTS(); + + /* Ensure we are reading the data into our memory context. */ + oldctx = MemoryContextSwitchTo(ApplyMessageContext); + + shmq_res = shm_mq_receive(mqh, &len, &data, false); + + if (shmq_res != SHM_MQ_SUCCESS) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("lost connection to the main apply worker"))); + + if (len == 0) + break; + + s.cursor = 0; + s.maxlen = -1; + s.data = (char *) data; + s.len = len; + + /* + * We use first byte of message for additional communication between + * main Logical replication worker and apply background workers, so if + * it differs from 'w', then process it first. + */ + c = pq_getmsgbyte(&s); + switch (c) + { + /* End message of streaming chunk */ + case LOGICAL_REP_MSG_STREAM_STOP: + elog(DEBUG1, "[Apply BGW #%u] ended processing streaming chunk," + "waiting on shm_mq_receive", pst->n); + + in_streamed_transaction = false; + pgstat_report_activity(STATE_IDLEINTRANSACTION, NULL); + continue; + case 'w': + break; + default: + elog(ERROR, "[Apply BGW #%u] unexpected message \"%c\"", + pst->n, c); + break; + } + + /* Ignore statistics fields that have been updated. */ + s.cursor += IGNORE_SIZE_IN_MESSAGE; + + apply_dispatch(&s); + + if (ConfigReloadPending) + { + ConfigReloadPending = false; + ProcessConfigFile(PGC_SIGHUP); + } + + MemoryContextSwitchTo(oldctx); + MemoryContextReset(ApplyMessageContext); + } + + MemoryContextSwitchTo(TopMemoryContext); + MemoryContextReset(ApplyContext); + + /* Pop the error context stack */ + error_context_stack = errcallback.previous; + + elog(DEBUG1, "[Apply BGW #%u] exiting", pst->n); + + /* Signal main process that we are done. */ + SetLatch(®istrant->procLatch); +} + +/* + * Set the exit status so that the main apply worker can realize we have + * shutdown. + */ +static void +apply_bgworker_shutdown(int code, Datum arg) +{ + SpinLockAcquire(&MyParallelState->mutex); + MyParallelState->status = APPLY_BGWORKER_EXIT; + SpinLockRelease(&MyParallelState->mutex); + + dsm_detach((dsm_segment *) DatumGetPointer(arg)); +} + +/* + * Apply Background Worker entry point + */ +void +ApplyBgworkerMain(Datum main_arg) +{ + volatile ApplyBgworkerShared *pst; + + dsm_handle handle; + dsm_segment *seg; + shm_toc *toc; + shm_mq *mq; + shm_mq_handle *mqh; + MemoryContext oldcontext; + RepOriginId originid; + int worker_slot = DatumGetInt32(main_arg); + char originname[NAMEDATALEN]; + + MemoryContextSwitchTo(TopMemoryContext); + + /* Init the memory context for the apply background worker to work in. */ + ApplyContext = AllocSetContextCreate(TopMemoryContext, + "ApplyContext", + ALLOCSET_DEFAULT_SIZES); + + /* + * Init the ApplyMessageContext which we clean up after each replication + * protocol message. + */ + ApplyMessageContext = AllocSetContextCreate(ApplyContext, + "ApplyMessageContext", + ALLOCSET_DEFAULT_SIZES); + + /* Setup signal handling */ + pqsignal(SIGHUP, SignalHandlerForConfigReload); + pqsignal(SIGTERM, die); + BackgroundWorkerUnblockSignals(); + + /* + * Connect to the dynamic shared memory segment. + * + * The backend that registered this worker passed us the ID of a shared + * memory segment to which we must attach for further instructions. In + * order to attach to dynamic shared memory, we need a resource owner. + * Once we've mapped the segment in our address space, attach to the table + * of contents so we can locate the various data structures we'll need to + * find within the segment. + */ + CurrentResourceOwner = ResourceOwnerCreate(NULL, "Logical apply worker"); + memcpy(&handle, MyBgworkerEntry->bgw_extra, sizeof(dsm_handle)); + seg = dsm_attach(handle); + if (seg == NULL) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("unable to map dynamic shared memory segment"))); + toc = shm_toc_attach(PG_LOGICAL_APPLY_SHM_MAGIC, dsm_segment_address(seg)); + if (toc == NULL) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("bad magic number in dynamic shared memory segment"))); + + before_shmem_exit(apply_bgworker_shutdown, PointerGetDatum(seg)); + + /* Look up the parallel state. */ + pst = shm_toc_lookup(toc, APPLY_BGWORKER_KEY_SHARED, false); + MyParallelState = pst; + + /* + * Attach to the message queue. + */ + mq = shm_toc_lookup(toc, APPLY_BGWORKER_KEY_MQ, false); + shm_mq_set_receiver(mq, MyProc); + mqh = shm_mq_attach(mq, seg, NULL); + + /* Run as replica session replication role. */ + SetConfigOption("session_replication_role", "replica", + PGC_SUSET, PGC_S_OVERRIDE); + + /* + * Now, we have initialized DSM. Attach to slot. + */ + logicalrep_worker_attach(worker_slot); + + /* Connect to our database. */ + BackgroundWorkerInitializeConnectionByOid(MyLogicalRepWorker->dbid, + MyLogicalRepWorker->userid, + 0); + + /* + * Set always-secure search path, so malicious users can't redirect user + * code (e.g. pg_index.indexprs). + */ + SetConfigOption("search_path", "", PGC_SUSET, PGC_S_OVERRIDE); + + /* + * Set the client encoding to the database encoding, since that is what + * the leader will expect. + */ + SetClientEncoding(GetDatabaseEncoding()); + + stream_xid = pst->stream_xid; + + StartTransactionCommand(); + oldcontext = MemoryContextSwitchTo(ApplyContext); + + MySubscription = GetSubscription(MyLogicalRepWorker->subid, true); + if (!MySubscription) + { + ereport(LOG, + (errmsg("logical replication apply worker for subscription %u will not " + "start because the subscription was removed during startup", + MyLogicalRepWorker->subid))); + proc_exit(0); + } + + MySubscriptionValid = true; + MemoryContextSwitchTo(oldcontext); + + /* Setup synchronous commit according to the user's wishes */ + SetConfigOption("synchronous_commit", MySubscription->synccommit, + PGC_BACKEND, PGC_S_OVERRIDE); + + /* Keep us informed about subscription changes. */ + CacheRegisterSyscacheCallback(SUBSCRIPTIONOID, + subscription_change_cb, + (Datum) 0); + + CommitTransactionCommand(); + + /* Setup replication origin tracking. */ + StartTransactionCommand(); + snprintf(originname, sizeof(originname), "pg_%u", MySubscription->oid); + originid = replorigin_by_name(originname, true); + if (!OidIsValid(originid)) + originid = replorigin_create(originname); + + /* + * The apply background worker doesn't need to monopolize this replication + * origin which was already acquired by its leader process. + */ + replorigin_session_setup(originid, false); + replorigin_session_origin = originid; + CommitTransactionCommand(); + + /* + * Allocate the origin name in long-lived context for error context + * message. + */ + apply_error_callback_arg.origin_name = MemoryContextStrdup(ApplyContext, + originname); + + elog(DEBUG1, "[Apply BGW #%u] started", pst->n); + + LogicalApplyBgwLoop(mqh, pst); + + /* + * We're done. Explicitly detach the shared memory segment so that we + * don't get a resource leak warning at commit time. This will fire any + * on_dsm_detach callbacks we've registered, as well. Once that's done, + * we can go ahead and exit. + */ + dsm_detach(seg); + proc_exit(0); +} + +/* + * Set up a dynamic shared memory segment. + * + * We set up a control region that contains a ApplyBgworkerShared, + * plus one region per message queue. There are as many message queues as + * the number of workers. + */ +static void +apply_bgworker_setup_dsm(ApplyBgworkerState *wstate) +{ + shm_toc_estimator e; + Size segsize; + dsm_segment *seg; + shm_toc *toc; + ApplyBgworkerShared *pst; + shm_mq *mq; + int64 queue_size = DSM_QUEUE_SIZE; + + /* + * Estimate how much shared memory we need. + * + * Because the TOC machinery may choose to insert padding of oddly-sized + * requests, we must estimate each chunk separately. + * + * We need one key to register the location of the header, and we need + * another key to track the location of the message queue. + */ + shm_toc_initialize_estimator(&e); + shm_toc_estimate_chunk(&e, sizeof(ApplyBgworkerShared)); + shm_toc_estimate_chunk(&e, (Size) queue_size); + + shm_toc_estimate_keys(&e, 1 + 1); + segsize = shm_toc_estimate(&e); + + /* Create the shared memory segment and establish a table of contents. */ + seg = dsm_create(shm_toc_estimate(&e), 0); + toc = shm_toc_create(PG_LOGICAL_APPLY_SHM_MAGIC, dsm_segment_address(seg), + segsize); + + /* Set up the header region. */ + pst = shm_toc_allocate(toc, sizeof(ApplyBgworkerShared)); + SpinLockInit(&pst->mutex); + pst->status = APPLY_BGWORKER_BUSY; + pst->server_version = walrcv_server_version(LogRepWorkerWalRcvConn); + pst->stream_xid = stream_xid; + pst->n = list_length(ApplyWorkersList) + 1; + + shm_toc_insert(toc, APPLY_BGWORKER_KEY_SHARED, pst); + + /* Set up message queue for the worker. */ + mq = shm_mq_create(shm_toc_allocate(toc, (Size) queue_size), + (Size) queue_size); + shm_toc_insert(toc, APPLY_BGWORKER_KEY_MQ, mq); + shm_mq_set_sender(mq, MyProc); + + /* Attach the queue. */ + wstate->mq_handle = shm_mq_attach(mq, seg, NULL); + + /* Return results to caller. */ + wstate->dsm_seg = seg; + wstate->pstate = pst; +} + +/* + * Start apply background worker process and allocate shared memory for it. + */ +static ApplyBgworkerState * +apply_bgworker_setup(void) +{ + MemoryContext oldcontext; + bool launched; + ApplyBgworkerState *wstate; + int napplyworkers; + + elog(DEBUG1, "setting up apply worker #%u", list_length(ApplyWorkersList) + 1); + + /* Check If there are free worker slot(s) */ + LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); + napplyworkers = logicalrep_apply_bgworker_count(MyLogicalRepWorker->subid); + LWLockRelease(LogicalRepWorkerLock); + if (napplyworkers >= max_apply_bgworkers_per_subscription) + return NULL; + + oldcontext = MemoryContextSwitchTo(ApplyContext); + + wstate = (ApplyBgworkerState *) palloc0(sizeof(ApplyBgworkerState)); + + /* Setup shared memory */ + apply_bgworker_setup_dsm(wstate); + + launched = logicalrep_worker_launch(MyLogicalRepWorker->dbid, + MySubscription->oid, + MySubscription->name, + MyLogicalRepWorker->userid, + InvalidOid, + dsm_segment_handle(wstate->dsm_seg)); + + if (launched) + ApplyWorkersList = lappend(ApplyWorkersList, wstate); + else + { + dsm_detach(wstate->dsm_seg); + wstate->dsm_seg = NULL; + + pfree(wstate); + wstate = NULL; + } + + MemoryContextSwitchTo(oldcontext); + + return wstate; +} + +/* + * Send the data to the specified apply background worker via shared-memory queue. + */ +void +apply_bgworker_send_data(ApplyBgworkerState *wstate, Size nbytes, const void *data) +{ + shm_mq_result result; + + result = shm_mq_send(wstate->mq_handle, nbytes, data, false, true); + + if (result != SHM_MQ_SUCCESS) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("could not send tuples to shared-memory queue"))); +} + +/* + * Wait until the status of apply background worker reaches the + * 'wait_for_status' + */ +void +apply_bgworker_wait_for(ApplyBgworkerState *wstate, + ApplyBgworkerStatus wait_for_status) +{ + for (;;) + { + char status; + + SpinLockAcquire(&wstate->pstate->mutex); + status = wstate->pstate->status; + SpinLockRelease(&wstate->pstate->mutex); + + /* Done if already in correct status. */ + if (status == wait_for_status) + break; + + /* If any workers (or the postmaster) have died, we have failed. */ + if (status == APPLY_BGWORKER_EXIT) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("background worker %u failed to apply transaction %u", + wstate->pstate->n, wstate->pstate->stream_xid))); + + /* Wait to be signalled. */ + WaitLatch(MyLatch, WL_LATCH_SET | WL_EXIT_ON_PM_DEATH, 0, + WAIT_EVENT_LOGICAL_APPLY_WORKER_STATE_CHANGE); + + /* Reset the latch so we don't spin. */ + ResetLatch(MyLatch); + + /* An interrupt may have occurred while we were waiting. */ + CHECK_FOR_INTERRUPTS(); + } +} + +/* + * Check the status of workers and report an error if any apply background + * worker has exited unexpectedly. + */ +void +apply_bgworker_check_status(void) +{ + ListCell *lc; + + if (am_apply_bgworker() || MySubscription->stream != SUBSTREAM_PARALLEL) + return; + + foreach(lc, ApplyWorkersList) + { + ApplyBgworkerState *wstate = (ApplyBgworkerState *) lfirst(lc); + + /* + * We don't lock here as in the worst case we will just detect the + * failure of worker a bit later. + */ + if (wstate->pstate->status == APPLY_BGWORKER_EXIT) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("background worker %u exited unexpectedly", + wstate->pstate->n))); + } + + /* + * Exit if any relation is not in the READY state and if any worker is + * handling the streaming transaction at the same time. Because for + * streaming transactions that is being applied in apply background + * worker, we cannot decide whether to apply the change for a relation + * that is not in the READY state (see should_apply_changes_for_rel) as we + * won't know remote_final_lsn by that time. + */ + if (list_length(ApplyWorkersFreeList) != list_length(ApplyWorkersList) && + !AllTablesyncsReady()) + { + ereport(LOG, + (errmsg("logical replication apply workers for subscription \"%s\" will restart", + MySubscription->name), + errdetail("Cannot handle streamed replication transaction by apply " + "background workers until all tables are synchronized"))); + + proc_exit(0); + } +} + +/* Set the apply background worker status */ +void +apply_bgworker_set_status(ApplyBgworkerStatus status) +{ + if (!am_apply_bgworker()) + return; + + elog(DEBUG1, "[Apply BGW #%u] set status to %d", MyParallelState->n, status); + + SpinLockAcquire(&MyParallelState->mutex); + MyParallelState->status = status; + SpinLockRelease(&MyParallelState->mutex); +} + +/* + * Define a savepoint for a subxact in apply background worker if needed. + * + * Inside apply background worker we can figure out that new subtransaction was + * started if new change arrived with different xid. In that case we can define + * named savepoint, so that we were able to commit/rollback it separately + * later. + * Special case is if the first change comes from subtransaction, then + * we check that current_xid differs from stream_xid. + */ +void +apply_bgworker_subxact_info_add(TransactionId current_xid) +{ + if (current_xid != stream_xid && + !list_member_int(subxactlist, (int) current_xid)) + { + MemoryContext oldctx; + char spname[MAXPGPATH]; + + snprintf(spname, MAXPGPATH, "savepoint_for_xid_%u", current_xid); + + elog(DEBUG1, "[Apply BGW #%u] defining savepoint %s", + MyParallelState->n, spname); + + DefineSavepoint(spname); + CommitTransactionCommand(); + + oldctx = MemoryContextSwitchTo(ApplyContext); + subxactlist = lappend_int(subxactlist, (int) current_xid); + MemoryContextSwitchTo(oldctx); + } +} diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c index c5c6a2ba68..d4d5093a0b 100644 --- a/src/backend/replication/logical/decode.c +++ b/src/backend/replication/logical/decode.c @@ -651,9 +651,10 @@ DecodeCommit(LogicalDecodingContext *ctx, XLogRecordBuffer *buf, { for (i = 0; i < parsed->nsubxacts; i++) { - ReorderBufferForget(ctx->reorder, parsed->subxacts[i], buf->origptr); + ReorderBufferForget(ctx->reorder, parsed->subxacts[i], buf->origptr, + commit_time); } - ReorderBufferForget(ctx->reorder, xid, buf->origptr); + ReorderBufferForget(ctx->reorder, xid, buf->origptr, commit_time); return; } @@ -821,10 +822,11 @@ DecodeAbort(LogicalDecodingContext *ctx, XLogRecordBuffer *buf, for (i = 0; i < parsed->nsubxacts; i++) { ReorderBufferAbort(ctx->reorder, parsed->subxacts[i], - buf->record->EndRecPtr); + buf->record->EndRecPtr, abort_time); } - ReorderBufferAbort(ctx->reorder, xid, buf->record->EndRecPtr); + ReorderBufferAbort(ctx->reorder, xid, buf->record->EndRecPtr, + abort_time); } /* update the decoding stats */ diff --git a/src/backend/replication/logical/launcher.c b/src/backend/replication/logical/launcher.c index 2bdab53e19..d11d2361c6 100644 --- a/src/backend/replication/logical/launcher.c +++ b/src/backend/replication/logical/launcher.c @@ -54,6 +54,7 @@ int max_logical_replication_workers = 4; int max_sync_workers_per_subscription = 2; +int max_apply_bgworkers_per_subscription = 2; LogicalRepWorker *MyLogicalRepWorker = NULL; @@ -73,6 +74,7 @@ static void logicalrep_launcher_onexit(int code, Datum arg); static void logicalrep_worker_onexit(int code, Datum arg); static void logicalrep_worker_detach(void); static void logicalrep_worker_cleanup(LogicalRepWorker *worker); +static void logicalrep_worker_stop_internal(LogicalRepWorker *worker); static bool on_commit_launcher_wakeup = false; @@ -151,8 +153,10 @@ get_subscription_list(void) * * This is only needed for cleaning up the shared memory in case the worker * fails to attach. + * + * Return false if the attach fails. Otherwise return true. */ -static void +static bool WaitForReplicationWorkerAttach(LogicalRepWorker *worker, uint16 generation, BackgroundWorkerHandle *handle) @@ -168,11 +172,11 @@ WaitForReplicationWorkerAttach(LogicalRepWorker *worker, LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); - /* Worker either died or has started; no need to do anything. */ + /* Worker either died or has started. Return false if died. */ if (!worker->in_use || worker->proc) { LWLockRelease(LogicalRepWorkerLock); - return; + return worker->in_use; } LWLockRelease(LogicalRepWorkerLock); @@ -187,7 +191,7 @@ WaitForReplicationWorkerAttach(LogicalRepWorker *worker, if (generation == worker->generation) logicalrep_worker_cleanup(worker); LWLockRelease(LogicalRepWorkerLock); - return; + return false; } /* @@ -223,6 +227,13 @@ logicalrep_worker_find(Oid subid, Oid relid, bool only_running) { LogicalRepWorker *w = &LogicalRepCtx->workers[i]; + /* + * We are only interested in the main apply worker or table sync worker + * here. + */ + if (w->subworker) + continue; + if (w->in_use && w->subid == subid && w->relid == relid && (!only_running || w->proc)) { @@ -259,11 +270,11 @@ logicalrep_workers_find(Oid subid, bool only_running) } /* - * Start new apply background worker, if possible. + * Start new background worker, if possible. */ -void +bool logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, Oid userid, - Oid relid) + Oid relid, dsm_handle subworker_dsm) { BackgroundWorker bgw; BackgroundWorkerHandle *bgw_handle; @@ -273,6 +284,10 @@ logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, Oid userid, LogicalRepWorker *worker = NULL; int nsyncworkers; TimestampTz now; + bool is_subworker = (subworker_dsm != DSM_HANDLE_INVALID); + + /* Sanity check : we don't support table sync in subworker. */ + Assert(!(is_subworker && OidIsValid(relid))); ereport(DEBUG1, (errmsg_internal("starting logical replication worker for subscription \"%s\"", @@ -350,7 +365,7 @@ retry: if (OidIsValid(relid) && nsyncworkers >= max_sync_workers_per_subscription) { LWLockRelease(LogicalRepWorkerLock); - return; + return false; } /* @@ -364,7 +379,7 @@ retry: (errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED), errmsg("out of logical replication worker slots"), errhint("You might need to increase max_logical_replication_workers."))); - return; + return false; } /* Prepare the worker slot. */ @@ -379,6 +394,7 @@ retry: worker->relstate = SUBREL_STATE_UNKNOWN; worker->relstate_lsn = InvalidXLogRecPtr; worker->stream_fileset = NULL; + worker->subworker = is_subworker; worker->last_lsn = InvalidXLogRecPtr; TIMESTAMP_NOBEGIN(worker->last_send_time); TIMESTAMP_NOBEGIN(worker->last_recv_time); @@ -396,19 +412,31 @@ retry: BGWORKER_BACKEND_DATABASE_CONNECTION; bgw.bgw_start_time = BgWorkerStart_RecoveryFinished; snprintf(bgw.bgw_library_name, BGW_MAXLEN, "postgres"); - snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyWorkerMain"); + + if (is_subworker) + snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyBgworkerMain"); + else + snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyWorkerMain"); + if (OidIsValid(relid)) snprintf(bgw.bgw_name, BGW_MAXLEN, "logical replication worker for subscription %u sync %u", subid, relid); + else if (is_subworker) + snprintf(bgw.bgw_name, BGW_MAXLEN, + "logical replication apply background worker for subscription %u", subid); else snprintf(bgw.bgw_name, BGW_MAXLEN, "logical replication worker for subscription %u", subid); + snprintf(bgw.bgw_type, BGW_MAXLEN, "logical replication worker"); bgw.bgw_restart_time = BGW_NEVER_RESTART; bgw.bgw_notify_pid = MyProcPid; bgw.bgw_main_arg = Int32GetDatum(slot); + if (is_subworker) + memcpy(bgw.bgw_extra, &subworker_dsm, sizeof(dsm_handle)); + if (!RegisterDynamicBackgroundWorker(&bgw, &bgw_handle)) { /* Failed to start worker, so clean up the worker slot. */ @@ -421,11 +449,11 @@ retry: (errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED), errmsg("out of background worker slots"), errhint("You might need to increase max_worker_processes."))); - return; + return false; } /* Now wait until it attaches. */ - WaitForReplicationWorkerAttach(worker, generation, bgw_handle); + return WaitForReplicationWorkerAttach(worker, generation, bgw_handle); } /* @@ -436,18 +464,27 @@ void logicalrep_worker_stop(Oid subid, Oid relid) { LogicalRepWorker *worker; - uint16 generation; LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); worker = logicalrep_worker_find(subid, relid, false); - /* No worker, nothing to do. */ - if (!worker) - { - LWLockRelease(LogicalRepWorkerLock); - return; - } + if (worker) + logicalrep_worker_stop_internal(worker); + + LWLockRelease(LogicalRepWorkerLock); +} + +/* + * Workhorse for logicalrep_worker_stop() and logicalrep_worker_detach(). Stop + * the worker and wait for it to die. + */ +static void +logicalrep_worker_stop_internal(LogicalRepWorker *worker) +{ + uint16 generation; + + Assert(LWLockHeldByMe(LogicalRepWorkerLock)); /* * Remember which generation was our worker so we can check if what we see @@ -485,10 +522,7 @@ logicalrep_worker_stop(Oid subid, Oid relid) * different, meaning that a different worker has taken the slot. */ if (!worker->in_use || worker->generation != generation) - { - LWLockRelease(LogicalRepWorkerLock); return; - } /* Worker has assigned proc, so it has started. */ if (worker->proc) @@ -522,8 +556,6 @@ logicalrep_worker_stop(Oid subid, Oid relid) LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); } - - LWLockRelease(LogicalRepWorkerLock); } /* @@ -599,6 +631,29 @@ logicalrep_worker_attach(int slot) static void logicalrep_worker_detach(void) { + /* + * If we are the main apply worker, stop all the apply background workers + * we started before. + */ + if (!MyLogicalRepWorker->subworker) + { + List *workers; + ListCell *lc; + + LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); + + workers = logicalrep_workers_find(MyLogicalRepWorker->subid, true); + foreach(lc, workers) + { + LogicalRepWorker *w = (LogicalRepWorker *) lfirst(lc); + + if (w->subworker) + logicalrep_worker_stop_internal(w); + } + + LWLockRelease(LogicalRepWorkerLock); + } + /* Block concurrent access. */ LWLockAcquire(LogicalRepWorkerLock, LW_EXCLUSIVE); @@ -621,6 +676,7 @@ logicalrep_worker_cleanup(LogicalRepWorker *worker) worker->userid = InvalidOid; worker->subid = InvalidOid; worker->relid = InvalidOid; + worker->subworker = false; } /* @@ -679,6 +735,30 @@ logicalrep_sync_worker_count(Oid subid) return res; } +/* + * Count the number of registered (not necessarily running) apply background + * workers for a subscription. + */ +int +logicalrep_apply_bgworker_count(Oid subid) +{ + int i; + int res = 0; + + Assert(LWLockHeldByMe(LogicalRepWorkerLock)); + + /* Search for attached worker for a given subscription id. */ + for (i = 0; i < max_logical_replication_workers; i++) + { + LogicalRepWorker *w = &LogicalRepCtx->workers[i]; + + if (w->subid == subid && w->subworker) + res++; + } + + return res; +} + /* * ApplyLauncherShmemSize * Compute space needed for replication launcher shared memory @@ -868,7 +948,7 @@ ApplyLauncherMain(Datum main_arg) wait_time = wal_retrieve_retry_interval; logicalrep_worker_launch(sub->dbid, sub->oid, sub->name, - sub->owner, InvalidOid); + sub->owner, InvalidOid, DSM_HANDLE_INVALID); } } diff --git a/src/backend/replication/logical/origin.c b/src/backend/replication/logical/origin.c index 21937ab2d3..50c567fb6e 100644 --- a/src/backend/replication/logical/origin.c +++ b/src/backend/replication/logical/origin.c @@ -1063,12 +1063,21 @@ ReplicationOriginExitCleanup(int code, Datum arg) * array doesn't have to be searched when calling * replorigin_session_advance(). * - * Obviously only one such cached origin can exist per process and the current + * Normally only one such cached origin can exist per process and the current * cached value can only be set again after the previous value is torn down * with replorigin_session_reset(). + * + * However, if the function parameter 'must_acquire' is false, we allow the + * process to use the same slot already acquired by another process. It's safe + * because 1) The only caller (apply background workers) will maintain the + * commit order by allowing only one process to commit at a time, so no two + * workers will be operating on the same origin at the same time (see comments + * in logical/worker.c). 2) Even though we try to advance the session's origin + * concurrently, it's safe to do so as we change/advance the session_origin + * LSNs under replicate_state LWLock. */ void -replorigin_session_setup(RepOriginId node) +replorigin_session_setup(RepOriginId node, bool must_acquire) { static bool registered_cleanup; int i; @@ -1110,7 +1119,7 @@ replorigin_session_setup(RepOriginId node) if (curstate->roident != node) continue; - else if (curstate->acquired_by != 0) + else if (curstate->acquired_by != 0 && must_acquire) { ereport(ERROR, (errcode(ERRCODE_OBJECT_IN_USE), @@ -1141,7 +1150,14 @@ replorigin_session_setup(RepOriginId node) Assert(session_replication_state->roident != InvalidRepOriginId); - session_replication_state->acquired_by = MyProcPid; + if (must_acquire) + session_replication_state->acquired_by = MyProcPid; + else if (session_replication_state->acquired_by == 0) + ereport(ERROR, + (errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED), + errmsg("apply background worker could not find replication state slot for replication origin with OID %u", + node), + errdetail("There is no replication state slot set by its main apply worker."))); LWLockRelease(ReplicationOriginLock); @@ -1321,7 +1337,7 @@ pg_replication_origin_session_setup(PG_FUNCTION_ARGS) name = text_to_cstring((text *) DatumGetPointer(PG_GETARG_DATUM(0))); origin = replorigin_by_name(name, false); - replorigin_session_setup(origin); + replorigin_session_setup(origin, true); replorigin_session_origin = origin; diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c index ff8513e2d2..affd08cfa4 100644 --- a/src/backend/replication/logical/proto.c +++ b/src/backend/replication/logical/proto.c @@ -1163,31 +1163,56 @@ logicalrep_read_stream_commit(StringInfo in, LogicalRepCommitData *commit_data) /* * Write STREAM ABORT to the output stream. Note that xid and subxid will be * same for the top-level transaction abort. + * + * If write_abort_lsn is true, send the abort_lsn and abort_time fields. + * Otherwise not. */ void logicalrep_write_stream_abort(StringInfo out, TransactionId xid, - TransactionId subxid) + ReorderBufferTXN *txn, XLogRecPtr abort_lsn, + bool write_abort_lsn) { pq_sendbyte(out, LOGICAL_REP_MSG_STREAM_ABORT); - Assert(TransactionIdIsValid(xid) && TransactionIdIsValid(subxid)); + Assert(TransactionIdIsValid(xid) && TransactionIdIsValid(txn->xid)); /* transaction ID */ pq_sendint32(out, xid); - pq_sendint32(out, subxid); + pq_sendint32(out, txn->xid); + + if (write_abort_lsn) + { + pq_sendint64(out, abort_lsn); + pq_sendint64(out, txn->xact_time.abort_time); + } } /* * Read STREAM ABORT from the output stream. + * + * If read_abort_lsn is true, try to read the abort_lsn and abort_time fields. + * Otherwise not. */ void -logicalrep_read_stream_abort(StringInfo in, TransactionId *xid, - TransactionId *subxid) +logicalrep_read_stream_abort(StringInfo in, + LogicalRepStreamAbortData *abort_data, + bool read_abort_lsn) { - Assert(xid && subxid); + Assert(abort_data); - *xid = pq_getmsgint(in, 4); - *subxid = pq_getmsgint(in, 4); + abort_data->xid = pq_getmsgint(in, 4); + abort_data->subxid = pq_getmsgint(in, 4); + + if (read_abort_lsn) + { + abort_data->abort_lsn = pq_getmsgint64(in); + abort_data->abort_time = pq_getmsgint64(in); + } + else + { + abort_data->abort_lsn = InvalidXLogRecPtr; + abort_data->abort_time = 0; + } } /* diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c index 88a37fde72..8989328046 100644 --- a/src/backend/replication/logical/reorderbuffer.c +++ b/src/backend/replication/logical/reorderbuffer.c @@ -2826,7 +2826,8 @@ ReorderBufferFinishPrepared(ReorderBuffer *rb, TransactionId xid, * disk. */ void -ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn) +ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn, + TimestampTz abort_time) { ReorderBufferTXN *txn; @@ -2837,6 +2838,8 @@ ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn) if (txn == NULL) return; + txn->xact_time.abort_time = abort_time; + /* For streamed transactions notify the remote node about the abort. */ if (rbtxn_is_streamed(txn)) { @@ -2911,7 +2914,8 @@ ReorderBufferAbortOld(ReorderBuffer *rb, TransactionId oldestRunningXid) * to this xid might re-create the transaction incompletely. */ void -ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn) +ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn, + TimestampTz abort_time) { ReorderBufferTXN *txn; @@ -2922,6 +2926,8 @@ ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn) if (txn == NULL) return; + txn->xact_time.abort_time = abort_time; + /* For streamed transactions notify the remote node about the abort. */ if (rbtxn_is_streamed(txn)) rb->stream_abort(rb, txn, lsn); diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c index 670c6fcada..8ffba7e2e5 100644 --- a/src/backend/replication/logical/tablesync.c +++ b/src/backend/replication/logical/tablesync.c @@ -568,7 +568,8 @@ process_syncing_tables_for_apply(XLogRecPtr current_lsn) MySubscription->oid, MySubscription->name, MyLogicalRepWorker->userid, - rstate->relid); + rstate->relid, + DSM_HANDLE_INVALID); hentry->last_start_time = now; } } @@ -589,6 +590,9 @@ process_syncing_tables_for_apply(XLogRecPtr current_lsn) void process_syncing_tables(XLogRecPtr current_lsn) { + if (MyLogicalRepWorker->subworker) + return; + if (am_tablesync_worker()) process_syncing_tables_for_sync(current_lsn); else @@ -1273,7 +1277,7 @@ LogicalRepSyncTableStart(XLogRecPtr *origin_startpos) * time this tablesync was launched. */ originid = replorigin_by_name(originname, false); - replorigin_session_setup(originid); + replorigin_session_setup(originid, true); replorigin_session_origin = originid; *origin_startpos = replorigin_session_get_progress(false); @@ -1384,7 +1388,7 @@ LogicalRepSyncTableStart(XLogRecPtr *origin_startpos) true /* go backward */ , true /* WAL log */ ); UnlockRelationOid(ReplicationOriginRelationId, RowExclusiveLock); - replorigin_session_setup(originid); + replorigin_session_setup(originid, true); replorigin_session_origin = originid; } else diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c index 38e3b1c1b3..9da11dd41c 100644 --- a/src/backend/replication/logical/worker.c +++ b/src/backend/replication/logical/worker.c @@ -22,8 +22,28 @@ * STREAMED TRANSACTIONS * --------------------- * Streamed transactions (large transactions exceeding a memory limit on the - * upstream) are not applied immediately, but instead, the data is written - * to temporary files and then applied at once when the final commit arrives. + * upstream) are applied using one of two approaches. + * + * 1) Separate background workers + * + * If streaming = parallel, We assign a new apply background worker (if + * available) as soon as the xact's first stream is received. The main apply + * worker will send changes to this new worker via shared memory. We keep this + * worker assigned till the transaction commit is received and also wait for + * the worker to finish at commit. This preserves commit ordering and avoids + * file I/O in most cases. We still need to spill to a file if there is no + * worker available. It is important to maintain commit order to avoid failures + * due to (a) transaction dependencies, say if we insert a row in the first + * transaction and update it in the second transaction then allowing to apply + * both in parallel can lead to failure in the update. (b) deadlocks, allowing + * transactions that update the same set of rows/tables in opposite order to be + * applied in parallel can lead to deadlocks. + * + * 2) Write to temporary files and apply when the final commit arrives + * + * If no worker is available to handle streamed transaction, the data is + * written to temporary files and then applied at once when the final commit + * arrives. * * Unlike the regular (non-streamed) case, handling streamed transactions has * to handle aborts of both the toplevel transaction and subtransactions. This @@ -219,20 +239,8 @@ typedef struct ApplyExecutionData PartitionTupleRouting *proute; /* partition routing info */ } ApplyExecutionData; -/* Struct for saving and restoring apply errcontext information */ -typedef struct ApplyErrorCallbackArg -{ - LogicalRepMsgType command; /* 0 if invalid */ - LogicalRepRelMapEntry *rel; - - /* Remote node information */ - int remote_attnum; /* -1 if invalid */ - TransactionId remote_xid; - XLogRecPtr finish_lsn; - char *origin_name; -} ApplyErrorCallbackArg; - -static ApplyErrorCallbackArg apply_error_callback_arg = +/* errcontext tracker */ +ApplyErrorCallbackArg apply_error_callback_arg = { .command = 0, .rel = NULL, @@ -242,7 +250,7 @@ static ApplyErrorCallbackArg apply_error_callback_arg = .origin_name = NULL, }; -static MemoryContext ApplyMessageContext = NULL; +MemoryContext ApplyMessageContext = NULL; MemoryContext ApplyContext = NULL; /* per stream context for streaming transactions */ @@ -251,27 +259,38 @@ static MemoryContext LogicalStreamingContext = NULL; WalReceiverConn *LogRepWorkerWalRcvConn = NULL; Subscription *MySubscription = NULL; -static bool MySubscriptionValid = false; +bool MySubscriptionValid = false; bool in_remote_transaction = false; static XLogRecPtr remote_final_lsn = InvalidXLogRecPtr; /* fields valid only when processing streamed transaction */ -static bool in_streamed_transaction = false; +bool in_streamed_transaction = false; + +TransactionId stream_xid = InvalidTransactionId; +static ApplyBgworkerState *stream_apply_worker = NULL; -static TransactionId stream_xid = InvalidTransactionId; +/* Check if we are applying the transaction in an apply background worker */ +#define apply_bgworker_active() (in_streamed_transaction && stream_apply_worker != NULL) + +/* + * The number of changes during one streaming block (only for apply background + * workers) + */ +static uint32 nchanges = 0; /* * We enable skipping all data modification changes (INSERT, UPDATE, etc.) for * the subscription if the remote transaction's finish LSN matches the subskiplsn. * Once we start skipping changes, we don't stop it until we skip all changes of * the transaction even if pg_subscription is updated and MySubscription->skiplsn - * gets changed or reset during that. Also, in streaming transaction cases, we - * don't skip receiving and spooling the changes since we decide whether or not + * gets changed or reset during that. Also, in streaming transaction cases (streaming = on), + * we don't skip receiving and spooling the changes since we decide whether or not * to skip applying the changes when starting to apply changes. The subskiplsn is * cleared after successfully skipping the transaction or applying non-empty * transaction. The latter prevents the mistakenly specified subskiplsn from - * being left. + * being left. Note that we cannot skip the streaming transaction in parallel + * mode, because we cannot get the finish LSN before applying the changes. */ static XLogRecPtr skip_xact_finish_lsn = InvalidXLogRecPtr; #define is_skipping_changes() (unlikely(!XLogRecPtrIsInvalid(skip_xact_finish_lsn))) @@ -324,9 +343,6 @@ static void maybe_reread_subscription(void); static void DisableSubscriptionAndExit(void); -/* prototype needed because of stream_commit */ -static void apply_dispatch(StringInfo s); - static void apply_handle_commit_internal(LogicalRepCommitData *commit_data); static void apply_handle_insert_internal(ApplyExecutionData *edata, ResultRelInfo *relinfo, @@ -359,7 +375,6 @@ static void stop_skipping_changes(void); static void clear_subscription_skip_lsn(XLogRecPtr finish_lsn); /* Functions for apply error callback */ -static void apply_error_callback(void *arg); static inline void set_apply_error_context_xact(TransactionId xid, XLogRecPtr lsn); static inline void reset_apply_error_context_info(void); @@ -426,40 +441,85 @@ end_replication_step(void) } /* - * Handle streamed transactions. + * Handle streamed transactions for both the main apply worker and the apply + * background workers. + * + * In streaming case (receiving a block of streamed transaction), for + * SUBSTREAM_ON mode, we simply redirect it to a file for the proper toplevel + * transaction, and for SUBSTREAM_PARALLEL mode, we send the changes to + * apply background worker (LOGICAL_REP_MSG_RELATION or LOGICAL_REP_MSG_TYPE + * changes will also be applied in main apply worker). * - * If in streaming mode (receiving a block of streamed transaction), we - * simply redirect it to a file for the proper toplevel transaction. + * For non-streamed transactions, returns false; + * For streamed transactions, returns true if in main apply worker, false + * otherwise. * - * Returns true for streamed transactions, false otherwise (regular mode). + * Exception: When parallel mode is applying streamed transaction in the main + * apply worker, (e.g. when addressing LOGICAL_REP_MSG_RELATION or + * LOGICAL_REP_MSG_TYPE changes), then return false. */ static bool handle_streamed_transaction(LogicalRepMsgType action, StringInfo s) { - TransactionId xid; + TransactionId current_xid = InvalidTransactionId; - /* not in streaming mode */ - if (!in_streamed_transaction) + /* Not in streaming mode */ + if (!(in_streamed_transaction || am_apply_bgworker())) return false; - Assert(stream_fd != NULL); Assert(TransactionIdIsValid(stream_xid)); /* * We should have received XID of the subxact as the first part of the * message, so extract it. */ - xid = pq_getmsgint(s, 4); + current_xid = pq_getmsgint(s, 4); - if (!TransactionIdIsValid(xid)) + if (!TransactionIdIsValid(current_xid)) ereport(ERROR, (errcode(ERRCODE_PROTOCOL_VIOLATION), errmsg_internal("invalid transaction ID in streamed replication transaction"))); - /* Add the new subxact to the array (unless already there). */ - subxact_info_add(xid); + if (am_apply_bgworker()) + { + /* Define a savepoint for a subxact if needed. */ + apply_bgworker_subxact_info_add(current_xid); + + return false; + } + + if (apply_bgworker_active()) + { + /* + * This is the main apply worker, but there is an apply background + * worker, so apply the changes of this transaction in that background + * worker. Pass the data to the worker. + */ + apply_bgworker_send_data(stream_apply_worker, s->len, s->data); + nchanges += 1; + + /* + * XXX The publisher side doesn't always send relation/type update + * messages after the streaming transaction, so also update the + * relation/type in main apply worker here. See function + * cleanup_rel_sync_cache. + */ + if (action == LOGICAL_REP_MSG_RELATION || + action == LOGICAL_REP_MSG_TYPE) + return false; + + return true; + } + + /* + * This is the main apply worker, but there is no apply background worker, + * so write to temporary files and apply when the final commit arrives. + * + * Add the new subxact to the array (unless already there). + */ + subxact_info_add(current_xid); - /* write the change to the current file */ + /* Write the change to the current file */ stream_write_change(action, s); return true; @@ -844,6 +904,9 @@ apply_handle_commit(StringInfo s) apply_handle_commit_internal(&commit_data); + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); + /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(commit_data.end_lsn); @@ -898,7 +961,9 @@ apply_handle_prepare_internal(LogicalRepPreparedTxnData *prepare_data) * BeginTransactionBlock is necessary to balance the EndTransactionBlock * called within the PrepareTransactionBlock below. */ - BeginTransactionBlock(); + if (!IsTransactionBlock()) + BeginTransactionBlock(); + CommitTransactionCommand(); /* Completes the preceding Begin command. */ /* @@ -950,6 +1015,9 @@ apply_handle_prepare(StringInfo s) in_remote_transaction = false; + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); + /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(prepare_data.end_lsn); @@ -1064,10 +1132,6 @@ apply_handle_rollback_prepared(StringInfo s) /* * Handle STREAM PREPARE. - * - * Logic is in two parts: - * 1. Replay all the spooled operations - * 2. Mark the transaction as prepared */ static void apply_handle_stream_prepare(StringInfo s) @@ -1088,24 +1152,76 @@ apply_handle_stream_prepare(StringInfo s) logicalrep_read_stream_prepare(s, &prepare_data); set_apply_error_context_xact(prepare_data.xid, prepare_data.prepare_lsn); - elog(DEBUG1, "received prepare for streamed transaction %u", prepare_data.xid); + if (am_apply_bgworker()) + { + /* Mark the transaction as prepared. */ + apply_handle_prepare_internal(&prepare_data); - /* Replay all the spooled operations. */ - apply_spooled_messages(prepare_data.xid, prepare_data.prepare_lsn); + CommitTransactionCommand(); - /* Mark the transaction as prepared. */ - apply_handle_prepare_internal(&prepare_data); + pgstat_report_stat(false); - CommitTransactionCommand(); + list_free(subxactlist); + subxactlist = NIL; - pgstat_report_stat(false); + apply_bgworker_set_status(APPLY_BGWORKER_FINISHED); + } + else + { + /* This is the main apply worker. */ + ApplyBgworkerState *wstate = apply_bgworker_find(prepare_data.xid); - store_flush_position(prepare_data.end_lsn); + elog(DEBUG1, "received prepare for streamed transaction %u", + prepare_data.xid); + + /* + * Check if we are processing this transaction in an apply background + * worker and if so, send the changes to that worker. + */ + if (wstate) + { + /* + * After sending the data to the apply background worker, wait for + * that worker to finish. This is necessary to maintain commit + * order which avoids failures due to transaction dependencies and + * deadlocks. + */ + apply_bgworker_send_data(wstate, s->len, s->data); + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_FINISHED); + apply_bgworker_free(wstate); + + pgstat_report_stat(false); + store_flush_position(prepare_data.end_lsn); + } + else + { + /* + * The transaction has been serialized to file, so replay all the + * spooled operations. + */ + apply_spooled_messages(prepare_data.xid, prepare_data.prepare_lsn); + + /* Mark the transaction as prepared. */ + apply_handle_prepare_internal(&prepare_data); + + CommitTransactionCommand(); + + pgstat_report_stat(false); + + store_flush_position(prepare_data.end_lsn); + + in_remote_transaction = false; + + /* Unlink the files with serialized changes and subxact info. */ + stream_cleanup_files(MyLogicalRepWorker->subid, prepare_data.xid); + } + } in_remote_transaction = false; + stream_apply_worker = NULL; - /* unlink the files with serialized changes and subxact info. */ - stream_cleanup_files(MyLogicalRepWorker->subid, prepare_data.xid); + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(prepare_data.end_lsn); @@ -1155,15 +1271,6 @@ apply_handle_stream_start(StringInfo s) (errcode(ERRCODE_PROTOCOL_VIOLATION), errmsg_internal("duplicate STREAM START message"))); - /* - * Start a transaction on stream start, this transaction will be committed - * on the stream stop unless it is a tablesync worker in which case it - * will be committed after processing all the messages. We need the - * transaction for handling the buffile, used for serializing the - * streaming data and subxact info. - */ - begin_replication_step(); - /* notify handle methods we're processing a remote transaction */ in_streamed_transaction = true; @@ -1177,36 +1284,93 @@ apply_handle_stream_start(StringInfo s) set_apply_error_context_xact(stream_xid, InvalidXLogRecPtr); - /* - * Initialize the worker's stream_fileset if we haven't yet. This will be - * used for the entire duration of the worker so create it in a permanent - * context. We create this on the very first streaming message from any - * transaction and then use it for this and other streaming transactions. - * Now, we could create a fileset at the start of the worker as well but - * then we won't be sure that it will ever be used. - */ - if (MyLogicalRepWorker->stream_fileset == NULL) + if (am_apply_bgworker()) { - MemoryContext oldctx; - - oldctx = MemoryContextSwitchTo(ApplyContext); + /* + * Make sure the handle apply_dispatch methods are aware we're in a + * remote transaction. + */ + in_remote_transaction = true; - MyLogicalRepWorker->stream_fileset = palloc(sizeof(FileSet)); - FileSetInit(MyLogicalRepWorker->stream_fileset); + /* Begin the transaction. */ + AcceptInvalidationMessages(); + maybe_reread_subscription(); - MemoryContextSwitchTo(oldctx); + StartTransactionCommand(); + BeginTransactionBlock(); + CommitTransactionCommand(); } + else + { + /* + * This is the main apply worker. Check if there is any free apply + * background worker we can use to process this transaction. + */ + if (first_segment) + stream_apply_worker = apply_bgworker_start(stream_xid); + else + stream_apply_worker = apply_bgworker_find(stream_xid); - /* open the spool file for this transaction */ - stream_open_file(MyLogicalRepWorker->subid, stream_xid, first_segment); + if (stream_apply_worker) + { + /* + * If we have found a free worker or if we are already applying this + * transaction in an apply background worker, then we pass the data to + * that worker. + */ + if (first_segment) + apply_bgworker_send_data(stream_apply_worker, s->len, s->data); - /* if this is not the first segment, open existing subxact file */ - if (!first_segment) - subxact_info_read(MyLogicalRepWorker->subid, stream_xid); + nchanges = 0; + elog(DEBUG1, "starting streaming of xid %u", stream_xid); + } + else + { + /* + * Since no apply background worker is available for the first + * stream start, serialize all the changes of the transaction. + * + * Start a transaction on stream start, this transaction will be + * committed on the stream stop unless it is a tablesync worker in + * which case it will be committed after processing all the + * messages. We need the transaction for handling the buffile, + * used for serializing the streaming data and subxact info. + */ + begin_replication_step(); - pgstat_report_activity(STATE_RUNNING, NULL); + /* + * Initialize the worker's stream_fileset if we haven't yet. This will + * be used for the entire duration of the worker so create it in a + * permanent context. We create this on the very first streaming + * message from any transaction and then use it for this and other + * streaming transactions. Now, we could create a fileset at the start + * of the worker as well but then we won't be sure that it will ever + * be used. + */ + if (MyLogicalRepWorker->stream_fileset == NULL) + { + MemoryContext oldctx; - end_replication_step(); + oldctx = MemoryContextSwitchTo(ApplyContext); + + MyLogicalRepWorker->stream_fileset = palloc(sizeof(FileSet)); + FileSetInit(MyLogicalRepWorker->stream_fileset); + + MemoryContextSwitchTo(oldctx); + } + + /* Open the spool file for this transaction. */ + stream_open_file(MyLogicalRepWorker->subid, stream_xid, first_segment); + + /* If this is not the first segment, open existing subxact file. */ + if (!first_segment) + subxact_info_read(MyLogicalRepWorker->subid, stream_xid); + + end_replication_step(); + } + } + + pgstat_report_activity(STATE_RUNNING, NULL); } /* @@ -1220,53 +1384,52 @@ apply_handle_stream_stop(StringInfo s) (errcode(ERRCODE_PROTOCOL_VIOLATION), errmsg_internal("STREAM STOP message without STREAM START"))); - /* - * Close the file with serialized changes, and serialize information about - * subxacts for the toplevel transaction. - */ - subxact_info_write(MyLogicalRepWorker->subid, stream_xid); - stream_close_file(); + if (apply_bgworker_active()) + { + char action = LOGICAL_REP_MSG_STREAM_STOP; - /* We must be in a valid transaction state */ - Assert(IsTransactionState()); + apply_bgworker_send_data(stream_apply_worker, 1, &action); - /* Commit the per-stream transaction */ - CommitTransactionCommand(); + elog(DEBUG1, "stopped streaming of xid %u, %u changes streamed", stream_xid, nchanges); + } + else + { + /* + * Close the file with serialized changes, and serialize information + * about subxacts for the toplevel transaction. + */ + subxact_info_write(MyLogicalRepWorker->subid, stream_xid); + stream_close_file(); - in_streamed_transaction = false; + /* We must be in a valid transaction state */ + Assert(IsTransactionState()); - /* Reset per-stream context */ - MemoryContextReset(LogicalStreamingContext); + /* Commit the per-stream transaction */ + CommitTransactionCommand(); + + /* Reset per-stream context */ + MemoryContextReset(LogicalStreamingContext); + } + + in_streamed_transaction = false; + stream_apply_worker = NULL; pgstat_report_activity(STATE_IDLE, NULL); reset_apply_error_context_info(); } /* - * Handle STREAM abort message. + * Handle STREAM ABORT message when the transaction was spilled to disk. */ static void -apply_handle_stream_abort(StringInfo s) +serialize_stream_abort(TransactionId xid, TransactionId subxid) { - TransactionId xid; - TransactionId subxid; - - if (in_streamed_transaction) - ereport(ERROR, - (errcode(ERRCODE_PROTOCOL_VIOLATION), - errmsg_internal("STREAM ABORT message without STREAM STOP"))); - - logicalrep_read_stream_abort(s, &xid, &subxid); - /* * If the two XIDs are the same, it's in fact abort of toplevel xact, so * just delete the files with serialized info. */ if (xid == subxid) - { - set_apply_error_context_xact(xid, InvalidXLogRecPtr); stream_cleanup_files(MyLogicalRepWorker->subid, xid); - } else { /* @@ -1290,8 +1453,6 @@ apply_handle_stream_abort(StringInfo s) bool found = false; char path[MAXPGPATH]; - set_apply_error_context_xact(subxid, InvalidXLogRecPtr); - subidx = -1; begin_replication_step(); subxact_info_read(MyLogicalRepWorker->subid, xid); @@ -1316,7 +1477,6 @@ apply_handle_stream_abort(StringInfo s) cleanup_subxact_info(); end_replication_step(); CommitTransactionCommand(); - reset_apply_error_context_info(); return; } @@ -1339,6 +1499,142 @@ apply_handle_stream_abort(StringInfo s) end_replication_step(); CommitTransactionCommand(); } +} + +/* + * Handle STREAM ABORT message. + */ +static void +apply_handle_stream_abort(StringInfo s) +{ + TransactionId xid; + TransactionId subxid; + LogicalRepStreamAbortData abort_data; + bool read_abort_lsn = false; + + if (in_streamed_transaction) + ereport(ERROR, + (errcode(ERRCODE_PROTOCOL_VIOLATION), + errmsg_internal("STREAM ABORT message without STREAM STOP"))); + + /* Check whether the publisher sends abort_lsn and abort_time. */ + if (am_apply_bgworker()) + read_abort_lsn = MyParallelState->server_version >= 160000; + + logicalrep_read_stream_abort(s, &abort_data, read_abort_lsn); + + xid = abort_data.xid; + subxid = abort_data.subxid; + + set_apply_error_context_xact(subxid, abort_data.abort_lsn); + + if (am_apply_bgworker()) + { + elog(DEBUG1, "[Apply BGW #%u] aborting current transaction xid=%u, subxid=%u", + MyParallelState->n, GetCurrentTransactionIdIfAny(), + GetCurrentSubTransactionId()); + + /* + * Update origin state so we can restart streaming from correct + * position in case of crash. + */ + if (read_abort_lsn) + { + replorigin_session_origin_lsn = abort_data.abort_lsn; + replorigin_session_origin_timestamp = abort_data.abort_time; + } + + /* + * If the two XIDs are the same, it's in fact abort of toplevel xact, + * so just free the subxactlist. + */ + if (subxid == xid) + { + AbortCurrentTransaction(); + + EndTransactionBlock(false); + CommitTransactionCommand(); + + in_remote_transaction = false; + pgstat_report_activity(STATE_IDLE, NULL); + + list_free(subxactlist); + subxactlist = NIL; + + apply_bgworker_set_status(APPLY_BGWORKER_FINISHED); + } + else + { + /* + * OK, so it's a subxact. Rollback to the savepoint. + * + * We also need to read the subxactlist, determine the offset + * tracked for the subxact, and truncate the list. + */ + int i; + bool found = false; + char spname[MAXPGPATH]; + + snprintf(spname, MAXPGPATH, "savepoint_for_xid_%u", subxid); + + elog(DEBUG1, "[Apply BGW #%u] rolling back to savepoint %s", + MyParallelState->n, spname); + + for (i = list_length(subxactlist) - 1; i >= 0; i--) + { + xid = (TransactionId) list_nth_int(subxactlist, i); + if (xid == subxid) + { + found = true; + break; + } + } + + if (found) + { + RollbackToSavepoint(spname); + CommitTransactionCommand(); + subxactlist = list_truncate(subxactlist, i + 1); + } + + pgstat_report_activity(STATE_IDLEINTRANSACTION, NULL); + } + } + else + { + /* This is the main apply worker. */ + ApplyBgworkerState *wstate = apply_bgworker_find(xid); + + /* + * Check if we are processing this transaction in an apply background + * worker and if so, send the changes to that worker. + */ + if (wstate) + { + /* Send STREAM ABORT message to the apply background worker. */ + apply_bgworker_send_data(wstate, s->len, s->data); + + /* + * After sending the data to the apply background worker, wait for + * that worker to finish. This is necessary to maintain commit + * order which avoids failures due to transaction dependencies and + * deadlocks. + */ + if (subxid == xid) + { + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_FINISHED); + apply_bgworker_free(wstate); + } + } + else + { + /* + * We are in main apply worker and the transaction has been + * serialized to file. + */ + serialize_stream_abort(xid, subxid); + } + } reset_apply_error_context_info(); } @@ -1468,8 +1764,8 @@ apply_spooled_messages(TransactionId xid, XLogRecPtr lsn) static void apply_handle_stream_commit(StringInfo s) { - TransactionId xid; LogicalRepCommitData commit_data; + TransactionId xid; if (in_streamed_transaction) ereport(ERROR, @@ -1479,14 +1775,81 @@ apply_handle_stream_commit(StringInfo s) xid = logicalrep_read_stream_commit(s, &commit_data); set_apply_error_context_xact(xid, commit_data.commit_lsn); - elog(DEBUG1, "received commit for streamed transaction %u", xid); + if (am_apply_bgworker()) + { + /* + * Update origin state so we can restart streaming from correct + * position in case of crash. + */ + replorigin_session_origin_lsn = commit_data.end_lsn; + replorigin_session_origin_timestamp = commit_data.committime; + + CommitTransactionCommand(); + EndTransactionBlock(false); + CommitTransactionCommand(); - apply_spooled_messages(xid, commit_data.commit_lsn); + in_remote_transaction = false; - apply_handle_commit_internal(&commit_data); + pgstat_report_stat(false); + + list_free(subxactlist); + subxactlist = NIL; + + apply_bgworker_set_status(APPLY_BGWORKER_FINISHED); + } + else + { + /* This is the main apply worker. */ + ApplyBgworkerState *wstate = apply_bgworker_find(xid); + + elog(DEBUG1, "received commit for streamed transaction %u", xid); + + /* + * Check if we are processing this transaction in an apply background + * worker and if so, send the changes to that worker. + */ + if (wstate) + { + /* Send STREAM COMMIT message to the apply background worker. */ + apply_bgworker_send_data(wstate, s->len, s->data); + + /* + * After sending the data to the apply background worker, wait for + * that worker to finish. This is necessary to maintain commit + * order which avoids failures due to transaction dependencies and + * deadlocks. + */ + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_FINISHED); - /* unlink the files with serialized changes and subxact info */ - stream_cleanup_files(MyLogicalRepWorker->subid, xid); + pgstat_report_stat(false); + store_flush_position(commit_data.end_lsn); + stop_skipping_changes(); + + apply_bgworker_free(wstate); + + /* + * The transaction is either non-empty or skipped, so we clear the + * subskiplsn. + */ + clear_subscription_skip_lsn(commit_data.commit_lsn); + } + else + { + /* + * The transaction has been serialized to file, so replay all the + * spooled operations. + */ + apply_spooled_messages(xid, commit_data.commit_lsn); + + apply_handle_commit_internal(&commit_data); + + /* Unlink the files with serialized changes and subxact info. */ + stream_cleanup_files(MyLogicalRepWorker->subid, xid); + } + } + + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(commit_data.end_lsn); @@ -2467,7 +2830,7 @@ apply_handle_truncate(StringInfo s) /* * Logical replication protocol message dispatcher. */ -static void +void apply_dispatch(StringInfo s) { LogicalRepMsgType action = pq_getmsgbyte(s); @@ -2636,6 +2999,10 @@ store_flush_position(XLogRecPtr remote_lsn) { FlushPosition *flushpos; + /* We only need to collect the LSN in main apply worker */ + if (am_apply_bgworker()) + return; + /* Need to do this in permanent context */ MemoryContextSwitchTo(ApplyContext); @@ -2650,7 +3017,7 @@ store_flush_position(XLogRecPtr remote_lsn) /* Update statistics of the worker. */ -static void +void UpdateWorkerStats(XLogRecPtr last_lsn, TimestampTz send_time, bool reply) { MyLogicalRepWorker->last_lsn = last_lsn; @@ -2812,6 +3179,9 @@ LogicalRepApplyLoop(XLogRecPtr last_received) AcceptInvalidationMessages(); maybe_reread_subscription(); + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); + /* Process any table synchronization changes. */ process_syncing_tables(last_received); } @@ -3113,7 +3483,7 @@ maybe_reread_subscription(void) /* * Callback from subscription syscache invalidation. */ -static void +void subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue) { MySubscriptionValid = false; @@ -3709,7 +4079,7 @@ ApplyWorkerMain(Datum main_arg) originid = replorigin_by_name(originname, true); if (!OidIsValid(originid)) originid = replorigin_create(originname); - replorigin_session_setup(originid); + replorigin_session_setup(originid, true); replorigin_session_origin = originid; origin_startpos = replorigin_session_get_progress(false); CommitTransactionCommand(); @@ -3750,13 +4120,14 @@ ApplyWorkerMain(Datum main_arg) server_version = walrcv_server_version(LogRepWorkerWalRcvConn); options.proto.logical.proto_version = + server_version >= 160000 ? LOGICALREP_PROTO_STREAM_PARALLEL_VERSION_NUM : server_version >= 150000 ? LOGICALREP_PROTO_TWOPHASE_VERSION_NUM : server_version >= 140000 ? LOGICALREP_PROTO_STREAM_VERSION_NUM : LOGICALREP_PROTO_VERSION_NUM; options.proto.logical.publication_names = MySubscription->publications; options.proto.logical.binary = MySubscription->binary; - options.proto.logical.streaming = MySubscription->stream; + options.proto.logical.streaming = (MySubscription->stream != SUBSTREAM_OFF); options.proto.logical.twophase = false; if (!am_tablesync_worker()) @@ -3914,7 +4285,8 @@ clear_subscription_skip_lsn(XLogRecPtr finish_lsn) XLogRecPtr myskiplsn = MySubscription->skiplsn; bool started_tx = false; - if (likely(XLogRecPtrIsInvalid(myskiplsn))) + if (likely(XLogRecPtrIsInvalid(myskiplsn)) || + am_apply_bgworker()) return; if (!IsTransactionState()) @@ -3986,7 +4358,7 @@ clear_subscription_skip_lsn(XLogRecPtr finish_lsn) } /* Error callback to give more context info about the change being applied */ -static void +void apply_error_callback(void *arg) { ApplyErrorCallbackArg *errarg = &apply_error_callback_arg; @@ -4014,23 +4386,47 @@ apply_error_callback(void *arg) errarg->remote_xid, LSN_FORMAT_ARGS(errarg->finish_lsn)); } - else if (errarg->remote_attnum < 0) - errcontext("processing remote data for replication origin \"%s\" during \"%s\" for replication target relation \"%s.%s\" in transaction %u finished at %X/%X", - errarg->origin_name, - logicalrep_message_type(errarg->command), - errarg->rel->remoterel.nspname, - errarg->rel->remoterel.relname, - errarg->remote_xid, - LSN_FORMAT_ARGS(errarg->finish_lsn)); else - errcontext("processing remote data for replication origin \"%s\" during \"%s\" for replication target relation \"%s.%s\" column \"%s\" in transaction %u finished at %X/%X", - errarg->origin_name, - logicalrep_message_type(errarg->command), - errarg->rel->remoterel.nspname, - errarg->rel->remoterel.relname, - errarg->rel->remoterel.attnames[errarg->remote_attnum], - errarg->remote_xid, - LSN_FORMAT_ARGS(errarg->finish_lsn)); + { + if (errarg->remote_attnum < 0) + { + if (XLogRecPtrIsInvalid(errarg->finish_lsn)) + errcontext("processing remote data for replication origin \"%s\" during \"%s\" for replication target relation \"%s.%s\" in transaction %u", + errarg->origin_name, + logicalrep_message_type(errarg->command), + errarg->rel->remoterel.nspname, + errarg->rel->remoterel.relname, + errarg->remote_xid); + else + errcontext("processing remote data for replication origin \"%s\" during \"%s\" for replication target relation \"%s.%s\" in transaction %u finished at %X/%X", + errarg->origin_name, + logicalrep_message_type(errarg->command), + errarg->rel->remoterel.nspname, + errarg->rel->remoterel.relname, + errarg->remote_xid, + LSN_FORMAT_ARGS(errarg->finish_lsn)); + } + else + { + if (XLogRecPtrIsInvalid(errarg->finish_lsn)) + errcontext("processing remote data for replication origin \"%s\" during \"%s\" for replication target relation \"%s.%s\" column \"%s\" in transaction %u", + errarg->origin_name, + logicalrep_message_type(errarg->command), + errarg->rel->remoterel.nspname, + errarg->rel->remoterel.relname, + errarg->rel->remoterel.attnames[errarg->remote_attnum], + errarg->remote_xid); + else + errcontext("processing remote data for replication origin \"%s\" during \"%s\" for replication target relation \"%s.%s\" column \"%s\" in transaction %u finished at %X/%X", + errarg->origin_name, + logicalrep_message_type(errarg->command), + errarg->rel->remoterel.nspname, + errarg->rel->remoterel.relname, + errarg->rel->remoterel.attnames[errarg->remote_attnum], + errarg->remote_xid, + LSN_FORMAT_ARGS(errarg->finish_lsn)); + } + } } /* Set transaction information of apply error callback */ diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c index 2cbca4a087..1aaca04982 100644 --- a/src/backend/replication/pgoutput/pgoutput.c +++ b/src/backend/replication/pgoutput/pgoutput.c @@ -1820,6 +1820,8 @@ pgoutput_stream_abort(struct LogicalDecodingContext *ctx, XLogRecPtr abort_lsn) { ReorderBufferTXN *toptxn; + bool write_abort_lsn = false; + PGOutputData *data = (PGOutputData *) ctx->output_plugin_private; /* * The abort should happen outside streaming block, even for streamed @@ -1832,8 +1834,13 @@ pgoutput_stream_abort(struct LogicalDecodingContext *ctx, Assert(rbtxn_is_streamed(toptxn)); + /* We only send abort_lsn and abort_time if the subscriber needs them. */ + if (data->protocol_version >= LOGICALREP_PROTO_STREAM_PARALLEL_VERSION_NUM) + write_abort_lsn = true; + OutputPluginPrepareWrite(ctx, true); - logicalrep_write_stream_abort(ctx->out, toptxn->xid, txn->xid); + logicalrep_write_stream_abort(ctx->out, toptxn->xid, txn, abort_lsn, + write_abort_lsn); OutputPluginWrite(ctx, true); cleanup_rel_sync_cache(toptxn->xid, false); diff --git a/src/backend/utils/activity/wait_event.c b/src/backend/utils/activity/wait_event.c index 87c15b9c6f..ba781e6f08 100644 --- a/src/backend/utils/activity/wait_event.c +++ b/src/backend/utils/activity/wait_event.c @@ -388,6 +388,9 @@ pgstat_get_wait_ipc(WaitEventIPC w) case WAIT_EVENT_HASH_GROW_BUCKETS_REINSERT: event_name = "HashGrowBucketsReinsert"; break; + case WAIT_EVENT_LOGICAL_APPLY_WORKER_STATE_CHANGE: + event_name = "LogicalApplyWorkerStateChange"; + break; case WAIT_EVENT_LOGICAL_SYNC_DATA: event_name = "LogicalSyncData"; break; diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c index 0328029d43..4284bcbcd1 100644 --- a/src/backend/utils/misc/guc.c +++ b/src/backend/utils/misc/guc.c @@ -3220,6 +3220,18 @@ static struct config_int ConfigureNamesInt[] = NULL, NULL, NULL }, + { + {"max_apply_bgworkers_per_subscription", + PGC_SIGHUP, + REPLICATION_SUBSCRIBERS, + gettext_noop("Maximum number of apply background workers per subscription."), + NULL, + }, + &max_apply_bgworkers_per_subscription, + 2, 0, MAX_BACKENDS, + NULL, NULL, NULL + }, + { {"log_rotation_age", PGC_SIGHUP, LOGGING_WHERE, gettext_noop("Sets the amount of time to wait before forcing " diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample index b4bc06e5f5..ad18710af4 100644 --- a/src/backend/utils/misc/postgresql.conf.sample +++ b/src/backend/utils/misc/postgresql.conf.sample @@ -360,6 +360,7 @@ #max_logical_replication_workers = 4 # taken from max_worker_processes # (change requires restart) #max_sync_workers_per_subscription = 2 # taken from max_logical_replication_workers +#max_apply_bgworkers_per_subscription = 2 # taken from max_logical_replication_workers #------------------------------------------------------------------------------ diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index f317f0a681..24927641b9 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -4449,7 +4449,7 @@ getSubscriptions(Archive *fout) if (fout->remoteVersion >= 140000) appendPQExpBufferStr(query, " s.substream,\n"); else - appendPQExpBufferStr(query, " false AS substream,\n"); + appendPQExpBufferStr(query, " 'f' AS substream,\n"); if (fout->remoteVersion >= 150000) appendPQExpBufferStr(query, @@ -4579,8 +4579,10 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo) if (strcmp(subinfo->subbinary, "t") == 0) appendPQExpBufferStr(query, ", binary = true"); - if (strcmp(subinfo->substream, "f") != 0) + if (strcmp(subinfo->substream, "t") == 0) appendPQExpBufferStr(query, ", streaming = on"); + else if (strcmp(subinfo->substream, "p") == 0) + appendPQExpBufferStr(query, ", streaming = parallel"); if (strcmp(subinfo->subtwophasestate, two_phase_disabled) != 0) appendPQExpBufferStr(query, ", two_phase = on"); diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h index d1260f590c..d54540f5f5 100644 --- a/src/include/catalog/pg_subscription.h +++ b/src/include/catalog/pg_subscription.h @@ -68,7 +68,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW bool subbinary; /* True if the subscription wants the * publisher to send data in binary */ - bool substream; /* Stream in-progress transactions. */ + char substream; /* Stream in-progress transactions. + * See SUBSTREAM_xxx constants. */ char subtwophasestate; /* Stream two-phase transactions */ @@ -109,7 +110,8 @@ typedef struct Subscription bool enabled; /* Indicates if the subscription is enabled */ bool binary; /* Indicates if the subscription wants data in * binary format */ - bool stream; /* Allow streaming in-progress transactions. */ + char stream; /* Allow streaming in-progress transactions. + * See SUBSTREAM_xxx constants. */ char twophasestate; /* Allow streaming two-phase transactions */ bool disableonerr; /* Indicates if the subscription should be * automatically disabled if a worker error @@ -120,6 +122,21 @@ typedef struct Subscription List *publications; /* List of publication names to subscribe to */ } Subscription; +/* Disallow streaming in-progress transactions */ +#define SUBSTREAM_OFF 'f' + +/* + * Streaming in-progress transactions are written to a temporary file and + * applied only after the transaction is committed on upstream. + */ +#define SUBSTREAM_ON 't' + +/* + * Streaming in-progress transactions are applied immediately via a background + * worker + */ +#define SUBSTREAM_PARALLEL 'p' + extern Subscription *GetSubscription(Oid subid, bool missing_ok); extern void FreeSubscription(Subscription *sub); extern void DisableSubscription(Oid subid); diff --git a/src/include/replication/logicallauncher.h b/src/include/replication/logicallauncher.h index f1e2821e25..ac8ef94381 100644 --- a/src/include/replication/logicallauncher.h +++ b/src/include/replication/logicallauncher.h @@ -14,6 +14,7 @@ extern PGDLLIMPORT int max_logical_replication_workers; extern PGDLLIMPORT int max_sync_workers_per_subscription; +extern PGDLLIMPORT int max_apply_bgworkers_per_subscription; extern void ApplyLauncherRegister(void); extern void ApplyLauncherMain(Datum main_arg); diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h index a771ab8ff3..0f74a3392b 100644 --- a/src/include/replication/logicalproto.h +++ b/src/include/replication/logicalproto.h @@ -32,12 +32,17 @@ * * LOGICALREP_PROTO_TWOPHASE_VERSION_NUM is the minimum protocol version with * support for two-phase commit decoding (at prepare time). Introduced in PG15. + * + * LOGICALREP_PROTO_STREAM_PARALLEL_VERSION_NUM is the minimum protocol version + * with support for streaming large transactions in apply background worker. + * Introduced in PG16. */ #define LOGICALREP_PROTO_MIN_VERSION_NUM 1 #define LOGICALREP_PROTO_VERSION_NUM 1 #define LOGICALREP_PROTO_STREAM_VERSION_NUM 2 #define LOGICALREP_PROTO_TWOPHASE_VERSION_NUM 3 -#define LOGICALREP_PROTO_MAX_VERSION_NUM LOGICALREP_PROTO_TWOPHASE_VERSION_NUM +#define LOGICALREP_PROTO_STREAM_PARALLEL_VERSION_NUM 4 +#define LOGICALREP_PROTO_MAX_VERSION_NUM LOGICALREP_PROTO_STREAM_PARALLEL_VERSION_NUM /* * Logical message types @@ -175,6 +180,17 @@ typedef struct LogicalRepRollbackPreparedTxnData char gid[GIDSIZE]; } LogicalRepRollbackPreparedTxnData; +/* + * Transaction protocol information for stream abort. + */ +typedef struct LogicalRepStreamAbortData +{ + TransactionId xid; + TransactionId subxid; + XLogRecPtr abort_lsn; + TimestampTz abort_time; +} LogicalRepStreamAbortData; + extern void logicalrep_write_begin(StringInfo out, ReorderBufferTXN *txn); extern void logicalrep_read_begin(StringInfo in, LogicalRepBeginData *begin_data); @@ -246,9 +262,12 @@ extern void logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn extern TransactionId logicalrep_read_stream_commit(StringInfo out, LogicalRepCommitData *commit_data); extern void logicalrep_write_stream_abort(StringInfo out, TransactionId xid, - TransactionId subxid); -extern void logicalrep_read_stream_abort(StringInfo in, TransactionId *xid, - TransactionId *subxid); + ReorderBufferTXN *txn, + XLogRecPtr abort_lsn, + bool write_abort_lsn); +extern void logicalrep_read_stream_abort(StringInfo in, + LogicalRepStreamAbortData *abort_data, + bool include_abort_lsn); extern char *logicalrep_message_type(LogicalRepMsgType action); #endif /* LOGICAL_PROTO_H */ diff --git a/src/include/replication/logicalworker.h b/src/include/replication/logicalworker.h index cd1b6e8afc..6a1af7f13c 100644 --- a/src/include/replication/logicalworker.h +++ b/src/include/replication/logicalworker.h @@ -13,6 +13,7 @@ #define LOGICALWORKER_H extern void ApplyWorkerMain(Datum main_arg); +extern void ApplyBgworkerMain(Datum main_arg); extern bool IsLogicalWorker(void); diff --git a/src/include/replication/origin.h b/src/include/replication/origin.h index 2d1b5e5c28..c7389b40a7 100644 --- a/src/include/replication/origin.h +++ b/src/include/replication/origin.h @@ -53,7 +53,7 @@ extern XLogRecPtr replorigin_get_progress(RepOriginId node, bool flush); extern void replorigin_session_advance(XLogRecPtr remote_commit, XLogRecPtr local_commit); -extern void replorigin_session_setup(RepOriginId node); +extern void replorigin_session_setup(RepOriginId node, bool must_acquire); extern void replorigin_session_reset(void); extern XLogRecPtr replorigin_session_get_progress(bool flush); diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h index d109d0baed..d2a80d79e5 100644 --- a/src/include/replication/reorderbuffer.h +++ b/src/include/replication/reorderbuffer.h @@ -301,6 +301,7 @@ typedef struct ReorderBufferTXN { TimestampTz commit_time; TimestampTz prepare_time; + TimestampTz abort_time; } xact_time; /* @@ -647,9 +648,11 @@ extern void ReorderBufferFinishPrepared(ReorderBuffer *rb, TransactionId xid, extern void ReorderBufferAssignChild(ReorderBuffer *, TransactionId, TransactionId, XLogRecPtr commit_lsn); extern void ReorderBufferCommitChild(ReorderBuffer *, TransactionId, TransactionId, XLogRecPtr commit_lsn, XLogRecPtr end_lsn); -extern void ReorderBufferAbort(ReorderBuffer *, TransactionId, XLogRecPtr lsn); +extern void ReorderBufferAbort(ReorderBuffer *, TransactionId, XLogRecPtr lsn, + TimestampTz abort_time); extern void ReorderBufferAbortOld(ReorderBuffer *, TransactionId xid); -extern void ReorderBufferForget(ReorderBuffer *, TransactionId, XLogRecPtr lsn); +extern void ReorderBufferForget(ReorderBuffer *, TransactionId, XLogRecPtr lsn, + TimestampTz abort_time); extern void ReorderBufferInvalidate(ReorderBuffer *, TransactionId, XLogRecPtr lsn); extern void ReorderBufferSetBaseSnapshot(ReorderBuffer *, TransactionId, XLogRecPtr lsn, struct SnapshotData *snap); diff --git a/src/include/replication/worker_internal.h b/src/include/replication/worker_internal.h index 901845abc2..5be8f5755e 100644 --- a/src/include/replication/worker_internal.h +++ b/src/include/replication/worker_internal.h @@ -17,8 +17,11 @@ #include "access/xlogdefs.h" #include "catalog/pg_subscription.h" #include "datatype/timestamp.h" +#include "replication/logicalrelation.h" #include "storage/fileset.h" #include "storage/lock.h" +#include "storage/shm_mq.h" +#include "storage/shm_toc.h" #include "storage/spin.h" @@ -60,6 +63,9 @@ typedef struct LogicalRepWorker */ FileSet *stream_fileset; + /* Indicates if this slot is used for an apply background worker. */ + bool subworker; + /* Stats. */ XLogRecPtr last_lsn; TimestampTz last_send_time; @@ -68,8 +74,68 @@ typedef struct LogicalRepWorker TimestampTz reply_time; } LogicalRepWorker; +/* Struct for saving and restoring apply errcontext information */ +typedef struct ApplyErrorCallbackArg +{ + LogicalRepMsgType command; /* 0 if invalid */ + LogicalRepRelMapEntry *rel; + + /* Remote node information */ + int remote_attnum; /* -1 if invalid */ + TransactionId remote_xid; + XLogRecPtr finish_lsn; + char *origin_name; +} ApplyErrorCallbackArg; + +/* + * Status of apply background worker. + */ +typedef enum ApplyBgworkerStatus +{ + APPLY_BGWORKER_BUSY = 0, /* assigned to a transaction */ + APPLY_BGWORKER_FINISHED, /* transaction is completed */ + APPLY_BGWORKER_EXIT /* exit */ +} ApplyBgworkerStatus; + +/* + * Struct for sharing information between apply main and apply background + * workers. + */ +typedef struct ApplyBgworkerShared +{ + slock_t mutex; + + /* Status of apply background worker. */ + ApplyBgworkerStatus status; + + /* server version of publisher. */ + int server_version; + + TransactionId stream_xid; + uint32 n; /* id of apply background worker */ +} ApplyBgworkerShared; + +/* + * Struct for maintaining an apply background worker. + */ +typedef struct ApplyBgworkerState +{ + shm_mq_handle *mq_handle; + dsm_segment *dsm_seg; + ApplyBgworkerShared volatile *pstate; +} ApplyBgworkerState; + /* Main memory context for apply worker. Permanent during worker lifetime. */ extern PGDLLIMPORT MemoryContext ApplyContext; +extern PGDLLIMPORT MemoryContext ApplyMessageContext; + +extern PGDLLIMPORT ApplyErrorCallbackArg apply_error_callback_arg; + +extern PGDLLIMPORT bool MySubscriptionValid; + +extern PGDLLIMPORT volatile ApplyBgworkerShared *MyParallelState; + +extern PGDLLIMPORT List *subxactlist; /* libpqreceiver connection */ extern PGDLLIMPORT struct WalReceiverConn *LogRepWorkerWalRcvConn; @@ -79,18 +145,22 @@ extern PGDLLIMPORT Subscription *MySubscription; extern PGDLLIMPORT LogicalRepWorker *MyLogicalRepWorker; extern PGDLLIMPORT bool in_remote_transaction; +extern PGDLLIMPORT bool in_streamed_transaction; +extern PGDLLIMPORT TransactionId stream_xid; extern void logicalrep_worker_attach(int slot); extern LogicalRepWorker *logicalrep_worker_find(Oid subid, Oid relid, bool only_running); extern List *logicalrep_workers_find(Oid subid, bool only_running); -extern void logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, - Oid userid, Oid relid); +extern bool logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, + Oid userid, Oid relid, + dsm_handle subworker_dsm); extern void logicalrep_worker_stop(Oid subid, Oid relid); extern void logicalrep_worker_wakeup(Oid subid, Oid relid); extern void logicalrep_worker_wakeup_ptr(LogicalRepWorker *worker); extern int logicalrep_sync_worker_count(Oid subid); +extern int logicalrep_apply_bgworker_count(Oid subid); extern void ReplicationOriginNameForTablesync(Oid suboid, Oid relid, char *originname, int szorgname); @@ -103,10 +173,38 @@ extern void process_syncing_tables(XLogRecPtr current_lsn); extern void invalidate_syncing_table_states(Datum arg, int cacheid, uint32 hashvalue); +extern void UpdateWorkerStats(XLogRecPtr last_lsn, TimestampTz send_time, + bool reply); + +extern void apply_dispatch(StringInfo s); + +/* Function for apply error callback */ +extern void apply_error_callback(void *arg); + +extern void subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue); + +/* Apply background worker setup and interactions */ +extern ApplyBgworkerState *apply_bgworker_start(TransactionId xid); +extern ApplyBgworkerState *apply_bgworker_find(TransactionId xid); +extern void apply_bgworker_wait_for(ApplyBgworkerState *wstate, + ApplyBgworkerStatus wait_for_status); +extern void apply_bgworker_send_data(ApplyBgworkerState *wstate, Size nbytes, + const void *data); +extern void apply_bgworker_free(ApplyBgworkerState *wstate); +extern void apply_bgworker_check_status(void); +extern void apply_bgworker_set_status(ApplyBgworkerStatus status); +extern void apply_bgworker_subxact_info_add(TransactionId current_xid); + static inline bool am_tablesync_worker(void) { return OidIsValid(MyLogicalRepWorker->relid); } +static inline bool +am_apply_bgworker(void) +{ + return MyLogicalRepWorker->subworker; +} + #endif /* WORKER_INTERNAL_H */ diff --git a/src/include/utils/wait_event.h b/src/include/utils/wait_event.h index b578e2ec75..c2d2a114d7 100644 --- a/src/include/utils/wait_event.h +++ b/src/include/utils/wait_event.h @@ -105,6 +105,7 @@ typedef enum WAIT_EVENT_HASH_GROW_BUCKETS_ALLOCATE, WAIT_EVENT_HASH_GROW_BUCKETS_ELECT, WAIT_EVENT_HASH_GROW_BUCKETS_REINSERT, + WAIT_EVENT_LOGICAL_APPLY_WORKER_STATE_CHANGE, WAIT_EVENT_LOGICAL_SYNC_DATA, WAIT_EVENT_LOGICAL_SYNC_STATE_CHANGE, WAIT_EVENT_MQ_INTERNAL, diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out index 5db7146e06..919266ae06 100644 --- a/src/test/regress/expected/subscription.out +++ b/src/test/regress/expected/subscription.out @@ -197,7 +197,7 @@ ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); DROP SUBSCRIPTION regress_testsub; -- fail - streaming must be boolean CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = foo); -ERROR: streaming requires a Boolean value +ERROR: streaming requires a Boolean value or "parallel" -- now it works CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = true); WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 34a76ceb60..4137dc77b4 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -116,6 +116,10 @@ Append AppendPath AppendRelInfo AppendState +ApplyBgworkerEntry +ApplyBgworkerShared +ApplyBgworkerState +ApplyBgworkerStatus ApplyErrorCallbackArg ApplyExecutionData ApplySubXactData @@ -1485,6 +1489,7 @@ LogicalRepRelId LogicalRepRelMapEntry LogicalRepRelation LogicalRepRollbackPreparedTxnData +LogicalRepStreamAbortData LogicalRepTupleData LogicalRepTyp LogicalRepWorker -- 2.23.0.windows.1 [application/octet-stream] v15-0002-Test-streaming-parallel-option-in-tap-test.patch (69.1K, ../../OS3PR01MB62755C6C9A75EB09F7218B589E839@OS3PR01MB6275.jpnprd01.prod.outlook.com/3-v15-0002-Test-streaming-parallel-option-in-tap-test.patch) download | inline diff: From fcffbe96cfb5e38ff3befd46b12b6dfdda9ba843 Mon Sep 17 00:00:00 2001 From: "shiy.fnst" <[email protected]> Date: Fri, 13 May 2022 14:50:30 +0800 Subject: [PATCH v15 2/4] Test streaming parallel option in tap test Change all TAP tests using the SUBSCRIPTION "streaming" parameter, so they now test both 'on' and 'parallel' values. --- src/test/subscription/t/015_stream.pl | 199 ++++--- src/test/subscription/t/016_stream_subxact.pl | 119 +++-- src/test/subscription/t/017_stream_ddl.pl | 188 ++++--- .../t/018_stream_subxact_abort.pl | 195 ++++--- .../t/019_stream_subxact_ddl_abort.pl | 110 +++- .../subscription/t/022_twophase_cascade.pl | 363 +++++++------ .../subscription/t/023_twophase_stream.pl | 498 ++++++++++-------- 7 files changed, 1035 insertions(+), 637 deletions(-) diff --git a/src/test/subscription/t/015_stream.pl b/src/test/subscription/t/015_stream.pl index 6561b189de..0bdd234935 100644 --- a/src/test/subscription/t/015_stream.pl +++ b/src/test/subscription/t/015_stream.pl @@ -8,6 +8,116 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" +# parameter so the same code can be run both for the streaming=on and +# streaming=parallel cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_parallel) = @_; + + # Interleave a pair of transactions, each exceeding the 64kB limit. + my $in = ''; + my $out = ''; + + my $timer = IPC::Run::timeout($PostgreSQL::Test::Utils::timeout_default); + + my $h = $node_publisher->background_psql('postgres', \$in, \$out, $timer, + on_error_stop => 0); + + # Check that a background worker starts if "streaming" parameter is + # specified as "parallel". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_parallel) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + $in .= q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + }; + $h->pump_nb; + + if ($is_parallel) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001, 9999) s(i); + DELETE FROM test_tab WHERE a > 5000; + COMMIT; + }); + + $in .= q{ + COMMIT; + \q + }; + $h->finish; # errors make the next test fail, so ignore them here + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), + 'check extra columns contain local defaults'); + + # Test the streaming in binary mode + $node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET (binary = on)"); + + # Insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001, 10000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + COMMIT; + }); + + $node_publisher->wait_for_catchup($appname); + + $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(6667|6667|6667), + 'check extra columns contain local defaults'); + + # Change the local values of the extra columns on the subscriber, + # update publisher, and check that subscriber retains the expected + # values. This is to ensure that non-streaming transactions behave + # properly after a streaming transaction. + $node_subscriber->safe_psql('postgres', + "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'" + ); + $node_publisher->safe_psql('postgres', + "UPDATE test_tab SET b = md5(a::text)"); + + $node_publisher->wait_for_catchup($appname); + + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab" + ); + is($result, qq(6667|6667|6667), + 'check extra columns contain locally changed data'); + + # Cleanup the test data + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE (a > 2)"); + $node_publisher->wait_for_catchup($appname); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -37,6 +147,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -54,82 +168,25 @@ my $result = "SELECT count(*), count(c), count(d = 999) FROM test_tab"); is($result, qq(2|2|2), 'check initial data was copied to subscriber'); -# Interleave a pair of transactions, each exceeding the 64kB limit. -my $in = ''; -my $out = ''; - -my $timer = IPC::Run::timeout($PostgreSQL::Test::Utils::timeout_default); - -my $h = $node_publisher->background_psql('postgres', \$in, \$out, $timer, - on_error_stop => 0); - -$in .= q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -}; -$h->pump_nb; - -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001, 9999) s(i); -DELETE FROM test_tab WHERE a > 5000; -COMMIT; -}); - -$in .= q{ -COMMIT; -\q -}; -$h->finish; # errors make the next test fail, so ignore them here - -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), 'check extra columns contain local defaults'); - -# Test the streaming in binary mode -$node_subscriber->safe_psql('postgres', - "ALTER SUBSCRIPTION tap_sub SET (binary = on)"); - -# Insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001, 10000) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -COMMIT; -}); - -$node_publisher->wait_for_catchup($appname); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(6667|6667|6667), 'check extra columns contain local defaults'); +###################################### +# Test using streaming mode 'parallel' +###################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" +); -# Change the local values of the extra columns on the subscriber, -# update publisher, and check that subscriber retains the expected -# values. This is to ensure that non-streaming transactions behave -# properly after a streaming transaction. $node_subscriber->safe_psql('postgres', - "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'" -); -$node_publisher->safe_psql('postgres', - "UPDATE test_tab SET b = md5(a::text)"); + "ALTER SUBSCRIPTION tap_sub SET(streaming = parallel, binary = off)"); -$node_publisher->wait_for_catchup($appname); +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab" -); -is($result, qq(6667|6667|6667), - 'check extra columns contain locally changed data'); +test_streaming($node_publisher, $node_subscriber, $appname, 1); $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/016_stream_subxact.pl b/src/test/subscription/t/016_stream_subxact.pl index f27f1694f2..45429dddba 100644 --- a/src/test/subscription/t/016_stream_subxact.pl +++ b/src/test/subscription/t/016_stream_subxact.pl @@ -8,6 +8,72 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" +# parameter so the same code can be run both for the streaming=on and +# streaming=parallel cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_parallel) = @_; + + # Check that a background worker starts if "streaming" parameter is + # specified as "parallel". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_parallel) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + # Insert, update and delete enough rows to exceed 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series( 3, 500) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501, 1000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001, 1500) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + SAVEPOINT s3; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501, 2000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + SAVEPOINT s4; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001, 2500) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + COMMIT; + }); + + if ($is_parallel) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(1667|1667|1667), + 'check data was copied to subscriber in streaming mode and extra columns contain local defaults' + ); + + # Cleanup the test data + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE (a > 2)"); + $node_publisher->wait_for_catchup($appname); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -37,6 +103,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -54,41 +124,26 @@ my $result = "SELECT count(*), count(c), count(d = 999) FROM test_tab"); is($result, qq(2|2|2), 'check initial data was copied to subscriber'); -# Insert, update and delete enough rows to exceed 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series( 3, 500) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501, 1000) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001, 1500) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -SAVEPOINT s3; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501, 2000) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -SAVEPOINT s4; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001, 2500) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -COMMIT; -}); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(1667|1667|1667), - 'check data was copied to subscriber in streaming mode and extra columns contain local defaults' +###################################### +# Test using streaming mode 'parallel' +###################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" ); +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = parallel)"); + +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; + +test_streaming($node_publisher, $node_subscriber, $appname, 1); + $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/017_stream_ddl.pl b/src/test/subscription/t/017_stream_ddl.pl index 0bce63b716..52dfef4780 100644 --- a/src/test/subscription/t/017_stream_ddl.pl +++ b/src/test/subscription/t/017_stream_ddl.pl @@ -8,6 +8,111 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" +# parameter so the same code can be run both for the streaming=on and +# streaming=parallel cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_parallel) = @_; + + # a small (non-streamed) transaction with DDL and DML + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab VALUES (3, md5(3::text)); + ALTER TABLE test_tab ADD COLUMN c INT; + SAVEPOINT s1; + INSERT INTO test_tab VALUES (4, md5(4::text), -4); + COMMIT; + }); + + # Check that a background worker starts if "streaming" parameter is + # specified as "parallel". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_parallel) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + # large (streamed) transaction with DDL and DML + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(5, 1000) s(i); + ALTER TABLE test_tab ADD COLUMN d INT; + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001, 2000) s(i); + COMMIT; + }); + + if ($is_parallel) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + # a small (non-streamed) transaction with DDL and DML + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab VALUES (2001, md5(2001::text), -2001, 2*2001); + ALTER TABLE test_tab ADD COLUMN e INT; + SAVEPOINT s1; + INSERT INTO test_tab VALUES (2002, md5(2002::text), -2002, 2*2002, -3*2002); + COMMIT; + }); + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d), count(e) FROM test_tab"); + is($result, qq(2002|1999|1002|1), + 'check data was copied to subscriber in streaming mode and extra columns contain local defaults' + ); + + # A large (streamed) transaction with DDL and DML. One of the DDL is performed + # after DML to ensure that we invalidate the schema sent for test_tab so that + # the next transaction has to send the schema again. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(2003,5000) s(i); + ALTER TABLE test_tab ADD COLUMN f INT; + COMMIT; + }); + + # A small transaction that won't get streamed. This is just to ensure that we + # send the schema again to reflect the last column added in the previous test. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i, 4*i FROM generate_series(5001,5005) s(i); + COMMIT; + }); + + $node_publisher->wait_for_catchup($appname); + + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d), count(e), count(f) FROM test_tab" + ); + is($result, qq(5005|5002|4005|3004|5), + 'check data was copied to subscriber for both streaming and non-streaming transactions' + ); + + # Cleanup the test data + $node_publisher->safe_psql( + 'postgres', q{ + DELETE FROM test_tab WHERE (a > 2); + ALTER TABLE test_tab DROP COLUMN c, DROP COLUMN d, DROP COLUMN e, DROP COLUMN f; + }); + $node_publisher->wait_for_catchup($appname); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -37,6 +142,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -54,76 +163,25 @@ my $result = "SELECT count(*), count(c), count(d = 999) FROM test_tab"); is($result, qq(2|0|0), 'check initial data was copied to subscriber'); -# a small (non-streamed) transaction with DDL and DML -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab VALUES (3, md5(3::text)); -ALTER TABLE test_tab ADD COLUMN c INT; -SAVEPOINT s1; -INSERT INTO test_tab VALUES (4, md5(4::text), -4); -COMMIT; -}); - -# large (streamed) transaction with DDL and DML -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(5, 1000) s(i); -ALTER TABLE test_tab ADD COLUMN d INT; -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001, 2000) s(i); -COMMIT; -}); - -# a small (non-streamed) transaction with DDL and DML -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab VALUES (2001, md5(2001::text), -2001, 2*2001); -ALTER TABLE test_tab ADD COLUMN e INT; -SAVEPOINT s1; -INSERT INTO test_tab VALUES (2002, md5(2002::text), -2002, 2*2002, -3*2002); -COMMIT; -}); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d), count(e) FROM test_tab"); -is($result, qq(2002|1999|1002|1), - 'check data was copied to subscriber in streaming mode and extra columns contain local defaults' +###################################### +# Test using streaming mode 'parallel' +###################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" ); -# A large (streamed) transaction with DDL and DML. One of the DDL is performed -# after DML to ensure that we invalidate the schema sent for test_tab so that -# the next transaction has to send the schema again. -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(2003,5000) s(i); -ALTER TABLE test_tab ADD COLUMN f INT; -COMMIT; -}); - -# A small transaction that won't get streamed. This is just to ensure that we -# send the schema again to reflect the last column added in the previous test. -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i, 4*i FROM generate_series(5001,5005) s(i); -COMMIT; -}); +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = parallel)"); -$node_publisher->wait_for_catchup($appname); +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d), count(e), count(f) FROM test_tab"); -is($result, qq(5005|5002|4005|3004|5), - 'check data was copied to subscriber for both streaming and non-streaming transactions' -); +test_streaming($node_publisher, $node_subscriber, $appname, 1); $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/018_stream_subxact_abort.pl b/src/test/subscription/t/018_stream_subxact_abort.pl index 7155442e76..68f0e4b0d1 100644 --- a/src/test/subscription/t/018_stream_subxact_abort.pl +++ b/src/test/subscription/t/018_stream_subxact_abort.pl @@ -8,6 +8,113 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" +# parameter so the same code can be run both for the streaming=on and +# streaming=parallel cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_parallel) = @_; + + # Check that a background worker starts if "streaming" parameter is + # specified as "parallel". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_parallel) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + # large (streamed) transaction with DDL, DML and ROLLBACKs + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i); + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501,1000) s(i); + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001,1500) s(i); + SAVEPOINT s3; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501,2000) s(i); + ROLLBACK TO s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001,2500) s(i); + ROLLBACK TO s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2501,3000) s(i); + SAVEPOINT s4; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3001,3500) s(i); + SAVEPOINT s5; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3501,4000) s(i); + COMMIT; + }); + + if ($is_parallel) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c) FROM test_tab"); + is($result, qq(2000|0), + 'check rollback to savepoint was reflected on subscriber and extra columns contain local defaults' + ); + + # large (streamed) transaction with subscriber receiving out of order + # subtransaction ROLLBACKs + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(4001,4500) s(i); + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001,5500) s(i); + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(6001,6500) s(i); + SAVEPOINT s3; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(7001,7500) s(i); + RELEASE s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(8001,8500) s(i); + ROLLBACK TO s1; + COMMIT; + }); + + $node_publisher->wait_for_catchup($appname); + + $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c) FROM test_tab"); + is($result, qq(2500|0), + 'check rollback to savepoint was reflected on subscriber'); + + # large (streamed) transaction with subscriber receiving rollback + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(8501,9000) s(i); + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(9001,9500) s(i); + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(9501,10000) s(i); + ROLLBACK; + }); + + $node_publisher->wait_for_catchup($appname); + + $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c) FROM test_tab"); + is($result, qq(2500|0), 'check rollback was reflected on subscriber'); + + # Cleanup the test data + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE (a > 2)"); + $node_publisher->wait_for_catchup($appname); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -36,6 +143,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -53,81 +164,25 @@ my $result = "SELECT count(*), count(c) FROM test_tab"); is($result, qq(2|0), 'check initial data was copied to subscriber'); -# large (streamed) transaction with DDL, DML and ROLLBACKs -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i); -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501,1000) s(i); -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001,1500) s(i); -SAVEPOINT s3; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501,2000) s(i); -ROLLBACK TO s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001,2500) s(i); -ROLLBACK TO s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2501,3000) s(i); -SAVEPOINT s4; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3001,3500) s(i); -SAVEPOINT s5; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3501,4000) s(i); -COMMIT; -}); - -$node_publisher->wait_for_catchup($appname); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c) FROM test_tab"); -is($result, qq(2000|0), - 'check rollback to savepoint was reflected on subscriber and extra columns contain local defaults' +###################################### +# Test using streaming mode 'parallel' +###################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" ); -# large (streamed) transaction with subscriber receiving out of order -# subtransaction ROLLBACKs -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(4001,4500) s(i); -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001,5500) s(i); -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(6001,6500) s(i); -SAVEPOINT s3; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(7001,7500) s(i); -RELEASE s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(8001,8500) s(i); -ROLLBACK TO s1; -COMMIT; -}); - -$node_publisher->wait_for_catchup($appname); +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = parallel)"); -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c) FROM test_tab"); -is($result, qq(2500|0), - 'check rollback to savepoint was reflected on subscriber'); - -# large (streamed) transaction with subscriber receiving rollback -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(8501,9000) s(i); -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(9001,9500) s(i); -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(9501,10000) s(i); -ROLLBACK; -}); +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c) FROM test_tab"); -is($result, qq(2500|0), 'check rollback was reflected on subscriber'); +test_streaming($node_publisher, $node_subscriber, $appname, 1); $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/019_stream_subxact_ddl_abort.pl b/src/test/subscription/t/019_stream_subxact_ddl_abort.pl index dbd0fca4d1..b276063721 100644 --- a/src/test/subscription/t/019_stream_subxact_ddl_abort.pl +++ b/src/test/subscription/t/019_stream_subxact_ddl_abort.pl @@ -9,6 +9,69 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" +# parameter so the same code can be run both for the streaming=on and +# streaming=parallel cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_parallel) = @_; + + # Check that a background worker starts if "streaming" parameter is + # specified as "parallel". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_parallel) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + # large (streamed) transaction with DDL, DML and ROLLBACKs + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i); + ALTER TABLE test_tab ADD COLUMN c INT; + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(501,1000) s(i); + ALTER TABLE test_tab ADD COLUMN d INT; + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001,1500) s(i); + ALTER TABLE test_tab ADD COLUMN e INT; + SAVEPOINT s3; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(1501,2000) s(i); + ALTER TABLE test_tab DROP COLUMN c; + ROLLBACK TO s1; + INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(501,1000) s(i); + COMMIT; + }); + + if ($is_parallel) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c) FROM test_tab"); + is($result, qq(1000|500), + 'check rollback to savepoint was reflected on subscriber and extra columns contain local defaults' + ); + + # Cleanup the test data + $node_publisher->safe_psql( + 'postgres', q{ + DELETE FROM test_tab WHERE (a > 2); + ALTER TABLE test_tab DROP COLUMN c; + }); + $node_publisher->wait_for_catchup($appname); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -37,6 +100,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -54,35 +121,26 @@ my $result = "SELECT count(*), count(c) FROM test_tab"); is($result, qq(2|0), 'check initial data was copied to subscriber'); -# large (streamed) transaction with DDL, DML and ROLLBACKs -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i); -ALTER TABLE test_tab ADD COLUMN c INT; -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(501,1000) s(i); -ALTER TABLE test_tab ADD COLUMN d INT; -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001,1500) s(i); -ALTER TABLE test_tab ADD COLUMN e INT; -SAVEPOINT s3; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(1501,2000) s(i); -ALTER TABLE test_tab DROP COLUMN c; -ROLLBACK TO s1; -INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(501,1000) s(i); -COMMIT; -}); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c) FROM test_tab"); -is($result, qq(1000|500), - 'check rollback to savepoint was reflected on subscriber and extra columns contain local defaults' +###################################### +# Test using streaming mode 'parallel' +###################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" ); +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = parallel)"); + +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; + +test_streaming($node_publisher, $node_subscriber, $appname, 1); + $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/022_twophase_cascade.pl b/src/test/subscription/t/022_twophase_cascade.pl index 7a797f37ba..0a4152d3be 100644 --- a/src/test/subscription/t/022_twophase_cascade.pl +++ b/src/test/subscription/t/022_twophase_cascade.pl @@ -11,6 +11,208 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" parameter +# so the same code can be run both for the streaming=on and streaming=parallel +# cases. +sub test_streaming +{ + my ($node_A, $node_B, $node_C, $appname_B, $appname_C, $streaming_mode) = + @_; + + my $oldpid_B = $node_A->safe_psql( + 'postgres', " + SELECT pid FROM pg_stat_replication + WHERE application_name = '$appname_B' AND state = 'streaming';"); + my $oldpid_C = $node_B->safe_psql( + 'postgres', " + SELECT pid FROM pg_stat_replication + WHERE application_name = '$appname_C' AND state = 'streaming';"); + + # Setup logical replication streaming mode + + $node_B->safe_psql( + 'postgres', " + ALTER SUBSCRIPTION tap_sub_B + SET (streaming = $streaming_mode);"); + $node_C->safe_psql( + 'postgres', " + ALTER SUBSCRIPTION tap_sub_C + SET (streaming = $streaming_mode)"); + + # Wait for subscribers to finish initialization + + $node_A->poll_query_until( + 'postgres', " + SELECT pid != $oldpid_B FROM pg_stat_replication + WHERE application_name = '$appname_B' AND state = 'streaming';" + ) or die "Timed out while waiting for apply to restart"; + $node_B->poll_query_until( + 'postgres', " + SELECT pid != $oldpid_C FROM pg_stat_replication + WHERE application_name = '$appname_C' AND state = 'streaming';" + ) or die "Timed out while waiting for apply to restart"; + + ############################### + # Test 2PC PREPARE / COMMIT PREPARED. + # 1. Data is streamed as a 2PC transaction. + # 2. Then do commit prepared. + # + # Expect all data is replicated on subscriber(s) after the commit. + ############################### + + # Check that a background worker starts if "streaming" optparameterion is + # specified as "parallel". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($streaming_mode eq 'parallel') + { + $node_B->append_conf('postgresql.conf', "log_min_messages = debug1"); + $node_B->reload; + + $node_C->append_conf('postgresql.conf', "log_min_messages = debug1"); + $node_C->reload; + } + + # Insert, update and delete enough rows to exceed the 64kB limit. + # Then 2PC PREPARE + $node_A->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + if ($streaming_mode eq 'parallel') + { + $node_B->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_B->append_conf('postgresql.conf', "log_min_messages = warning"); + $node_B->reload; + + $node_C->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_C->append_conf('postgresql.conf', "log_min_messages = warning"); + $node_C->reload; + } + + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); + + # check the transaction state is prepared on subscriber(s) + my $result = + $node_B->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber B'); + $result = + $node_C->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber C'); + + # 2PC COMMIT + $node_A->safe_psql('postgres', "COMMIT PREPARED 'test_prepared_tab';"); + + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); + + # check that transaction was committed on subscriber(s) + $result = $node_B->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), + 'Rows inserted by 2PC have committed on subscriber B, and extra columns have local defaults' + ); + $result = $node_C->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), + 'Rows inserted by 2PC have committed on subscriber C, and extra columns have local defaults' + ); + + # check the transaction state is ended on subscriber(s) + $result = + $node_B->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is committed on subscriber B'); + $result = + $node_C->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is committed on subscriber C'); + + ############################### + # Test 2PC PREPARE with a nested ROLLBACK TO SAVEPOINT. + # 0. Cleanup from previous test leaving only 2 rows. + # 1. Insert one more row. + # 2. Record a SAVEPOINT. + # 3. Data is streamed using 2PC. + # 4. Do rollback to SAVEPOINT prior to the streamed inserts. + # 5. Then COMMIT PREPARED. + # + # Expect data after the SAVEPOINT is aborted leaving only 3 rows (= 2 original + 1 from step 1). + ############################### + + # First, delete the data except for 2 rows (delete will be replicated) + $node_A->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); + + # 2PC PREPARE with a nested ROLLBACK TO SAVEPOINT + $node_A->safe_psql( + 'postgres', " + BEGIN; + INSERT INTO test_tab VALUES (9999, 'foobar'); + SAVEPOINT sp_inner; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + ROLLBACK TO SAVEPOINT sp_inner; + PREPARE TRANSACTION 'outer'; + "); + + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); + + # check the transaction state prepared on subscriber(s) + $result = + $node_B->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber B'); + $result = + $node_C->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber C'); + + # 2PC COMMIT + $node_A->safe_psql('postgres', "COMMIT PREPARED 'outer';"); + + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); + + # check the transaction state is ended on subscriber + $result = + $node_B->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is ended on subscriber B'); + $result = + $node_C->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is ended on subscriber C'); + + # check inserts are visible at subscriber(s). + # All the streamed data (prior to the SAVEPOINT) should be rolled back. + # (9999, 'foobar') should be committed. + $result = $node_B->safe_psql('postgres', + "SELECT count(*) FROM test_tab where b = 'foobar';"); + is($result, qq(1), 'Rows committed are present on subscriber B'); + $result = + $node_B->safe_psql('postgres', "SELECT count(*) FROM test_tab;"); + is($result, qq(3), 'Rows committed are present on subscriber B'); + $result = $node_C->safe_psql('postgres', + "SELECT count(*) FROM test_tab where b = 'foobar';"); + is($result, qq(1), 'Rows committed are present on subscriber C'); + $result = + $node_C->safe_psql('postgres', "SELECT count(*) FROM test_tab;"); + is($result, qq(3), 'Rows committed are present on subscriber C'); + + # Cleanup the test data + $node_A->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); +} + ############################### # Setup a cascade of pub/sub nodes. # node_A -> node_B -> node_C @@ -260,160 +462,15 @@ is($result, qq(21), 'Rows committed are present on subscriber C'); # 2PC + STREAMING TESTS # --------------------- -my $oldpid_B = $node_A->safe_psql( - 'postgres', " - SELECT pid FROM pg_stat_replication - WHERE application_name = '$appname_B' AND state = 'streaming';"); -my $oldpid_C = $node_B->safe_psql( - 'postgres', " - SELECT pid FROM pg_stat_replication - WHERE application_name = '$appname_C' AND state = 'streaming';"); - -# Setup logical replication (streaming = on) - -$node_B->safe_psql( - 'postgres', " - ALTER SUBSCRIPTION tap_sub_B - SET (streaming = on);"); -$node_C->safe_psql( - 'postgres', " - ALTER SUBSCRIPTION tap_sub_C - SET (streaming = on)"); - -# Wait for subscribers to finish initialization - -$node_A->poll_query_until( - 'postgres', " - SELECT pid != $oldpid_B FROM pg_stat_replication - WHERE application_name = '$appname_B' AND state = 'streaming';" -) or die "Timed out while waiting for apply to restart"; -$node_B->poll_query_until( - 'postgres', " - SELECT pid != $oldpid_C FROM pg_stat_replication - WHERE application_name = '$appname_C' AND state = 'streaming';" -) or die "Timed out while waiting for apply to restart"; - -############################### -# Test 2PC PREPARE / COMMIT PREPARED. -# 1. Data is streamed as a 2PC transaction. -# 2. Then do commit prepared. -# -# Expect all data is replicated on subscriber(s) after the commit. -############################### - -# Insert, update and delete enough rows to exceed the 64kB limit. -# Then 2PC PREPARE -$node_A->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_A->wait_for_catchup($appname_B); -$node_B->wait_for_catchup($appname_C); +################################ +# Test using streaming mode 'on' +################################ +test_streaming($node_A, $node_B, $node_C, $appname_B, $appname_C, 'on'); -# check the transaction state is prepared on subscriber(s) -$result = - $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber B'); -$result = - $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber C'); - -# 2PC COMMIT -$node_A->safe_psql('postgres', "COMMIT PREPARED 'test_prepared_tab';"); - -$node_A->wait_for_catchup($appname_B); -$node_B->wait_for_catchup($appname_C); - -# check that transaction was committed on subscriber(s) -$result = $node_B->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), - 'Rows inserted by 2PC have committed on subscriber B, and extra columns have local defaults' -); -$result = $node_C->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), - 'Rows inserted by 2PC have committed on subscriber C, and extra columns have local defaults' -); - -# check the transaction state is ended on subscriber(s) -$result = - $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is committed on subscriber B'); -$result = - $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is committed on subscriber C'); - -############################### -# Test 2PC PREPARE with a nested ROLLBACK TO SAVEPOINT. -# 0. Cleanup from previous test leaving only 2 rows. -# 1. Insert one more row. -# 2. Record a SAVEPOINT. -# 3. Data is streamed using 2PC. -# 4. Do rollback to SAVEPOINT prior to the streamed inserts. -# 5. Then COMMIT PREPARED. -# -# Expect data after the SAVEPOINT is aborted leaving only 3 rows (= 2 original + 1 from step 1). -############################### - -# First, delete the data except for 2 rows (delete will be replicated) -$node_A->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); - -# 2PC PREPARE with a nested ROLLBACK TO SAVEPOINT -$node_A->safe_psql( - 'postgres', " - BEGIN; - INSERT INTO test_tab VALUES (9999, 'foobar'); - SAVEPOINT sp_inner; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - ROLLBACK TO SAVEPOINT sp_inner; - PREPARE TRANSACTION 'outer'; - "); - -$node_A->wait_for_catchup($appname_B); -$node_B->wait_for_catchup($appname_C); - -# check the transaction state prepared on subscriber(s) -$result = - $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber B'); -$result = - $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber C'); - -# 2PC COMMIT -$node_A->safe_psql('postgres', "COMMIT PREPARED 'outer';"); - -$node_A->wait_for_catchup($appname_B); -$node_B->wait_for_catchup($appname_C); - -# check the transaction state is ended on subscriber -$result = - $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is ended on subscriber B'); -$result = - $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is ended on subscriber C'); - -# check inserts are visible at subscriber(s). -# All the streamed data (prior to the SAVEPOINT) should be rolled back. -# (9999, 'foobar') should be committed. -$result = $node_B->safe_psql('postgres', - "SELECT count(*) FROM test_tab where b = 'foobar';"); -is($result, qq(1), 'Rows committed are present on subscriber B'); -$result = $node_B->safe_psql('postgres', "SELECT count(*) FROM test_tab;"); -is($result, qq(3), 'Rows committed are present on subscriber B'); -$result = $node_C->safe_psql('postgres', - "SELECT count(*) FROM test_tab where b = 'foobar';"); -is($result, qq(1), 'Rows committed are present on subscriber C'); -$result = $node_C->safe_psql('postgres', "SELECT count(*) FROM test_tab;"); -is($result, qq(3), 'Rows committed are present on subscriber C'); +###################################### +# Test using streaming mode 'parallel' +###################################### +test_streaming($node_A, $node_B, $node_C, $appname_B, $appname_C, 'parallel'); ############################### # check all the cleanup diff --git a/src/test/subscription/t/023_twophase_stream.pl b/src/test/subscription/t/023_twophase_stream.pl index d8475d25a4..b89414ab74 100644 --- a/src/test/subscription/t/023_twophase_stream.pl +++ b/src/test/subscription/t/023_twophase_stream.pl @@ -8,6 +8,266 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" +# parameter so the same code can be run both for the streaming=on and +# streaming=parallel cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_parallel) = @_; + + ############################### + # Test 2PC PREPARE / COMMIT PREPARED. + # 1. Data is streamed as a 2PC transaction. + # 2. Then do commit prepared. + # + # Expect all data is replicated on subscriber side after the commit. + ############################### + + # Check that a background worker starts if "streaming" parameter is + # specified as "parallel". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_parallel) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + # check that 2PC gets replicated to subscriber + # Insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + if ($is_parallel) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is in prepared state on subscriber + my $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber'); + + # 2PC transaction gets committed + $node_publisher->safe_psql('postgres', + "COMMIT PREPARED 'test_prepared_tab';"); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is committed on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), + 'Rows inserted by 2PC have committed on subscriber, and extra columns contain local defaults' + ); + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is committed on subscriber'); + + ############################### + # Test 2PC PREPARE / ROLLBACK PREPARED. + # 1. Table is deleted back to 2 rows which are replicated on subscriber. + # 2. Data is streamed using 2PC. + # 3. Do rollback prepared. + # + # Expect data rolls back leaving only the original 2 rows. + ############################### + + # First, delete the data except for 2 rows (will be replicated) + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE a > 2;"); + + # Then insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is in prepared state on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber'); + + # 2PC transaction gets aborted + $node_publisher->safe_psql('postgres', + "ROLLBACK PREPARED 'test_prepared_tab';"); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is aborted on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(2|2|2), + 'Rows inserted by 2PC are rolled back, leaving only the original 2 rows' + ); + + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is aborted on subscriber'); + + ############################### + # Check that 2PC COMMIT PREPARED is decoded properly on crash restart. + # 1. insert, update and delete enough rows to exceed the 64kB limit. + # 2. Then server crashes before the 2PC transaction is committed. + # 3. After servers are restarted the pending transaction is committed. + # + # Expect all data is replicated on subscriber side after the commit. + # Note: both publisher and subscriber do crash/restart. + ############################### + + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + $node_subscriber->stop('immediate'); + $node_publisher->stop('immediate'); + + $node_publisher->start; + $node_subscriber->start; + + # commit post the restart + $node_publisher->safe_psql('postgres', + "COMMIT PREPARED 'test_prepared_tab';"); + $node_publisher->wait_for_catchup($appname); + + # check inserts are visible + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), + 'Rows inserted by 2PC have committed on subscriber, and extra columns contain local defaults' + ); + + ############################### + # Do INSERT after the PREPARE but before ROLLBACK PREPARED. + # 1. Table is deleted back to 2 rows which are replicated on subscriber. + # 2. Data is streamed using 2PC. + # 3. A single row INSERT is done which is after the PREPARE. + # 4. Then do a ROLLBACK PREPARED. + # + # Expect the 2PC data rolls back leaving only 3 rows on the subscriber + # (the original 2 + inserted 1). + ############################### + + # First, delete the data except for 2 rows (will be replicated) + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE a > 2;"); + + # Then insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is in prepared state on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber'); + + # Insert a different record (now we are outside of the 2PC transaction) + # Note: the 2PC transaction still holds row locks so make sure this insert is for a separate primary key + $node_publisher->safe_psql('postgres', + "INSERT INTO test_tab VALUES (99999, 'foobar')"); + + # 2PC transaction gets aborted + $node_publisher->safe_psql('postgres', + "ROLLBACK PREPARED 'test_prepared_tab';"); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is aborted on subscriber, + # but the extra INSERT outside of the 2PC still was replicated + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3|3|3), + 'check the outside insert was copied to subscriber'); + + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is aborted on subscriber'); + + ############################### + # Do INSERT after the PREPARE but before COMMIT PREPARED. + # 1. Table is deleted back to 2 rows which are replicated on subscriber. + # 2. Data is streamed using 2PC. + # 3. A single row INSERT is done which is after the PREPARE. + # 4. Then do a COMMIT PREPARED. + # + # Expect 2PC data + the extra row are on the subscriber + # (the 3334 + inserted 1 = 3335). + ############################### + + # First, delete the data except for 2 rows (will be replicated) + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE a > 2;"); + + # Then insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is in prepared state on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber'); + + # Insert a different record (now we are outside of the 2PC transaction) + # Note: the 2PC transaction still holds row locks so make sure this insert is for a separare primary key + $node_publisher->safe_psql('postgres', + "INSERT INTO test_tab VALUES (99999, 'foobar')"); + + # 2PC transaction gets committed + $node_publisher->safe_psql('postgres', + "COMMIT PREPARED 'test_prepared_tab';"); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is committed on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3335|3335|3335), + 'Rows inserted by 2PC (as well as outside insert) have committed on subscriber, and extra columns contain local defaults' + ); + + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is committed on subscriber'); + + # Cleanup the test data + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE a > 2;"); + $node_publisher->wait_for_catchup($appname); +} + ############################### # Setup ############################### @@ -48,6 +308,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql( 'postgres', " CREATE SUBSCRIPTION tap_sub @@ -70,236 +334,30 @@ my $twophase_query = $node_subscriber->poll_query_until('postgres', $twophase_query) or die "Timed out while waiting for subscriber to enable twophase"; -############################### # Check initial data was copied to subscriber -############################### my $result = $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab"); is($result, qq(2|2|2), 'check initial data was copied to subscriber'); -############################### -# Test 2PC PREPARE / COMMIT PREPARED. -# 1. Data is streamed as a 2PC transaction. -# 2. Then do commit prepared. -# -# Expect all data is replicated on subscriber side after the commit. -############################### - -# check that 2PC gets replicated to subscriber -# Insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is in prepared state on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber'); - -# 2PC transaction gets committed -$node_publisher->safe_psql('postgres', - "COMMIT PREPARED 'test_prepared_tab';"); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is committed on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), - 'Rows inserted by 2PC have committed on subscriber, and extra columns contain local defaults' -); -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is committed on subscriber'); - -############################### -# Test 2PC PREPARE / ROLLBACK PREPARED. -# 1. Table is deleted back to 2 rows which are replicated on subscriber. -# 2. Data is streamed using 2PC. -# 3. Do rollback prepared. -# -# Expect data rolls back leaving only the original 2 rows. -############################### - -# First, delete the data except for 2 rows (will be replicated) -$node_publisher->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); - -# Then insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is in prepared state on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber'); - -# 2PC transaction gets aborted -$node_publisher->safe_psql('postgres', - "ROLLBACK PREPARED 'test_prepared_tab';"); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is aborted on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(2|2|2), - 'Rows inserted by 2PC are rolled back, leaving only the original 2 rows'); - -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is aborted on subscriber'); - -############################### -# Check that 2PC COMMIT PREPARED is decoded properly on crash restart. -# 1. insert, update and delete enough rows to exceed the 64kB limit. -# 2. Then server crashes before the 2PC transaction is committed. -# 3. After servers are restarted the pending transaction is committed. -# -# Expect all data is replicated on subscriber side after the commit. -# Note: both publisher and subscriber do crash/restart. -############################### - -$node_publisher->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_subscriber->stop('immediate'); -$node_publisher->stop('immediate'); - -$node_publisher->start; -$node_subscriber->start; - -# commit post the restart -$node_publisher->safe_psql('postgres', - "COMMIT PREPARED 'test_prepared_tab';"); -$node_publisher->wait_for_catchup($appname); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -# check inserts are visible -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), - 'Rows inserted by 2PC have committed on subscriber, and extra columns contain local defaults' +###################################### +# Test using streaming mode 'parallel' +###################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" ); -############################### -# Do INSERT after the PREPARE but before ROLLBACK PREPARED. -# 1. Table is deleted back to 2 rows which are replicated on subscriber. -# 2. Data is streamed using 2PC. -# 3. A single row INSERT is done which is after the PREPARE. -# 4. Then do a ROLLBACK PREPARED. -# -# Expect the 2PC data rolls back leaving only 3 rows on the subscriber -# (the original 2 + inserted 1). -############################### - -# First, delete the data except for 2 rows (will be replicated) -$node_publisher->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); - -# Then insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is in prepared state on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber'); - -# Insert a different record (now we are outside of the 2PC transaction) -# Note: the 2PC transaction still holds row locks so make sure this insert is for a separate primary key -$node_publisher->safe_psql('postgres', - "INSERT INTO test_tab VALUES (99999, 'foobar')"); - -# 2PC transaction gets aborted -$node_publisher->safe_psql('postgres', - "ROLLBACK PREPARED 'test_prepared_tab';"); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is aborted on subscriber, -# but the extra INSERT outside of the 2PC still was replicated -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3|3|3), 'check the outside insert was copied to subscriber'); - -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is aborted on subscriber'); - -############################### -# Do INSERT after the PREPARE but before COMMIT PREPARED. -# 1. Table is deleted back to 2 rows which are replicated on subscriber. -# 2. Data is streamed using 2PC. -# 3. A single row INSERT is done which is after the PREPARE. -# 4. Then do a COMMIT PREPARED. -# -# Expect 2PC data + the extra row are on the subscriber -# (the 3334 + inserted 1 = 3335). -############################### - -# First, delete the data except for 2 rows (will be replicated) -$node_publisher->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); - -# Then insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is in prepared state on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber'); - -# Insert a different record (now we are outside of the 2PC transaction) -# Note: the 2PC transaction still holds row locks so make sure this insert is for a separare primary key -$node_publisher->safe_psql('postgres', - "INSERT INTO test_tab VALUES (99999, 'foobar')"); - -# 2PC transaction gets committed -$node_publisher->safe_psql('postgres', - "COMMIT PREPARED 'test_prepared_tab';"); - -$node_publisher->wait_for_catchup($appname); +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = parallel)"); -# check that transaction is committed on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3335|3335|3335), - 'Rows inserted by 2PC (as well as outside insert) have committed on subscriber, and extra columns contain local defaults' -); +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is committed on subscriber'); +test_streaming($node_publisher, $node_subscriber, $appname, 1); ############################### # check all the cleanup -- 2.23.0.windows.1 [application/octet-stream] v15-0003-Add-some-checks-before-using-apply-background-wo.patch (36.6K, ../../OS3PR01MB62755C6C9A75EB09F7218B589E839@OS3PR01MB6275.jpnprd01.prod.outlook.com/4-v15-0003-Add-some-checks-before-using-apply-background-wo.patch) download | inline diff: From 062b9c2151da30b04e3e9f26f6adde82a3462f3c Mon Sep 17 00:00:00 2001 From: wangw <[email protected]> Date: Tue, 14 Jun 2022 11:23:52 +0800 Subject: [PATCH v15 3/4] Add some checks before using apply background worker to apply changes. streaming=parallel mode has two requirements: 1) The unique column in the relation on the subscriber-side should also be the unique column on the publisher-side; 2) There cannot be any non-immutable functions in the subscriber-side replicated table. Look for functions in the following places: * a. Trigger functions * b. Column default value expressions and domain constraints * c. Constraint expressions * d. Foreign keys --- doc/src/sgml/ref/create_subscription.sgml | 4 + .../replication/logical/applybgworker.c | 42 ++ src/backend/replication/logical/proto.c | 66 ++- src/backend/replication/logical/relation.c | 199 +++++++++ src/backend/replication/logical/tablesync.c | 1 + src/backend/replication/logical/worker.c | 23 +- src/backend/utils/cache/typcache.c | 17 + src/include/replication/logicalproto.h | 1 + src/include/replication/logicalrelation.h | 15 + src/include/replication/worker_internal.h | 1 + src/include/utils/typcache.h | 2 + .../subscription/t/022_twophase_cascade.pl | 6 + .../subscription/t/032_streaming_apply.pl | 391 ++++++++++++++++++ src/tools/pgindent/typedefs.list | 1 + 14 files changed, 759 insertions(+), 10 deletions(-) create mode 100644 src/test/subscription/t/032_streaming_apply.pl diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml index 71dd4aca81..270e3d382e 100644 --- a/doc/src/sgml/ref/create_subscription.sgml +++ b/doc/src/sgml/ref/create_subscription.sgml @@ -240,6 +240,10 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl transaction is committed. Note that if an error happens when applying changes in a background worker, the finish LSN of the remote transaction might not be reported in the server log. + Parallel mode has two requirements: 1) the unique column in the + relation on the subscriber-side should also be the unique column on + the publisher-side; 2) there cannot be any non-immutable functions + in the subscriber-side replicated table. </para> </listitem> </varlistentry> diff --git a/src/backend/replication/logical/applybgworker.c b/src/backend/replication/logical/applybgworker.c index dfa49be98c..92b4b4e4ed 100644 --- a/src/backend/replication/logical/applybgworker.c +++ b/src/backend/replication/logical/applybgworker.c @@ -773,3 +773,45 @@ apply_bgworker_subxact_info_add(TransactionId current_xid) MemoryContextSwitchTo(oldctx); } } + +/* + * Check if changes on this relation can be applied by an apply background + * worker. + * + * Although the commit order is maintained only allowing one process to commit + * at a time, the access order to the relation has changed. This could cause + * unexpected problems if the unique column on the replicated table is + * inconsistent with the publisher-side or contains non-immutable functions + * when applying transactions in the apply background worker. + */ +void +apply_bgworker_relation_check(LogicalRepRelMapEntry *rel) +{ + /* Skip check if not an apply background worker. */ + if (!am_apply_bgworker()) + return; + + /* + * Partition table checks are done later in function + * apply_handle_tuple_routing. + */ + if (rel->localrel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + return; + + /* + * Return if changes on this relation can be applied by an apply background + * worker. + */ + if (rel->parallel == PARALLEL_SAFE) + return; + + /* We are in error mode and should give user correct error. */ + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot replicate target relation \"%s.%s\" in parallel " + "mode", rel->remoterel.nspname, rel->remoterel.relname), + errdetail("The unique column on subscriber is not the unique " + "column on publisher or there is at least one " + "non-immutable function."), + errhint("Please change the streaming option to 'on' instead of 'parallel'."))); +} diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c index affd08cfa4..12a9799f5c 100644 --- a/src/backend/replication/logical/proto.c +++ b/src/backend/replication/logical/proto.c @@ -23,7 +23,8 @@ /* * Protocol message flags. */ -#define LOGICALREP_IS_REPLICA_IDENTITY 1 +#define ATTR_IS_REPLICA_IDENTITY (1 << 0) +#define ATTR_IS_UNIQUE (1 << 1) #define MESSAGE_TRANSACTIONAL (1<<0) #define TRUNCATE_CASCADE (1<<0) @@ -933,11 +934,55 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns) TupleDesc desc; int i; uint16 nliveatts = 0; - Bitmapset *idattrs = NULL; + Bitmapset *idattrs = NULL, + *attunique = NULL; bool replidentfull; desc = RelationGetDescr(rel); + if (rel->rd_rel->relhasindex) + { + List *indexoidlist = RelationGetIndexList(rel); + ListCell *indexoidscan; + + foreach(indexoidscan, indexoidlist) + { + Oid indexoid = lfirst_oid(indexoidscan); + Relation indexRel; + + /* Look up the description for index */ + indexRel = RelationIdGetRelation(indexoid); + + if (!RelationIsValid(indexRel)) + elog(ERROR, "could not open relation with OID %u", indexoid); + + if (indexRel->rd_index->indisunique) + { + int i; + + /* Add referenced attributes to idindexattrs */ + for (i = 0; i < indexRel->rd_index->indnatts; i++) + { + int attrnum = indexRel->rd_index->indkey.values[i]; + + /* + * We don't include non-key columns into idindexattrs + * bitmaps. See RelationGetIndexAttrBitmap. + */ + if (attrnum != 0) + { + if (i < indexRel->rd_index->indnkeyatts && + !bms_is_member(attrnum - FirstLowInvalidHeapAttributeNumber, attunique)) + attunique = bms_add_member(attunique, + attrnum - FirstLowInvalidHeapAttributeNumber); + } + } + } + RelationClose(indexRel); + } + list_free(indexoidlist); + } + /* send number of live attributes */ for (i = 0; i < desc->natts; i++) { @@ -974,7 +1019,11 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns) if (replidentfull || bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber, idattrs)) - flags |= LOGICALREP_IS_REPLICA_IDENTITY; + flags |= ATTR_IS_REPLICA_IDENTITY; + + if (bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber, + attunique)) + flags |= ATTR_IS_UNIQUE; pq_sendbyte(out, flags); @@ -1001,7 +1050,8 @@ logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel) int natts; char **attnames; Oid *atttyps; - Bitmapset *attkeys = NULL; + Bitmapset *attkeys = NULL, + *attunique = NULL; natts = pq_getmsgint(in, 2); attnames = palloc(natts * sizeof(char *)); @@ -1012,11 +1062,14 @@ logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel) { uint8 flags; - /* Check for replica identity column */ + /* Check for replica identity and unique column */ flags = pq_getmsgbyte(in); - if (flags & LOGICALREP_IS_REPLICA_IDENTITY) + if (flags & ATTR_IS_REPLICA_IDENTITY) attkeys = bms_add_member(attkeys, i); + if (flags & ATTR_IS_UNIQUE) + attunique = bms_add_member(attunique, i); + /* attribute name */ attnames[i] = pstrdup(pq_getmsgstring(in)); @@ -1030,6 +1083,7 @@ logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel) rel->attnames = attnames; rel->atttyps = atttyps; rel->attkeys = attkeys; + rel->attunique = attunique; rel->natts = natts; } diff --git a/src/backend/replication/logical/relation.c b/src/backend/replication/logical/relation.c index e989047681..072de1557c 100644 --- a/src/backend/replication/logical/relation.c +++ b/src/backend/replication/logical/relation.c @@ -19,12 +19,19 @@ #include "access/table.h" #include "catalog/namespace.h" +#include "catalog/pg_proc.h" #include "catalog/pg_subscription_rel.h" +#include "commands/trigger.h" #include "executor/executor.h" #include "nodes/makefuncs.h" +#include "optimizer/optimizer.h" #include "replication/logicalrelation.h" #include "replication/worker_internal.h" +#include "rewrite/rewriteHandler.h" #include "utils/inval.h" +#include "utils/lsyscache.h" +#include "utils/syscache.h" +#include "utils/typcache.h" static MemoryContext LogicalRepRelMapContext = NULL; @@ -91,6 +98,26 @@ logicalrep_relmap_invalidate_cb(Datum arg, Oid reloid) } } +/* + * Relcache invalidation callback to reset parallel flag. + */ +static void +logicalrep_relmap_reset_parallel_cb(Datum arg, int cacheid, uint32 hashvalue) +{ + HASH_SEQ_STATUS hash_seq; + LogicalRepRelMapEntry *entry; + + if (LogicalRepRelMap == NULL) + return; + + hash_seq_init(&hash_seq, LogicalRepRelMap); + while ((entry = hash_seq_search(&hash_seq)) != NULL) + { + entry->parallel = PARALLEL_UNKNOWN; + entry->localrelvalid = false; + } +} + /* * Initialize the relation map cache. */ @@ -116,6 +143,9 @@ logicalrep_relmap_init(void) /* Watch for invalidation events. */ CacheRegisterRelcacheCallback(logicalrep_relmap_invalidate_cb, (Datum) 0); + CacheRegisterSyscacheCallback(PROCOID, + logicalrep_relmap_reset_parallel_cb, + (Datum) 0); } /* @@ -142,6 +172,7 @@ logicalrep_relmap_free_entry(LogicalRepRelMapEntry *entry) pfree(remoterel->atttyps); } bms_free(remoterel->attkeys); + bms_free(remoterel->attunique); if (entry->attrmap) free_attrmap(entry->attrmap); @@ -190,6 +221,7 @@ logicalrep_relmap_update(LogicalRepRelation *remoterel) } entry->remoterel.replident = remoterel->replident; entry->remoterel.attkeys = bms_copy(remoterel->attkeys); + entry->remoterel.attunique = bms_copy(remoterel->attunique); MemoryContextSwitchTo(oldctx); } @@ -310,6 +342,166 @@ logicalrep_rel_mark_updatable(LogicalRepRelMapEntry *entry) } } +/* + * Check if changes on one relation can be applied by an apply background + * worker and assign the 'parallel' flag. + * + * There are two requirements for applying changes in an apply background + * worker: 1) The unique column in the relation on the subscriber-side should + * also be the unique column on the publisher-side; 2) There cannot be any + * non-immutable functions in the subscriber-side. + * + * We just mark the relation entry as 'PARALLEL_RESTRICTED' here if changes on + * one relation can not be applied by an apply background worker and leave it + * to apply_bgworker_relation_check() to throw the actual error if needed. + */ +static void +logicalrep_rel_mark_parallel(LogicalRepRelMapEntry *entry) +{ + Bitmapset *ukey; + int i; + TupleDesc tupdesc; + int attnum; + List *fkeys = NIL; + + /* Fast path if we marked 'parallel' flag. */ + if (entry->parallel != PARALLEL_UNKNOWN) + return; + + /* Initialize the flag. */ + entry->parallel = PARALLEL_SAFE; + + /* + * First, we check if the unique column in the relation on the + * subscriber-side is also the unique column on the publisher-side. + */ + ukey = RelationGetIndexAttrBitmap(entry->localrel, + INDEX_ATTR_BITMAP_KEY); + + if (ukey) + { + i = -1; + while ((i = bms_next_member(ukey, i)) >= 0) + { + attnum = AttrNumberGetAttrOffset(i + FirstLowInvalidHeapAttributeNumber); + + if (entry->attrmap->attnums[attnum] < 0 || + !bms_is_member(entry->attrmap->attnums[attnum], entry->remoterel.attunique)) + { + entry->parallel = PARALLEL_RESTRICTED; + return; + } + } + } + + /* + * Then, We check if there is any non-immutable function in the local + * table. Look for functions in the following places: + * a. trigger functions; + * b. Column default value expressions and domain constraints; + * c. Constraint expressions; + * d. Foreign keys. + */ + /* Check the trigger functions. */ + if (entry->localrel->trigdesc != NULL) + { + for (i = 0; i < entry->localrel->trigdesc->numtriggers; i++) + { + Trigger *trig = entry->localrel->trigdesc->triggers + i; + + if (trig->tgenabled != TRIGGER_FIRES_ALWAYS && + trig->tgenabled != TRIGGER_FIRES_ON_REPLICA) + continue; + + if (func_volatile(trig->tgfoid) != PROVOLATILE_IMMUTABLE) + { + entry->parallel = PARALLEL_RESTRICTED; + return; + } + } + } + + /* Check the columns. */ + tupdesc = RelationGetDescr(entry->localrel); + for (attnum = 0; attnum < tupdesc->natts; attnum++) + { + Form_pg_attribute att = TupleDescAttr(tupdesc, attnum); + + /* We don't need info for dropped or generated attributes */ + if (att->attisdropped || att->attgenerated) + continue; + + /* + * We don't need to check columns that only exist on the + * subscriber + */ + if (entry->attrmap->attnums[attnum] < 0) + continue; + + if (att->atthasdef) + { + Node *defaultexpr; + + defaultexpr = build_column_default(entry->localrel, attnum + 1); + if (contain_mutable_functions(defaultexpr)) + { + entry->parallel = PARALLEL_RESTRICTED; + return; + } + } + + /* + * If the column is of a DOMAIN type, determine whether + * that domain has any CHECK expressions that are not + * immutable. + */ + if (get_typtype(att->atttypid) == TYPTYPE_DOMAIN) + { + List *domain_constraints; + ListCell *lc; + + domain_constraints = GetDomainConstraints(att->atttypid); + + foreach(lc, domain_constraints) + { + DomainConstraintState *con = (DomainConstraintState *) lfirst(lc); + + if (con->check_expr && contain_mutable_functions((Node *) con->check_expr)) + { + entry->parallel = PARALLEL_RESTRICTED; + return; + } + } + } + } + + /* Check the constraints. */ + if (tupdesc->constr) + { + ConstrCheck *check = tupdesc->constr->check; + + /* + * Determine if there are any CHECK constraints which + * contains non-immutable function. + */ + for (i = 0; i < tupdesc->constr->num_check; i++) + { + Expr *check_expr = stringToNode(check[i].ccbin); + + if (contain_mutable_functions((Node *) check_expr)) + { + entry->parallel = PARALLEL_RESTRICTED; + return; + } + } + } + + /* Check the foreign keys. */ + fkeys = RelationGetFKeyList(entry->localrel); + if (fkeys) + entry->parallel = PARALLEL_RESTRICTED; +} + /* * Open the local relation associated with the remote one. * @@ -438,6 +630,9 @@ logicalrep_rel_open(LogicalRepRelId remoteid, LOCKMODE lockmode) */ logicalrep_rel_mark_updatable(entry); + /* Set if changes could be applied in the apply background worker. */ + logicalrep_rel_mark_parallel(entry); + entry->localrelvalid = true; } @@ -653,6 +848,7 @@ logicalrep_partition_open(LogicalRepRelMapEntry *root, } entry->remoterel.replident = remoterel->replident; entry->remoterel.attkeys = bms_copy(remoterel->attkeys); + entry->remoterel.attunique = bms_copy(remoterel->attunique); } entry->localrel = partrel; @@ -696,6 +892,9 @@ logicalrep_partition_open(LogicalRepRelMapEntry *root, /* Set if the table's replica identity is enough to apply update/delete. */ logicalrep_rel_mark_updatable(entry); + /* Set if changes could be applied in the apply background worker. */ + logicalrep_rel_mark_parallel(entry); + entry->localrelvalid = true; /* state and statelsn are left set to 0. */ diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c index 8ffba7e2e5..3cdbf8b457 100644 --- a/src/backend/replication/logical/tablesync.c +++ b/src/backend/replication/logical/tablesync.c @@ -884,6 +884,7 @@ fetch_remote_table_info(char *nspname, char *relname, lrel->attnames = palloc0(MaxTupleAttributeNumber * sizeof(char *)); lrel->atttyps = palloc0(MaxTupleAttributeNumber * sizeof(Oid)); lrel->attkeys = NULL; + lrel->attunique = NULL; /* * Store the columns as a list of names. Ignore those that are not diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c index 9da11dd41c..81e0005d12 100644 --- a/src/backend/replication/logical/worker.c +++ b/src/backend/replication/logical/worker.c @@ -1388,6 +1388,14 @@ apply_handle_stream_stop(StringInfo s) { char action = LOGICAL_REP_MSG_STREAM_STOP; + /* + * Unlike stream_commit, we don't need to wait here for stream_stop to + * finish. Allowing the other transaction to be applied before + * stream_stop is finished can lead to failures if the unique + * index/constraint is different between publisher and subscriber. But + * for such cases, we don't allow streamed transactions to be applied + * in parallel. See apply_bgworker_relation_check. + */ apply_bgworker_send_data(stream_apply_worker, 1, &action); elog(DEBUG1, "stopped streaming of xid %u, %u changes streamed", stream_xid, nchanges); @@ -2040,6 +2048,8 @@ apply_handle_insert(StringInfo s) /* Set relation for error callback */ apply_error_callback_arg.rel = rel; + apply_bgworker_relation_check(rel); + /* Initialize the executor state. */ edata = create_edata_for_relation(rel); estate = edata->estate; @@ -2183,6 +2193,8 @@ apply_handle_update(StringInfo s) /* Check if we can do the update. */ check_relation_updatable(rel); + apply_bgworker_relation_check(rel); + /* Initialize the executor state. */ edata = create_edata_for_relation(rel); estate = edata->estate; @@ -2351,6 +2363,8 @@ apply_handle_delete(StringInfo s) /* Check if we can do the delete. */ check_relation_updatable(rel); + apply_bgworker_relation_check(rel); + /* Initialize the executor state. */ edata = create_edata_for_relation(rel); estate = edata->estate; @@ -2536,13 +2550,14 @@ apply_handle_tuple_routing(ApplyExecutionData *edata, } MemoryContextSwitchTo(oldctx); + part_entry = logicalrep_partition_open(relmapentry, partrel, + attrmap); + /* Check if we can do the update or delete on the leaf partition. */ if (operation == CMD_UPDATE || operation == CMD_DELETE) - { - part_entry = logicalrep_partition_open(relmapentry, partrel, - attrmap); check_relation_updatable(part_entry); - } + + apply_bgworker_relation_check(part_entry); switch (operation) { diff --git a/src/backend/utils/cache/typcache.c b/src/backend/utils/cache/typcache.c index 808f9ebd0d..b248899d82 100644 --- a/src/backend/utils/cache/typcache.c +++ b/src/backend/utils/cache/typcache.c @@ -2540,6 +2540,23 @@ compare_values_of_enum(TypeCacheEntry *tcache, Oid arg1, Oid arg2) return 0; } +/* + * GetDomainConstraints --- get DomainConstraintState list of specified domain type + */ +List * +GetDomainConstraints(Oid type_id) +{ + TypeCacheEntry *typentry; + List *constraints = NIL; + + typentry = lookup_type_cache(type_id, TYPECACHE_DOMAIN_CONSTR_INFO); + + if(typentry->domainData != NULL) + constraints = typentry->domainData->constraints; + + return constraints; +} + /* * Load (or re-load) the enumData member of the typcache entry. */ diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h index 0f74a3392b..130c4eb82f 100644 --- a/src/include/replication/logicalproto.h +++ b/src/include/replication/logicalproto.h @@ -113,6 +113,7 @@ typedef struct LogicalRepRelation char replident; /* replica identity */ char relkind; /* remote relation kind */ Bitmapset *attkeys; /* Bitmap of key columns */ + Bitmapset *attunique; /* Bitmap of unique columns */ } LogicalRepRelation; /* Type mapping info */ diff --git a/src/include/replication/logicalrelation.h b/src/include/replication/logicalrelation.h index 78cd7e77f5..015545cd8d 100644 --- a/src/include/replication/logicalrelation.h +++ b/src/include/replication/logicalrelation.h @@ -15,6 +15,19 @@ #include "access/attmap.h" #include "replication/logicalproto.h" +/* + * States to determine if changes on one relation can be applied by an apply + * background worker. + */ +typedef enum RelParallel +{ + PARALLEL_UNKNOWN = 0, /* unknown */ + PARALLEL_SAFE, /* Can apply changes in an apply background + worker */ + PARALLEL_RESTRICTED /* Can not apply changes in an apply background + worker */ +} RelParallel; + typedef struct LogicalRepRelMapEntry { LogicalRepRelation remoterel; /* key is remoterel.remoteid */ @@ -31,6 +44,8 @@ typedef struct LogicalRepRelMapEntry Relation localrel; /* relcache entry (NULL when closed) */ AttrMap *attrmap; /* map of local attributes to remote ones */ bool updatable; /* Can apply updates/deletes? */ + RelParallel parallel; /* Can apply changes in an apply + background worker? */ /* Sync state. */ char state; diff --git a/src/include/replication/worker_internal.h b/src/include/replication/worker_internal.h index 5be8f5755e..b28f4ab977 100644 --- a/src/include/replication/worker_internal.h +++ b/src/include/replication/worker_internal.h @@ -194,6 +194,7 @@ extern void apply_bgworker_free(ApplyBgworkerState *wstate); extern void apply_bgworker_check_status(void); extern void apply_bgworker_set_status(ApplyBgworkerStatus status); extern void apply_bgworker_subxact_info_add(TransactionId current_xid); +extern void apply_bgworker_relation_check(LogicalRepRelMapEntry *rel); static inline bool am_tablesync_worker(void) diff --git a/src/include/utils/typcache.h b/src/include/utils/typcache.h index 431ad7f1b3..ed7c2e7f48 100644 --- a/src/include/utils/typcache.h +++ b/src/include/utils/typcache.h @@ -199,6 +199,8 @@ extern uint64 assign_record_type_identifier(Oid type_id, int32 typmod); extern int compare_values_of_enum(TypeCacheEntry *tcache, Oid arg1, Oid arg2); +extern List *GetDomainConstraints(Oid type_id); + extern size_t SharedRecordTypmodRegistryEstimate(void); extern void SharedRecordTypmodRegistryInit(SharedRecordTypmodRegistry *, diff --git a/src/test/subscription/t/022_twophase_cascade.pl b/src/test/subscription/t/022_twophase_cascade.pl index 0a4152d3be..30a01f7305 100644 --- a/src/test/subscription/t/022_twophase_cascade.pl +++ b/src/test/subscription/t/022_twophase_cascade.pl @@ -39,6 +39,12 @@ sub test_streaming ALTER SUBSCRIPTION tap_sub_C SET (streaming = $streaming_mode)"); + if ($streaming_mode eq 'parallel') + { + $node_C->safe_psql( + 'postgres', "ALTER TABLE test_tab ALTER c DROP DEFAULT"); + } + # Wait for subscribers to finish initialization $node_A->poll_query_until( diff --git a/src/test/subscription/t/032_streaming_apply.pl b/src/test/subscription/t/032_streaming_apply.pl new file mode 100644 index 0000000000..eca4328676 --- /dev/null +++ b/src/test/subscription/t/032_streaming_apply.pl @@ -0,0 +1,391 @@ +# Copyright (c) 2022, PostgreSQL Global Development Group + +# Test the restrictions of streaming mode "parallel" in logical replication + +use strict; +use warnings; +use PostgreSQL::Test::Cluster; +use PostgreSQL::Test::Utils; +use Test::More; + +my $offset = 0; + +# Create publisher node +my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); +$node_publisher->init(allows_streaming => 'logical'); +$node_publisher->append_conf('postgresql.conf', + 'logical_decoding_work_mem = 64kB'); +$node_publisher->start; + +# Create subscriber node +my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber'); +$node_subscriber->init; +$node_subscriber->start; + +# Setup structure on publisher +$node_publisher->safe_psql('postgres', + "CREATE TABLE test_tab (a int primary key, b varchar)"); +$node_publisher->safe_psql('postgres', + "CREATE TABLE test_tab_partitioned (a int primary key, b varchar)"); + +# Setup structure on subscriber +# We need to test normal table and partition table. +$node_subscriber->safe_psql('postgres', + "CREATE TABLE test_tab (a int primary key, b varchar)"); +$node_subscriber->safe_psql('postgres', + "CREATE TABLE test_tab_partitioned (a int primary key, b varchar) PARTITION BY RANGE(a)"); +$node_subscriber->safe_psql('postgres', + "CREATE TABLE test_tab_partition (LIKE test_tab_partitioned)"); +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partitioned ATTACH PARTITION test_tab_partition DEFAULT" +); + +# Setup logical replication +my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres'; +$node_publisher->safe_psql('postgres', + "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); +$node_publisher->safe_psql('postgres', + "CREATE PUBLICATION tap_pub_partitioned FOR TABLE test_tab_partitioned"); + +my $appname = 'tap_sub'; +$node_subscriber->safe_psql( + 'postgres', " + CREATE SUBSCRIPTION tap_sub + CONNECTION '$publisher_connstr application_name=$appname' + PUBLICATION tap_pub, tap_pub_partitioned + WITH (streaming = parallel, copy_data = false)"); + +$node_publisher->wait_for_catchup($appname); + +# It is not allowed that the unique index on the publisher and the subscriber +# is different. Check the error reported by background worker in this case. +# First we check the unique index on normal table. +$node_subscriber->safe_psql('postgres', + "CREATE UNIQUE INDEX test_tab_b_idx ON test_tab (b)"); + +# Check that a background worker starts if "streaming" option is specified as +# "parallel". We have to look for the DEBUG1 log messages about that, so +# temporarily bump up the log verbosity. +$node_subscriber->append_conf('postgresql.conf', "log_min_messages = debug1"); +$node_subscriber->reload; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); +$node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); +$node_subscriber->reload; + +$node_subscriber->wait_for_log( + qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab" in parallel mode/, + $offset); + +# Drop the unique index on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', "DROP INDEX test_tab_b_idx"); + +$node_publisher->wait_for_catchup($appname); + +my $result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); +is($result, qq(5000), 'data replicated to subscriber after dropping index'); + +# Then we check the unique index on partition table. +$node_subscriber->safe_psql('postgres', + "CREATE UNIQUE INDEX test_tab_b_partition_idx ON test_tab_partition (b)"); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab_partitioned SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log( + qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab_partitioned" in parallel mode/, + $offset); + +# Drop the unique index on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', "DROP INDEX test_tab_b_partition_idx"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); +is($result, qq(5000), 'data replicated to subscriber after dropping index'); + +# Triggers which execute non-immutable function are not allowed on the +# subscriber side. Check the error reported by background worker in this case. +# First we check the trigger function on normal table. +$node_subscriber->safe_psql( + 'postgres', qq{ +CREATE FUNCTION trigger_func() RETURNS TRIGGER AS \$\$ + BEGIN + RETURN NULL; + END +\$\$ language plpgsql; +CREATE TRIGGER insert_trig +BEFORE INSERT ON test_tab +FOR EACH ROW EXECUTE PROCEDURE trigger_func(); +ALTER TABLE test_tab ENABLE REPLICA TRIGGER insert_trig; +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', "DELETE FROM test_tab"); + +$node_subscriber->wait_for_log( + qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab" in parallel mode/, + $offset); + +# Drop the trigger on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "DROP TRIGGER insert_trig ON test_tab"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); +is($result, qq(0), 'data replicated to subscriber after dropping trigger'); + +# Then we check the trigger function on partition table. +$node_subscriber->safe_psql( + 'postgres', qq{ +CREATE TRIGGER insert_trig +BEFORE INSERT ON test_tab_partition +FOR EACH ROW EXECUTE PROCEDURE trigger_func(); +ALTER TABLE test_tab_partition ENABLE REPLICA TRIGGER insert_trig; +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', "DELETE FROM test_tab_partitioned"); + +$node_subscriber->wait_for_log( + qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab_partitioned" in parallel mode/, + $offset); + +# Drop the trigger on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "DROP TRIGGER insert_trig ON test_tab_partition"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); +is($result, qq(0), 'data replicated to subscriber after dropping trigger'); + +# It is not allowed that column default value expression contains a +# non-immutable function on the subscriber side. Check the error reported by +# background worker in this case. +# First we check the column default value expression on normal table. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab ALTER COLUMN b SET DEFAULT random()"); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log( + qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab" in parallel mode/, + $offset); + +# Drop default value on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab ALTER COLUMN b DROP DEFAULT"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); +is($result, qq(5000), + 'data replicated to subscriber after dropping default value expression'); + +# Then we check the column default value expression on partition table. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partition ALTER COLUMN b SET DEFAULT random()"); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab_partitioned SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log( + qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab_partitioned" in parallel mode/, + $offset); + +# Drop default value on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partition ALTER COLUMN b DROP DEFAULT"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); +is($result, qq(5000), + 'data replicated to subscriber after dropping default value expression'); + +# It is not allowed that domain constraint expression contains a non-immutable +# function on the subscriber side. Check the error reported by background +# worker in this case. +# Because the column type of the partition table must be the same as its parent +# table, only test normal table here. +$node_subscriber->safe_psql( + 'postgres', qq{ +CREATE DOMAIN test_domain AS int CHECK(VALUE > random()); +ALTER TABLE test_tab ALTER COLUMN a TYPE test_domain; +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', "DELETE FROM test_tab"); + +$node_subscriber->wait_for_log( + qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab" in parallel mode/, + $offset); + +# Drop domain constraint expression on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab ALTER COLUMN a TYPE int"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); +is($result, qq(0), + 'data replicated to subscriber after dropping domain constraint expression' +); + +# It is not allowed that constraint expression contains a non-immutable function +# on the subscriber side. Check the error reported by background worker in this +# case. +# First we check the constraint expression on normal table. +$node_subscriber->safe_psql( + 'postgres', qq{ +ALTER TABLE test_tab ADD CONSTRAINT test_tab_con check (a > random()); +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log( + qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab" in parallel mode/, + $offset); + +# Drop constraint on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab DROP CONSTRAINT test_tab_con"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); +is($result, qq(5000), + 'data replicated to subscriber after dropping constraint expression'); + +# Then we check the constraint expression on partition table. +$node_subscriber->safe_psql( + 'postgres', qq{ +ALTER TABLE test_tab_partition ADD CONSTRAINT test_tab_con check (a > random()); +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', "DELETE FROM test_tab_partitioned"); + +$node_subscriber->wait_for_log( + qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab_partitioned" in parallel mode/, + $offset); + +# Drop constraint on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partition DROP CONSTRAINT test_tab_con"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); +is($result, qq(0), + 'data replicated to subscriber after dropping constraint expression'); + +# It is not allowed that foreign key on the subscriber side. Check the error +# reported by background worker in this case. +# First we check the foreign key on normal table. +$node_publisher->safe_psql('postgres', "DELETE FROM test_tab"); +$node_publisher->wait_for_catchup($appname); +$node_subscriber->safe_psql( + 'postgres', qq{ +CREATE TABLE test_tab_f (a int primary key); +ALTER TABLE test_tab ADD CONSTRAINT test_tabfk FOREIGN KEY(a) REFERENCES test_tab_f(a); +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log( + qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab" in parallel mode/, + $offset); + +# Drop the foreign key constraint on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab DROP CONSTRAINT test_tabfk"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); +is($result, qq(5000), + 'data replicated to subscriber after dropping the foreign key'); + +# Then we check the foreign key on partition table. +$node_publisher->wait_for_catchup($appname); +$node_subscriber->safe_psql( + 'postgres', qq{ +CREATE TABLE test_tab_partition_f (a int primary key); +ALTER TABLE test_tab_partition ADD CONSTRAINT test_tab_patition_fk FOREIGN KEY(a) REFERENCES test_tab_partition_f(a); +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab_partitioned SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log( + qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab_partitioned" in parallel mode/, + $offset); + +# Drop the foreign key constraint on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partition DROP CONSTRAINT test_tab_patition_fk"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); +is($result, qq(5000), + 'data replicated to subscriber after dropping the foreign key'); + +$node_subscriber->stop; +$node_publisher->stop; + +done_testing(); diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 4137dc77b4..ae7cd99159 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -2271,6 +2271,7 @@ RelMapFile RelMapping RelOptInfo RelOptKind +RelParallel RelToCheck RelToCluster RelabelType -- 2.23.0.windows.1 [application/octet-stream] v15-0004-Retry-to-apply-streaming-xact-only-in-apply-work.patch (26.6K, ../../OS3PR01MB62755C6C9A75EB09F7218B589E839@OS3PR01MB6275.jpnprd01.prod.outlook.com/5-v15-0004-Retry-to-apply-streaming-xact-only-in-apply-work.patch) download | inline diff: From 25d936d1190c4ba6ec4c6cacb72e96b73f85c598 Mon Sep 17 00:00:00 2001 From: wangw <[email protected]> Date: Wed, 15 Jun 2022 11:02:08 +0800 Subject: [PATCH v15 4/4] Retry to apply streaming xact only in apply worker If the user sets the subscription_parameter "streaming" to "parallel", when applying a streaming transaction, we will try to apply this transaction in apply background worker. However, when the changes in this transaction cannot be applied in apply background worker, the background worker will exit with an error. In this case, we can retry applying this streaming transaction in "on" mode. In this way, we may avoid blocking logical replication here. So we introduce field "subretry" in catalog "pg_subscription". When the subscriber exit with an error, we will try to set this flag to true, and when the transaction is applied successfully, we will try to set this flag to false. Then when we try to apply a streaming transaction in apply background worker, we can see if this transaction has failed before based on the "subretry" field. --- doc/src/sgml/catalogs.sgml | 9 + doc/src/sgml/ref/create_subscription.sgml | 5 + src/backend/catalog/pg_subscription.c | 1 + src/backend/catalog/system_views.sql | 4 +- src/backend/commands/subscriptioncmds.c | 1 + .../replication/logical/applybgworker.c | 15 +- src/backend/replication/logical/worker.c | 89 ++++++++++ src/bin/pg_dump/pg_dump.c | 5 +- src/include/catalog/pg_subscription.h | 4 + .../subscription/t/032_streaming_apply.pl | 154 +++++++++++------- 10 files changed, 221 insertions(+), 66 deletions(-) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index 815cae6082..28f3d121d9 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -7907,6 +7907,15 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>subretry</structfield> <type>bool</type> + </para> + <para> + True if the previous apply change failed and a retry was required. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>subconninfo</structfield> <type>text</type> diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml index 270e3d382e..bd5361991e 100644 --- a/doc/src/sgml/ref/create_subscription.sgml +++ b/doc/src/sgml/ref/create_subscription.sgml @@ -244,6 +244,11 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl relation on the subscriber-side should also be the unique column on the publisher-side; 2) there cannot be any non-immutable functions in the subscriber-side replicated table. + When applying a streaming transaction, if either requirement is not + met, the background worker will exit with an error. + <literal>parallel</literal> mode is disregarded when retrying; + instead the transaction will be applied using <literal>on</literal> + mode. </para> </listitem> </varlistentry> diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c index 8856ce3b50..9b7f09653d 100644 --- a/src/backend/catalog/pg_subscription.c +++ b/src/backend/catalog/pg_subscription.c @@ -71,6 +71,7 @@ GetSubscription(Oid subid, bool missing_ok) sub->stream = subform->substream; sub->twophasestate = subform->subtwophasestate; sub->disableonerr = subform->subdisableonerr; + sub->retry = subform->subretry; /* Get conninfo */ datum = SysCacheGetAttr(SUBSCRIPTIONOID, diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql index fedaed533b..10f4dd6785 100644 --- a/src/backend/catalog/system_views.sql +++ b/src/backend/catalog/system_views.sql @@ -1298,8 +1298,8 @@ REVOKE ALL ON pg_replication_origin_status FROM public; -- All columns of pg_subscription except subconninfo are publicly readable. REVOKE ALL ON pg_subscription FROM public; GRANT SELECT (oid, subdbid, subskiplsn, subname, subowner, subenabled, - subbinary, substream, subtwophasestate, subdisableonerr, subslotname, - subsynccommit, subpublications) + subbinary, substream, subtwophasestate, subdisableonerr, + subretry, subslotname, subsynccommit, subpublications) ON pg_subscription TO public; CREATE VIEW pg_stat_subscription_stats AS diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c index 5f349067cc..33aef31b30 100644 --- a/src/backend/commands/subscriptioncmds.c +++ b/src/backend/commands/subscriptioncmds.c @@ -662,6 +662,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt, LOGICALREP_TWOPHASE_STATE_PENDING : LOGICALREP_TWOPHASE_STATE_DISABLED); values[Anum_pg_subscription_subdisableonerr - 1] = BoolGetDatum(opts.disableonerr); + values[Anum_pg_subscription_subretry - 1] = BoolGetDatum(false); values[Anum_pg_subscription_subconninfo - 1] = CStringGetTextDatum(conninfo); if (opts.slot_name) diff --git a/src/backend/replication/logical/applybgworker.c b/src/backend/replication/logical/applybgworker.c index 92b4b4e4ed..9bcc8669fb 100644 --- a/src/backend/replication/logical/applybgworker.c +++ b/src/backend/replication/logical/applybgworker.c @@ -106,6 +106,18 @@ apply_bgworker_can_start(TransactionId xid) if (!XLogRecPtrIsInvalid(MySubscription->skiplsn)) return false; + /* + * Don't use apply background workers for retries, because it is possible + * that the last time we tried to apply a transaction using an apply + * background worker the checks failed (see function + * apply_bgworker_relation_check). + */ + if (MySubscription->retry) + { + elog(DEBUG1, "apply background workers are not used for retries"); + return false; + } + /* * For streaming transactions that are being applied in apply background * worker, we cannot decide whether to apply the change for a relation @@ -812,6 +824,5 @@ apply_bgworker_relation_check(LogicalRepRelMapEntry *rel) "mode", rel->remoterel.nspname, rel->remoterel.relname), errdetail("The unique column on subscriber is not the unique " "column on publisher or there is at least one " - "non-immutable function."), - errhint("Please change the streaming option to 'on' instead of 'parallel'."))); + "non-immutable function."))); } diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c index 81e0005d12..a4aa6de533 100644 --- a/src/backend/replication/logical/worker.c +++ b/src/backend/replication/logical/worker.c @@ -378,6 +378,8 @@ static void clear_subscription_skip_lsn(XLogRecPtr finish_lsn); static inline void set_apply_error_context_xact(TransactionId xid, XLogRecPtr lsn); static inline void reset_apply_error_context_info(void); +static void set_subscription_retry(bool retry); + /* * Should this worker apply changes for given relation. * @@ -904,6 +906,9 @@ apply_handle_commit(StringInfo s) apply_handle_commit_internal(&commit_data); + /* Reset the retry flag. */ + set_subscription_retry(false); + /* Check the status of apply background worker if any. */ apply_bgworker_check_status(); @@ -1015,6 +1020,9 @@ apply_handle_prepare(StringInfo s) in_remote_transaction = false; + /* Reset the retry flag. */ + set_subscription_retry(false); + /* Check the status of apply background worker if any. */ apply_bgworker_check_status(); @@ -1068,6 +1076,9 @@ apply_handle_commit_prepared(StringInfo s) store_flush_position(prepare_data.end_lsn); in_remote_transaction = false; + /* Reset the retry flag. */ + set_subscription_retry(false); + /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(prepare_data.end_lsn); @@ -1123,6 +1134,9 @@ apply_handle_rollback_prepared(StringInfo s) store_flush_position(rollback_data.rollback_end_lsn); in_remote_transaction = false; + /* Reset the retry flag. */ + set_subscription_retry(false); + /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(rollback_data.rollback_end_lsn); @@ -1215,6 +1229,9 @@ apply_handle_stream_prepare(StringInfo s) /* Unlink the files with serialized changes and subxact info. */ stream_cleanup_files(MyLogicalRepWorker->subid, prepare_data.xid); } + + /* Reset the retry flag. */ + set_subscription_retry(false); } in_remote_transaction = false; @@ -1642,6 +1659,9 @@ apply_handle_stream_abort(StringInfo s) */ serialize_stream_abort(xid, subxid); } + + /* Reset the retry flag. */ + set_subscription_retry(false); } reset_apply_error_context_info(); @@ -1854,6 +1874,9 @@ apply_handle_stream_commit(StringInfo s) /* Unlink the files with serialized changes and subxact info. */ stream_cleanup_files(MyLogicalRepWorker->subid, xid); } + + /* Reset the retry flag. */ + set_subscription_retry(false); } /* Check the status of apply background worker if any. */ @@ -3897,6 +3920,9 @@ start_table_sync(XLogRecPtr *origin_startpos, char **myslotname) } PG_CATCH(); { + /* Set the retry flag. */ + set_subscription_retry(true); + if (MySubscription->disableonerr) DisableSubscriptionAndExit(); else @@ -3935,6 +3961,9 @@ start_apply(XLogRecPtr origin_startpos) } PG_CATCH(); { + /* Set the retry flag. */ + set_subscription_retry(true); + if (MySubscription->disableonerr) DisableSubscriptionAndExit(); else @@ -4461,3 +4490,63 @@ reset_apply_error_context_info(void) apply_error_callback_arg.remote_attnum = -1; set_apply_error_context_xact(InvalidTransactionId, InvalidXLogRecPtr); } + +/* + * Set subretry of pg_subscription catalog. + * + * If retry is true, subscriber is about to exit with an error. Otherwise, it + * means that the transaction was applied successfully. + */ +static void +set_subscription_retry(bool retry) +{ + Relation rel; + HeapTuple tup; + bool started_tx = false; + bool nulls[Natts_pg_subscription]; + bool replaces[Natts_pg_subscription]; + Datum values[Natts_pg_subscription]; + + if (MySubscription->retry == retry || + am_apply_bgworker()) + return; + + if (!IsTransactionState()) + { + StartTransactionCommand(); + started_tx = true; + } + + /* Look up the subscription in the catalog */ + rel = table_open(SubscriptionRelationId, RowExclusiveLock); + tup = SearchSysCacheCopy1(SUBSCRIPTIONOID, + ObjectIdGetDatum(MySubscription->oid)); + + if (!HeapTupleIsValid(tup)) + elog(ERROR, "subscription \"%s\" does not exist", MySubscription->name); + + LockSharedObject(SubscriptionRelationId, MySubscription->oid, 0, + AccessShareLock); + + /* Form a new tuple. */ + memset(values, 0, sizeof(values)); + memset(nulls, false, sizeof(nulls)); + memset(replaces, false, sizeof(replaces)); + + /* reset subretry */ + values[Anum_pg_subscription_subretry - 1] = BoolGetDatum(retry); + replaces[Anum_pg_subscription_subretry - 1] = true; + + tup = heap_modify_tuple(tup, RelationGetDescr(rel), values, nulls, + replaces); + + /* Update the catalog. */ + CatalogTupleUpdate(rel, &tup->t_self, tup); + + /* Cleanup. */ + heap_freetuple(tup); + table_close(rel, NoLock); + + if (started_tx) + CommitTransactionCommand(); +} diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 24927641b9..9ce774fc39 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -4471,8 +4471,9 @@ getSubscriptions(Archive *fout) ntups = PQntuples(res); /* - * Get subscription fields. We don't include subskiplsn in the dump as - * after restoring the dump this value may no longer be relevant. + * Get subscription fields. We don't include subskiplsn and subretry in + * the dump as after restoring the dump this value may no longer be + * relevant. */ i_tableoid = PQfnumber(res, "tableoid"); i_oid = PQfnumber(res, "oid"); diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h index d54540f5f5..5f4e058ec1 100644 --- a/src/include/catalog/pg_subscription.h +++ b/src/include/catalog/pg_subscription.h @@ -76,6 +76,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW bool subdisableonerr; /* True if a worker error should cause the * subscription to be disabled */ + bool subretry BKI_DEFAULT(f); /* True if the previous apply change failed. */ + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* Connection string to the publisher */ text subconninfo BKI_FORCE_NOT_NULL; @@ -116,6 +118,8 @@ typedef struct Subscription bool disableonerr; /* Indicates if the subscription should be * automatically disabled if a worker error * occurs */ + bool retry; /* Indicates if the previous apply change + * failed. */ char *conninfo; /* Connection string to the publisher */ char *slotname; /* Name of the replication slot */ char *synccommit; /* Synchronous commit setting for worker */ diff --git a/src/test/subscription/t/032_streaming_apply.pl b/src/test/subscription/t/032_streaming_apply.pl index eca4328676..1bcb4c65b7 100644 --- a/src/test/subscription/t/032_streaming_apply.pl +++ b/src/test/subscription/t/032_streaming_apply.pl @@ -57,8 +57,13 @@ $node_subscriber->safe_psql( $node_publisher->wait_for_catchup($appname); +# ============================================================================ # It is not allowed that the unique index on the publisher and the subscriber -# is different. Check the error reported by background worker in this case. +# is different. Check the error reported by background worker in this case. And +# after retrying in apply worker, we check if the data is replicated +# successfully. +# ============================================================================ + # First we check the unique index on normal table. $node_subscriber->safe_psql('postgres', "CREATE UNIQUE INDEX test_tab_b_idx ON test_tab (b)"); @@ -82,14 +87,15 @@ $node_subscriber->wait_for_log( qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab" in parallel mode/, $offset); -# Drop the unique index on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', "DROP INDEX test_tab_b_idx"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); my $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); -is($result, qq(5000), 'data replicated to subscriber after dropping index'); +is($result, qq(5000), 'data replicated to subscribers after retrying because of unique index'); + +# Drop the unique index on the subscriber. +$node_subscriber->safe_psql('postgres', "DROP INDEX test_tab_b_idx"); # Then we check the unique index on partition table. $node_subscriber->safe_psql('postgres', @@ -106,17 +112,20 @@ $node_subscriber->wait_for_log( qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab_partitioned" in parallel mode/, $offset); -# Drop the unique index on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', "DROP INDEX test_tab_b_partition_idx"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); -is($result, qq(5000), 'data replicated to subscriber after dropping index'); +is($result, qq(5000), 'data replicated to subscribers after retrying because of unique index'); + +# Drop the unique index on the subscriber. +$node_subscriber->safe_psql('postgres', "DROP INDEX test_tab_b_partition_idx"); # Triggers which execute non-immutable function are not allowed on the # subscriber side. Check the error reported by background worker in this case. +# And after retrying in apply worker, we check if the data is replicated +# successfully. # First we check the trigger function on normal table. $node_subscriber->safe_psql( 'postgres', qq{ @@ -140,15 +149,16 @@ $node_subscriber->wait_for_log( qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab" in parallel mode/, $offset); -# Drop the trigger on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "DROP TRIGGER insert_trig ON test_tab"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); -is($result, qq(0), 'data replicated to subscriber after dropping trigger'); +is($result, qq(0), 'data replicated to subscribers after retrying because of trigger'); + +# Drop the trigger on the subscriber. +$node_subscriber->safe_psql('postgres', + "DROP TRIGGER insert_trig ON test_tab"); # Then we check the trigger function on partition table. $node_subscriber->safe_psql( @@ -168,19 +178,24 @@ $node_subscriber->wait_for_log( qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab_partitioned" in parallel mode/, $offset); -# Drop the trigger on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "DROP TRIGGER insert_trig ON test_tab_partition"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); -is($result, qq(0), 'data replicated to subscriber after dropping trigger'); +is($result, qq(0), 'data replicated to subscribers after retrying because of trigger'); +# Drop the trigger on the subscriber. +$node_subscriber->safe_psql('postgres', + "DROP TRIGGER insert_trig ON test_tab_partition"); + +# ============================================================================ # It is not allowed that column default value expression contains a # non-immutable function on the subscriber side. Check the error reported by -# background worker in this case. +# background worker in this case. And after retrying in apply worker, we check +# if the data is replicated successfully. +# ============================================================================ + # First we check the column default value expression on normal table. $node_subscriber->safe_psql('postgres', "ALTER TABLE test_tab ALTER COLUMN b SET DEFAULT random()"); @@ -196,16 +211,17 @@ $node_subscriber->wait_for_log( qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab" in parallel mode/, $offset); -# Drop default value on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "ALTER TABLE test_tab ALTER COLUMN b DROP DEFAULT"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); is($result, qq(5000), - 'data replicated to subscriber after dropping default value expression'); + 'data replicated to subscribers after retrying because of column default value'); + +# Drop default value on the subscriber. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab ALTER COLUMN b DROP DEFAULT"); # Then we check the column default value expression on partition table. $node_subscriber->safe_psql('postgres', @@ -222,20 +238,25 @@ $node_subscriber->wait_for_log( qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab_partitioned" in parallel mode/, $offset); -# Drop default value on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "ALTER TABLE test_tab_partition ALTER COLUMN b DROP DEFAULT"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); is($result, qq(5000), - 'data replicated to subscriber after dropping default value expression'); + 'data replicated to subscribers after retrying because of column default value'); + +# Drop default value on the subscriber. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partition ALTER COLUMN b DROP DEFAULT"); +# ============================================================================ # It is not allowed that domain constraint expression contains a non-immutable # function on the subscriber side. Check the error reported by background -# worker in this case. +# worker in this case. And after retrying in apply worker, we check if the data +# is replicated successfully. +# ============================================================================ + # Because the column type of the partition table must be the same as its parent # table, only test normal table here. $node_subscriber->safe_psql( @@ -253,21 +274,26 @@ $node_subscriber->wait_for_log( qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab" in parallel mode/, $offset); -# Drop domain constraint expression on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "ALTER TABLE test_tab ALTER COLUMN a TYPE int"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); is($result, qq(0), - 'data replicated to subscriber after dropping domain constraint expression' + 'data replicated to subscribers after retrying because of domain' ); -# It is not allowed that constraint expression contains a non-immutable function -# on the subscriber side. Check the error reported by background worker in this -# case. +# Drop domain constraint expression on the subscriber. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab ALTER COLUMN a TYPE int"); + +# ============================================================================ +# It is not allowed that constraint expression contains a non-immutable +# function on the subscriber side. Check the error reported by background +# worker in this case. And after retrying in apply worker, we check if the data +# is replicated successfully. +# ============================================================================ + # First we check the constraint expression on normal table. $node_subscriber->safe_psql( 'postgres', qq{ @@ -285,16 +311,17 @@ $node_subscriber->wait_for_log( qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab" in parallel mode/, $offset); -# Drop constraint on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "ALTER TABLE test_tab DROP CONSTRAINT test_tab_con"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); is($result, qq(5000), - 'data replicated to subscriber after dropping constraint expression'); + 'data replicated to subscribers after retrying because of constraint'); + +# Drop constraint on the subscriber. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab DROP CONSTRAINT test_tab_con"); # Then we check the constraint expression on partition table. $node_subscriber->safe_psql( @@ -311,19 +338,24 @@ $node_subscriber->wait_for_log( qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab_partitioned" in parallel mode/, $offset); -# Drop constraint on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "ALTER TABLE test_tab_partition DROP CONSTRAINT test_tab_con"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); is($result, qq(0), - 'data replicated to subscriber after dropping constraint expression'); + 'data replicated to subscribers after retrying because of constraint'); +# Drop constraint on the subscriber. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partition DROP CONSTRAINT test_tab_con"); + +# ============================================================================ # It is not allowed that foreign key on the subscriber side. Check the error -# reported by background worker in this case. +# reported by background worker in this case. And after retrying in apply +# worker, we check if the data is replicated successfully. +# ============================================================================ + # First we check the foreign key on normal table. $node_publisher->safe_psql('postgres', "DELETE FROM test_tab"); $node_publisher->wait_for_catchup($appname); @@ -344,16 +376,17 @@ $node_subscriber->wait_for_log( qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab" in parallel mode/, $offset); -# Drop the foreign key constraint on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "ALTER TABLE test_tab DROP CONSTRAINT test_tabfk"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); is($result, qq(5000), - 'data replicated to subscriber after dropping the foreign key'); + 'data replicated to subscribers after retrying because of foreign key'); + +# Drop the foreign key constraint on the subscriber. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab DROP CONSTRAINT test_tabfk"); # Then we check the foreign key on partition table. $node_publisher->wait_for_catchup($appname); @@ -374,16 +407,17 @@ $node_subscriber->wait_for_log( qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab_partitioned" in parallel mode/, $offset); -# Drop the foreign key constraint on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "ALTER TABLE test_tab_partition DROP CONSTRAINT test_tab_patition_fk"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); is($result, qq(5000), - 'data replicated to subscriber after dropping the foreign key'); + 'data replicated to subscribers after retrying because of foreign key'); + +# Drop the foreign key constraint on the subscriber. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partition DROP CONSTRAINT test_tab_patition_fk"); $node_subscriber->stop; $node_publisher->stop; -- 2.23.0.windows.1 ^ permalink raw reply [nested|flat] 66+ messages in thread
* RE: Perform streaming logical transactions by background workers and parallel apply @ 2022-07-07 03:45 [email protected] <[email protected]> parent: Amit Kapila <[email protected]> 0 siblings, 0 replies; 66+ messages in thread From: [email protected] @ 2022-07-07 03:45 UTC (permalink / raw) To: Amit Kapila <[email protected]>; Peter Smith <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; PostgreSQL Hackers <[email protected]> On Fri, Jul 1, 2022 at 17:44 PM Amit Kapila <[email protected]> wrote: > Thanks for your comments. > On Fri, Jul 1, 2022 at 12:13 PM Peter Smith <[email protected]> wrote: > > > > ====== > > > > 1.2 doc/src/sgml/protocol.sgml - Protocol constants > > > > Previously I wrote that since there are protocol changes here, > > shouldn’t there also be some corresponding LOGICALREP_PROTO_XXX > > constants and special checking added in the worker.c? > > > > But you said [1 comment #6] you think it is OK because... > > > > IMO, I still disagree with the reply. The fact is that the protocol > > *has* been changed, so IIUC that is precisely the reason for having > > those protocol constants. > > > > e.g I am guessing you might assign the new one somewhere here: > > -- > > server_version = walrcv_server_version(LogRepWorkerWalRcvConn); > > options.proto.logical.proto_version = > > server_version >= 150000 ? > LOGICALREP_PROTO_TWOPHASE_VERSION_NUM : > > server_version >= 140000 ? > LOGICALREP_PROTO_STREAM_VERSION_NUM : > > LOGICALREP_PROTO_VERSION_NUM; > > -- > > > > And then later you would refer to this new protocol version (instead > > of the server version) when calling to the apply_handle_stream_abort > > function. > > > > ====== > > > > One point related to this that occurred to me is how it will behave if > the publisher is of version >=16 whereas the subscriber is of versions > <=15? Won't in that case publisher sends the new fields but > subscribers won't be reading those which may cause some problems. Makes sense. Fixed this point. As Peter-san suggested, I added a new protocol macro LOGICALREP_PROTO_STREAM_PARALLEL_VERSION_NUM. This new macro marks the version that supports apply background worker (it means we will read abort_lsn and abort_time). And the publisher sends abort_lsn and abort_time fields only if subscriber will read them. (see function logicalrep_write_stream_abort) The new patches were attached in [1]. [1] - https://www.postgresql.org/message-id/OS3PR01MB62755C6C9A75EB09F7218B589E839%40OS3PR01MB6275.jpnprd0... Regards, Wang wei ^ permalink raw reply [nested|flat] 66+ messages in thread
* RE: Perform streaming logical transactions by background workers and parallel apply @ 2022-07-07 03:46 [email protected] <[email protected]> parent: Peter Smith <[email protected]> 0 siblings, 0 replies; 66+ messages in thread From: [email protected] @ 2022-07-07 03:46 UTC (permalink / raw) To: Peter Smith <[email protected]>; +Cc: Amit Kapila <[email protected]>; Masahiko Sawada <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; PostgreSQL Hackers <[email protected]> On Mon, Jul 4, 2022 at 12:12 AM Peter Smith <[email protected]> wrote: > Below are some review comments for patch v14-0003: Thanks for your comments. > 3.1 Commit message > > If any of the following checks are violated, an error will be reported. > 1. The unique columns between publisher and subscriber are difference. > 2. There is any non-immutable function present in expression in > subscriber's relation. Check from the following 4 items: > a. The function in triggers; > b. Column default value expressions and domain constraints; > c. Constraint expressions. > d. The foreign keys. > > SUGGESTION (rewording to match the docs and the code). > > Add some checks before using apply background worker to apply changes. > streaming=parallel mode has two requirements: > 1) The unique columns must be the same between publisher and subscriber > 2) There cannot be any non-immutable functions in the subscriber-side > replicated table. Look for functions in the following places: > * a. Trigger functions > * b. Column default value expressions and domain constraints > * c. Constraint expressions > * d. Foreign keys > > ====== > > 3.2 doc/src/sgml/ref/create_subscription.sgml > > + To run in this mode, there are following two requirements. The first > + is that the unique column should be the same between publisher and > + subscriber; the second is that there should not be any non-immutable > + function in subscriber-side replicated table. > > SUGGESTION > Parallel mode has two requirements: 1) the unique columns must be the > same between publisher and subscriber; 2) there cannot be any > non-immutable functions in the subscriber-side replicated table. I did not write clearly enough before. So I made some slight modifications to the first requirement you suggested. Like this: ``` 1) The unique column in the relation on the subscriber-side should also be the unique column on the publisher-side; ``` > 3.9 src/backend/replication/logical/proto.c - logicalrep_write_attrs > > This big slab of new code to get the BMS looks very similar to > RelationGetIdentityKeyBitmap. So perhaps this code should be > encapsulated in another function like that one (in relcache.c?) and > then just called from logicalrep_write_attrs I think the file relcache.c should contain cache-build operations, and the code I added doesn't have this operation. So I didn't change. > 3.12 src/backend/replication/logical/relation.c - > logicalrep_rel_mark_safe_in_apply_bgworker > > I did not really understand why you used an enum for one flag > (volatility) but not the other one (sameunique); shouldn’t both of > these be tri-values: unknown/yes/no? > > For E.g. there is a quick exit from this function if the > FUNCTION_UNKNOWN, but there is no equivalent quick exit for the > sameunique? It seems inconsistent. After rethinking patch 0003, I think we only need one flag. So I merged flags 'volatility' and 'sameunique' into a new flag 'parallel'. It is a tri-state flag. And I also made some related modifications. > 3.14 src/backend/replication/logical/relation.c - > logicalrep_rel_mark_safe_in_apply_bgworker > > There are lots of places setting FUNCTION_NONIMMUTABLE, so I think > this code might be tidier if you just have a single return at the end > of this function and 'goto' it. > > e.g. > if (...) > goto function_not_immutable; > > ... > > return; > > function_not_immutable: > entry->volatility = FUNCTION_NONIMMUTABLE; Personally, I do not like to use the `goto` syntax if it is not necessary, because the `goto` syntax will forcibly change the flow of code execution. > 3.17 src/backend/utils/cache/typcache.c > > +/* > + * GetDomainConstraints --- get DomainConstraintState list of > specified domain type > + */ > +List * > +GetDomainConstraints(Oid type_id) > > This is an unusual-looking function header comment, with the function > name and the "---". Not sure about this. Please refer to function lookup_rowtype_tupdesc_internal. > 3.19 > > @@ -31,6 +42,11 @@ typedef struct LogicalRepRelMapEntry > Relation localrel; /* relcache entry (NULL when closed) */ > AttrMap *attrmap; /* map of local attributes to remote ones */ > bool updatable; /* Can apply updates/deletes? */ > + bool sameunique; /* Are all unique columns of the local > + relation contained by the unique columns in > + remote? */ > > (This is similar to review comment 3.12) > > I felt it was inconsistent for this to be a boolean but for the > 'volatility' member to be an enum. AFAIK these 2 flags are similar > kinds – e.g. essentially tri-state flags unknown/true/false so I > thought they should be treated the same. E.g. both enums? Please refer to the reply to #3.12. > 3.22 .../subscription/t/032_streaming_apply.pl > > 3.22.a > +# Setup structure on publisher > > "structure"? > > 3.22.b > +# Setup structure on subscriber > > "structure"? Just refer to other subscription test. > 3.23 > > +# Check that a background worker starts if "streaming" option is specified as > +# "parallel". We have to look for the DEBUG1 log messages about that, so > +# temporarily bump up the log verbosity. > +$node_subscriber->append_conf('postgresql.conf', "log_min_messages = > debug1"); > +$node_subscriber->reload; > + > +$node_publisher->safe_psql('postgres', > + "INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1, > 5000) s(i)" > +); > + > +$node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); > +$node_subscriber->append_conf('postgresql.conf', > + "log_min_messages = warning"); > +$node_subscriber->reload; > > I didn't really think it was necessary to bump this log level, and to > verify that the bgworker is started, because this test is anyway going > to ensure that the ERROR "cannot replicate relation with different > unique index" happens, so that is already implicitly ensuring the > bgworker was used. Since it takes almost no time, I think a more detailed confirmation is fine. The rest of the comments are improved as suggested. The new patches were attached in [1]. [1] - https://www.postgresql.org/message-id/OS3PR01MB62755C6C9A75EB09F7218B589E839%40OS3PR01MB6275.jpnprd0... Regards, Wang wei ^ permalink raw reply [nested|flat] 66+ messages in thread
* RE: Perform streaming logical transactions by background workers and parallel apply @ 2022-07-07 03:47 [email protected] <[email protected]> parent: Peter Smith <[email protected]> 0 siblings, 0 replies; 66+ messages in thread From: [email protected] @ 2022-07-07 03:47 UTC (permalink / raw) To: Peter Smith <[email protected]>; +Cc: Amit Kapila <[email protected]>; Masahiko Sawada <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; PostgreSQL Hackers <[email protected]> On Mon, Jul 4, 2022 at 14:47 AM Peter Smith <[email protected]> wrote: > Below are some review comments for patch v14-0004: Thanks for your comments. > 4.0 General. > > This comment is an after-thought but as I write this mail I am > wondering if most of this 0004 patch is even necessary at all? Instead > of introducing a new column and all the baggage that goes with it, > can't the same functionality be achieved just by toggling the > streaming mode 'substream' value from 'p' (parallel) to 't' (on) > whenever an error occurs causing a retry? Anyway, if you do change it > this way then most of the following comments can be disregarded. In the approach that you mentioned, after retrying, the transaction will always be applied in "on" mode. This will change the user's setting. That is to say, in most cases, user needs to manually reset option "streaming" to "parallel". I think it might not be very friendly. > 4.6 src/backend/replication/logical/worker.c > > 4.6.a - apply_handle_commit > > + /* Set the flag that we will not retry later. */ > + set_subscription_retry(false); > > But the comment is wrong, isn't it? Shouldn’t it just say that we are > not *currently* retrying? And can’t this just anyway be redundant if > only the catalog column has a DEFAULT value of false? > > 4.6.b - apply_handle_prepare > Ditto > > 4.6.c - apply_handle_commit_prepared > Ditto > > 4.6.d - apply_handle_rollback_prepared > Ditto > > 4.6.e - apply_handle_stream_prepare > Ditto > > 4.6.f - apply_handle_stream_abort > Ditto > > 4.6.g - apply_handle_stream_commit > Ditto Set default value of the field "subretry" to "false" as you suggested. We need to reset this field to false after retrying to apply a streaming transaction in main apply worker ("on" mode). I think this comment is not clear. So I change it to ``` Reset the retry flag. ``` > 4.7 src/backend/replication/logical/worker.c > > 4.7.a - start_table_sync > > @@ -3894,6 +3917,9 @@ start_table_sync(XLogRecPtr *origin_startpos, > char **myslotname) > } > PG_CATCH(); > { > + /* Set the flag that we will retry later. */ > + set_subscription_retry(true); > > Maybe this should say more like "Flag that the next apply will be the > result of a retry" > > 4.7.b - start_apply > Ditto Similar to the reply in #4.6, I changed it to `Set the retry flag.`. > 4.8 src/backend/replication/logical/worker.c - set_subscription_retry > > + > +/* > + * Set subretry of pg_subscription catalog. > + * > + * If retry is true, subscriber is about to exit with an error. Otherwise, it > + * means that the changes was applied successfully. > + */ > +static void > +set_subscription_retry(bool retry) > > "changes" -> "change" ? I did not make it clear before. I modified "changes" to "transaction". > 4.8 src/backend/replication/logical/worker.c - set_subscription_retry > > Isn't this flag only every used when streaming=parallel? But it does > not seem ot be checking that anywhere before potentiall executing all > this code when maybe will never be used. Yes, currently this field is only checked by apply background worker. > 4.9 src/include/catalog/pg_subscription.h > > @@ -76,6 +76,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) > BKI_SHARED_RELATION BKI_ROW > bool subdisableonerr; /* True if a worker error should cause the > * subscription to be disabled */ > > + bool subretry; /* True if the previous apply change failed. */ > > I was wondering if you can give this column a DEFAULT value of false, > because then perhaps most of the patch code from worker.c may be able > to be eliminated. Please refer to the reply to #4.6. The rest of the comments are improved as suggested. The new patches were attached in [1]. [1] - https://www.postgresql.org/message-id/OS3PR01MB62755C6C9A75EB09F7218B589E839%40OS3PR01MB6275.jpnprd0... Regards, Wang wei ^ permalink raw reply [nested|flat] 66+ messages in thread
* RE: Perform streaming logical transactions by background workers and parallel apply @ 2022-07-07 10:20 [email protected] <[email protected]> parent: [email protected] <[email protected]> 0 siblings, 1 reply; 66+ messages in thread From: [email protected] @ 2022-07-07 10:20 UTC (permalink / raw) To: Peter Smith <[email protected]>; +Cc: Amit Kapila <[email protected]>; Masahiko Sawada <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; PostgreSQL Hackers <[email protected]> On Fri, Jul 7, 2022 at 11:44 AM I wrote: > Attach the new patches. I found a failure on CFbot [1], which after investigation I think is due to my previous modification (see response to #1.10 in [2]). For a streaming transaction, if we failed in the first chunk of streamed changes for this transaction in the apply background worker, we will set the status of this apply background worker to APPLY_BGWORKER_EXIT. And at the same time, main apply worker obtains apply background worker in the function apply_bgworker_find when processing the second chunk of streamed changes for this transaction, the status of apply background worker is APPLY_BGWORKER_EXIT. So the following assertion will fail: ``` Assert(status == APPLY_BGWORKER_BUSY); ``` To fix this, before invoking function assert, I try to detect the failure of apply background worker. If the status is APPLY_BGWORKER_EXIT, then exit with an error. I also made some other small improvements. Attach the new patches. [1] - https://cirrus-ci.com/task/6383178511286272?logs=test_world#L2636 [2] - https://www.postgresql.org/message-id/OS3PR01MB62755C6C9A75EB09F7218B589E839%40OS3PR01MB6275.jpnprd0... Regards, Wang wei Attachments: [application/octet-stream] v16-0001-Perform-streaming-logical-transactions-by-backgr.patch (105.9K, ../../OS3PR01MB627594D75E870BBB45E2E80A9E839@OS3PR01MB6275.jpnprd01.prod.outlook.com/2-v16-0001-Perform-streaming-logical-transactions-by-backgr.patch) download | inline diff: From 4d3bc1a5ba7b15f6d5fd0a975912052fcf124604 Mon Sep 17 00:00:00 2001 From: "houzj.fnst" <[email protected]> Date: Wed, 20 Apr 2022 16:45:07 +0800 Subject: [PATCH v16 1/4] Perform streaming logical transactions by background workers Currently, for large transactions, the publisher sends the data in multiple streams (changes divided into chunks depending upon logical_decoding_work_mem), and then on the subscriber-side, the apply worker writes the changes into temporary files and once it receives the commit, it reads from the file and applies the entire transaction. To improve the performance of such transactions, we can instead allow them to be applied via background workers. In this approach, we assign a new apply background worker (if available) as soon as the xact's first stream is received and the main apply worker will send changes to this new worker via shared memory. The apply background worker will directly apply the change instead of writing it to temporary files. We keep this worker assigned till the transaction commit is received and also wait for the worker to finish at commit. This preserves commit ordering and avoids writing to and reading from file in most cases. We still need to spill if there is no worker available. This patch also extends the SUBSCRIPTION 'streaming' parameter so that the user can control whether to apply the streaming transaction in an apply background worker or spill the change to disk. The user can set the streaming parameter to 'on/off', 'parallel'. The parameter value 'parallel' means the streaming will be applied via an apply background worker, if available. The parameter value 'on' means the streaming transaction will be spilled to disk. The default value is 'off' (same as current behaviour). --- doc/src/sgml/catalogs.sgml | 10 +- doc/src/sgml/config.sgml | 25 + doc/src/sgml/logical-replication.sgml | 10 + doc/src/sgml/protocol.sgml | 19 + doc/src/sgml/ref/create_subscription.sgml | 24 +- src/backend/access/transam/xact.c | 13 + src/backend/commands/subscriptioncmds.c | 66 +- src/backend/postmaster/bgworker.c | 3 + src/backend/replication/logical/Makefile | 1 + .../replication/logical/applybgworker.c | 786 ++++++++++++++++++ src/backend/replication/logical/decode.c | 10 +- src/backend/replication/logical/launcher.c | 130 ++- src/backend/replication/logical/origin.c | 26 +- src/backend/replication/logical/proto.c | 41 +- .../replication/logical/reorderbuffer.c | 10 +- src/backend/replication/logical/tablesync.c | 10 +- src/backend/replication/logical/worker.c | 688 +++++++++++---- src/backend/replication/pgoutput/pgoutput.c | 9 +- src/backend/utils/activity/wait_event.c | 3 + src/backend/utils/misc/guc.c | 12 + src/backend/utils/misc/postgresql.conf.sample | 1 + src/bin/pg_dump/pg_dump.c | 6 +- src/include/catalog/pg_subscription.h | 21 +- src/include/replication/logicallauncher.h | 1 + src/include/replication/logicalproto.h | 27 +- src/include/replication/logicalworker.h | 1 + src/include/replication/origin.h | 2 +- src/include/replication/reorderbuffer.h | 7 +- src/include/replication/worker_internal.h | 102 ++- src/include/utils/wait_event.h | 1 + src/test/regress/expected/subscription.out | 2 +- src/tools/pgindent/typedefs.list | 5 + 32 files changed, 1852 insertions(+), 220 deletions(-) create mode 100644 src/backend/replication/logical/applybgworker.c diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index 4f3f375a84..815cae6082 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -7873,11 +7873,15 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l <row> <entry role="catalog_table_entry"><para role="column_definition"> - <structfield>substream</structfield> <type>bool</type> + <structfield>substream</structfield> <type>char</type> </para> <para> - If true, the subscription will allow streaming of in-progress - transactions + Controls how to handle the streaming of in-progress transactions: + <literal>f</literal> = disallow streaming of in-progress transactions, + <literal>t</literal> = spill the changes of in-progress transactions to + disk and apply at once after the transaction is committed on the + publisher, + <literal>p</literal> = apply changes directly using a background worker </para></entry> </row> diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml index 37fd80388c..6bbd986195 100644 --- a/doc/src/sgml/config.sgml +++ b/doc/src/sgml/config.sgml @@ -4970,6 +4970,31 @@ ANY <replaceable class="parameter">num_sync</replaceable> ( <replaceable class=" </listitem> </varlistentry> + <varlistentry id="guc-max-apply-bgworkers-per-subscription" xreflabel="max_apply_bgworkers_per_subscription"> + <term><varname>max_apply_bgworkers_per_subscription</varname> (<type>integer</type>) + <indexterm> + <primary><varname>max_apply_bgworkers_per_subscription</varname> configuration parameter</primary> + </indexterm> + </term> + <listitem> + <para> + Maximum number of apply background workers per subscription. This + parameter controls the amount of parallelism of the streaming of + in-progress transactions when subscription parameter + <literal>streaming = parallel</literal>. + </para> + <para> + The apply background workers are taken from the pool defined by + <varname>max_logical_replication_workers</varname>. + </para> + <para> + The default value is 2. This parameter can only be set in the + <filename>postgresql.conf</filename> file or on the server command + line. + </para> + </listitem> + </varlistentry> + </variablelist> </sect2> diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml index bdf1e7b727..92997f9299 100644 --- a/doc/src/sgml/logical-replication.sgml +++ b/doc/src/sgml/logical-replication.sgml @@ -1153,6 +1153,16 @@ CONTEXT: processing remote data for replication origin "pg_16395" during "INSER might not violate any constraint. This can easily make the subscriber inconsistent. </para> + + <para> + When the streaming mode is <literal>parallel</literal>, the finish LSN of + failed transactions may not be logged. In that case, it may be necessary to + change the streaming mode to <literal>on</literal> and cause the same + conflicts again so the finish LSN of the failed transaction will be written + to the server log. For the usage of finish LSN, please refer to <link + linkend="sql-altersubscription"><command>ALTER SUBSCRIPTION ... + SKIP</command></link>. + </para> </sect1> <sect1 id="logical-replication-restrictions"> diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml index c0b89a3c01..7e88ba9631 100644 --- a/doc/src/sgml/protocol.sgml +++ b/doc/src/sgml/protocol.sgml @@ -6809,6 +6809,25 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;" </listitem> </varlistentry> + <varlistentry> + <term>Int64 (XLogRecPtr)</term> + <listitem> + <para> + The LSN of the abort. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Int64 (TimestampTz)</term> + <listitem> + <para> + Abort timestamp of the transaction. The value is in number + of microseconds since PostgreSQL epoch (2000-01-01). + </para> + </listitem> + </varlistentry> + <varlistentry> <term>Int32 (TransactionId)</term> <listitem> diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml index 34b3264b26..71dd4aca81 100644 --- a/doc/src/sgml/ref/create_subscription.sgml +++ b/doc/src/sgml/ref/create_subscription.sgml @@ -217,13 +217,29 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl </varlistentry> <varlistentry> - <term><literal>streaming</literal> (<type>boolean</type>)</term> + <term><literal>streaming</literal> (<type>enum</type>)</term> <listitem> <para> Specifies whether to enable streaming of in-progress transactions - for this subscription. By default, all transactions - are fully decoded on the publisher and only then sent to the - subscriber as a whole. + for this subscription. The default value is <literal>off</literal>, + meaning all transactions are fully decoded on the publisher and only + then sent to the subscriber as a whole. + </para> + + <para> + If set to <literal>on</literal>, the incoming changes are written to + temporary files and then applied only after the transaction is + committed on the publisher. + </para> + + <para> + If set to <literal>parallel</literal>, incoming changes are directly + applied via one of the apply background workers, if available. If no + background worker is free to handle streaming transaction then the + changes are written to temporary files and applied after the + transaction is committed. Note that if an error happens when + applying changes in a background worker, the finish LSN of the + remote transaction might not be reported in the server log. </para> </listitem> </varlistentry> diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c index 116de1175b..3e61a57b50 100644 --- a/src/backend/access/transam/xact.c +++ b/src/backend/access/transam/xact.c @@ -1711,6 +1711,7 @@ RecordTransactionAbort(bool isSubXact) int nchildren; TransactionId *children; TimestampTz xact_time; + bool replorigin; /* * If we haven't been assigned an XID, nobody will care whether we aborted @@ -1741,6 +1742,13 @@ RecordTransactionAbort(bool isSubXact) elog(PANIC, "cannot abort transaction %u, it was already committed", xid); + /* + * Are we using the replication origins feature? Or, in other words, + * are we replaying remote actions? + */ + replorigin = (replorigin_session_origin != InvalidRepOriginId && + replorigin_session_origin != DoNotReplicateId); + /* Fetch the data we need for the abort record */ nrels = smgrGetPendingDeletes(false, &rels); nchildren = xactGetCommittedChildren(&children); @@ -1765,6 +1773,11 @@ RecordTransactionAbort(bool isSubXact) MyXactFlags, InvalidTransactionId, NULL); + if (replorigin) + /* Move LSNs forward for this replication origin */ + replorigin_session_advance(replorigin_session_origin_lsn, + XactLastRecEnd); + /* * Report the latest async abort LSN, so that the WAL writer knows to * flush this abort. There's nothing to be gained by delaying this, since diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c index bdc1208724..5f349067cc 100644 --- a/src/backend/commands/subscriptioncmds.c +++ b/src/backend/commands/subscriptioncmds.c @@ -83,7 +83,7 @@ typedef struct SubOpts bool copy_data; bool refresh; bool binary; - bool streaming; + char streaming; bool twophase; bool disableonerr; XLogRecPtr lsn; @@ -95,6 +95,62 @@ static List *merge_publications(List *oldpublist, List *newpublist, bool addpub, static void ReportSlotConnectionError(List *rstates, Oid subid, char *slotname, char *err); +/* + * Extract the streaming mode value from a DefElem. This is like + * defGetBoolean() but also accepts the special value of "parallel". + */ +static char +defGetStreamingMode(DefElem *def) +{ + /* + * If no value given, assume "true" is meant. + */ + if (def->arg == NULL) + return SUBSTREAM_ON; + + /* + * Allow 0, 1, "false", "true", "off", "on" or "parallel". + */ + switch (nodeTag(def->arg)) + { + case T_Integer: + switch (intVal(def->arg)) + { + case 0: + return SUBSTREAM_OFF; + case 1: + return SUBSTREAM_ON; + default: + /* otherwise, error out below */ + break; + } + break; + default: + { + char *sval = defGetString(def); + + /* + * The set of strings accepted here should match up with the + * grammar's opt_boolean_or_string production. + */ + if (pg_strcasecmp(sval, "false") == 0 || + pg_strcasecmp(sval, "off") == 0) + return SUBSTREAM_OFF; + if (pg_strcasecmp(sval, "true") == 0 || + pg_strcasecmp(sval, "on") == 0) + return SUBSTREAM_ON; + if (pg_strcasecmp(sval, "parallel") == 0) + return SUBSTREAM_PARALLEL; + } + break; + } + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s requires a Boolean value or \"parallel\"", + def->defname))); + return SUBSTREAM_OFF; /* keep compiler quiet */ +} + /* * Common option parsing function for CREATE and ALTER SUBSCRIPTION commands. * @@ -132,7 +188,7 @@ parse_subscription_options(ParseState *pstate, List *stmt_options, if (IsSet(supported_opts, SUBOPT_BINARY)) opts->binary = false; if (IsSet(supported_opts, SUBOPT_STREAMING)) - opts->streaming = false; + opts->streaming = SUBSTREAM_OFF; if (IsSet(supported_opts, SUBOPT_TWOPHASE_COMMIT)) opts->twophase = false; if (IsSet(supported_opts, SUBOPT_DISABLE_ON_ERR)) @@ -233,7 +289,7 @@ parse_subscription_options(ParseState *pstate, List *stmt_options, errorConflictingDefElem(defel, pstate); opts->specified_opts |= SUBOPT_STREAMING; - opts->streaming = defGetBoolean(defel); + opts->streaming = defGetStreamingMode(defel); } else if (strcmp(defel->defname, "two_phase") == 0) { @@ -600,7 +656,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt, values[Anum_pg_subscription_subowner - 1] = ObjectIdGetDatum(owner); values[Anum_pg_subscription_subenabled - 1] = BoolGetDatum(opts.enabled); values[Anum_pg_subscription_subbinary - 1] = BoolGetDatum(opts.binary); - values[Anum_pg_subscription_substream - 1] = BoolGetDatum(opts.streaming); + values[Anum_pg_subscription_substream - 1] = CharGetDatum(opts.streaming); values[Anum_pg_subscription_subtwophasestate - 1] = CharGetDatum(opts.twophase ? LOGICALREP_TWOPHASE_STATE_PENDING : @@ -1059,7 +1115,7 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt, if (IsSet(opts.specified_opts, SUBOPT_STREAMING)) { values[Anum_pg_subscription_substream - 1] = - BoolGetDatum(opts.streaming); + CharGetDatum(opts.streaming); replaces[Anum_pg_subscription_substream - 1] = true; } diff --git a/src/backend/postmaster/bgworker.c b/src/backend/postmaster/bgworker.c index 40601aefd9..40ccb8993c 100644 --- a/src/backend/postmaster/bgworker.c +++ b/src/backend/postmaster/bgworker.c @@ -128,6 +128,9 @@ static const struct }, { "ApplyWorkerMain", ApplyWorkerMain + }, + { + "ApplyBgworkerMain", ApplyBgworkerMain } }; diff --git a/src/backend/replication/logical/Makefile b/src/backend/replication/logical/Makefile index c4e2fdeb71..cbfb5d794e 100644 --- a/src/backend/replication/logical/Makefile +++ b/src/backend/replication/logical/Makefile @@ -15,6 +15,7 @@ include $(top_builddir)/src/Makefile.global override CPPFLAGS := -I$(srcdir) $(CPPFLAGS) OBJS = \ + applybgworker.o \ decode.o \ launcher.o \ logical.o \ diff --git a/src/backend/replication/logical/applybgworker.c b/src/backend/replication/logical/applybgworker.c new file mode 100644 index 0000000000..bb2b180e29 --- /dev/null +++ b/src/backend/replication/logical/applybgworker.c @@ -0,0 +1,786 @@ +/*------------------------------------------------------------------------- + * applybgworker.c + * Support routines for applying xact by apply background worker + * + * Copyright (c) 2016-2022, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/replication/logical/applybgworker.c + * + * This file contains routines that are intended to support setting up, using, + * and tearing down a ApplyBgworkerState. + * + * Refer to the comments in file header of logical/worker.c to see more + * information about apply background worker. + * + *------------------------------------------------------------------------- + */ + +#include "postgres.h" + +#include "libpq/pqformat.h" +#include "mb/pg_wchar.h" +#include "pgstat.h" +#include "postmaster/interrupt.h" +#include "replication/logicallauncher.h" +#include "replication/logicalworker.h" +#include "replication/origin.h" +#include "replication/walreceiver.h" +#include "replication/worker_internal.h" +#include "storage/ipc.h" +#include "storage/procarray.h" +#include "tcop/tcopprot.h" +#include "utils/inval.h" +#include "utils/memutils.h" +#include "utils/resowner.h" +#include "utils/syscache.h" + +#define PG_LOGICAL_APPLY_SHM_MAGIC 0x79fb2447 // TODO Consider change + +/* + * DSM keys for apply background worker. Unlike other parallel execution code, + * since we don't need to worry about DSM keys conflicting with plan_node_id we + * can use small integers. + */ +#define APPLY_BGWORKER_KEY_SHARED 1 +#define APPLY_BGWORKER_KEY_MQ 2 + +/* Queue size of DSM, 16 MB for now. */ +#define DSM_QUEUE_SIZE 160000000 + +/* + * There are three fields in message: start_lsn, end_lsn and send_time. Because + * we have updated these statistics in apply worker, we could ignore these + * fields in apply background worker. (see function LogicalRepApplyLoop) + */ +#define IGNORE_SIZE_IN_MESSAGE (3 * sizeof(uint64)) + +/* + * Entry for a hash table we use to map from xid to our apply background worker + * state. + */ +typedef struct ApplyBgworkerEntry +{ + TransactionId xid; + ApplyBgworkerState *wstate; +} ApplyBgworkerEntry; + +/* Apply background workers hash table (initialized on first use) */ +static HTAB *ApplyWorkersHash = NULL; +static List *ApplyWorkersFreeList = NIL; +static List *ApplyWorkersList = NIL; + +/* + * Information shared between main apply worker and apply background worker. + */ +volatile ApplyBgworkerShared *MyParallelState = NULL; + +List *subxactlist = NIL; + +static bool apply_bgworker_can_start(TransactionId xid); +static ApplyBgworkerState *apply_bgworker_setup(void); +static void apply_bgworker_setup_dsm(ApplyBgworkerState *wstate); + +/* + * Check if starting a new apply background worker is allowed. + */ +static bool +apply_bgworker_can_start(TransactionId xid) +{ + if (!TransactionIdIsValid(xid)) + return false; + + /* + * Don't start a new background worker if not in streaming parallel mode. + */ + if (MySubscription->stream != SUBSTREAM_PARALLEL) + return false; + + /* + * Don't start a new background worker if user has set skiplsn as it's + * possible that user want to skip the streaming transaction. For + * streaming transaction, we need to spill the transaction to disk so that + * we can get the last LSN of the transaction to judge whether to skip + * before starting to apply the change. + */ + if (!XLogRecPtrIsInvalid(MySubscription->skiplsn)) + return false; + + /* + * For streaming transactions that are being applied in apply background + * worker, we cannot decide whether to apply the change for a relation + * that is not in the READY state (see should_apply_changes_for_rel) as we + * won't know remote_final_lsn by that time. So, we don't start new apply + * background worker in this case. + */ + if (!AllTablesyncsReady()) + return false; + + return true; +} + +/* + * Try to start an apply background worker and, if successful, cache it in + * ApplyWorkersHash keyed by the specified xid. + */ +ApplyBgworkerState * +apply_bgworker_start(TransactionId xid) +{ + bool found; + ApplyBgworkerState *wstate; + ApplyBgworkerEntry *entry = NULL; + + if (!apply_bgworker_can_start(xid)) + return NULL; + + /* First time through, initialize apply workers hashtable */ + if (ApplyWorkersHash == NULL) + { + HASHCTL ctl; + + MemSet(&ctl, 0, sizeof(ctl)); + ctl.keysize = sizeof(TransactionId); + ctl.entrysize = sizeof(ApplyBgworkerEntry); + ctl.hcxt = ApplyContext; + + ApplyWorkersHash = hash_create("logical apply workers hash", 8, &ctl, + HASH_ELEM | HASH_BLOBS | HASH_CONTEXT); + } + + /* + * Now, we try to get an apply background worker. If there is at least one + * worker in the free list, then take one. Otherwise, we try to start a + * new apply background worker. + */ + if (list_length(ApplyWorkersFreeList) > 0) + { + wstate = (ApplyBgworkerState *) llast(ApplyWorkersFreeList); + ApplyWorkersFreeList = list_delete_last(ApplyWorkersFreeList); + Assert(wstate->pstate->status == APPLY_BGWORKER_FINISHED); + } + else + { + wstate = apply_bgworker_setup(); + + if (wstate == NULL) + return NULL; + } + + /* + * Create entry for requested transaction. + */ + entry = hash_search(ApplyWorkersHash, &xid, HASH_ENTER, &found); + if (found) + elog(ERROR, "hash table corrupted"); + + /* Fill up the hash entry */ + wstate->pstate->status = APPLY_BGWORKER_BUSY; + wstate->pstate->server_version = walrcv_server_version(LogRepWorkerWalRcvConn); + wstate->pstate->stream_xid = xid; + entry->wstate = wstate; + entry->xid = xid; + + return wstate; +} + +/* + * Try to look up worker inside ApplyWorkersHash for requested xid. + */ +ApplyBgworkerState * +apply_bgworker_find(TransactionId xid) +{ + bool found; + ApplyBgworkerEntry *entry = NULL; + + if (!TransactionIdIsValid(xid)) + return NULL; + + if (ApplyWorkersHash == NULL) + return NULL; + + /* + * Find entry for requested transaction. + */ + entry = hash_search(ApplyWorkersHash, &xid, HASH_FIND, &found); + if (found) + { + char status = entry->wstate->pstate->status; + + /* If any workers (or the postmaster) have died, we have failed. */ + if (status == APPLY_BGWORKER_EXIT) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("background worker %u failed to apply transaction %u", + entry->wstate->pstate->n, + entry->wstate->pstate->stream_xid))); + + Assert(status == APPLY_BGWORKER_BUSY); + + return entry->wstate; + } + else + return NULL; +} + +/* + * Add the worker to the free list and remove the entry from the hash table. + */ +void +apply_bgworker_free(ApplyBgworkerState *wstate) +{ + MemoryContext oldctx; + TransactionId xid = wstate->pstate->stream_xid; + + Assert(wstate->pstate->status == APPLY_BGWORKER_FINISHED); + + oldctx = MemoryContextSwitchTo(ApplyContext); + + hash_search(ApplyWorkersHash, &xid, HASH_REMOVE, NULL); + + elog(DEBUG1, "adding finished apply worker #%u for xid %u to the free list", + wstate->pstate->n, wstate->pstate->stream_xid); + + ApplyWorkersFreeList = lappend(ApplyWorkersFreeList, wstate); + + MemoryContextSwitchTo(oldctx); +} + +/* Apply Background Worker main loop */ +static void +LogicalApplyBgwLoop(shm_mq_handle *mqh, volatile ApplyBgworkerShared *pst) +{ + shm_mq_result shmq_res; + PGPROC *registrant; + ErrorContextCallback errcallback; + + registrant = BackendPidGetProc(MyBgworkerEntry->bgw_notify_pid); + SetLatch(®istrant->procLatch); + + /* + * Push apply error context callback. Fields will be filled applying a + * change. + */ + errcallback.callback = apply_error_callback; + errcallback.previous = error_context_stack; + error_context_stack = &errcallback; + + for (;;) + { + void *data; + Size len; + int c; + StringInfoData s; + MemoryContext oldctx; + + CHECK_FOR_INTERRUPTS(); + + /* Ensure we are reading the data into our memory context. */ + oldctx = MemoryContextSwitchTo(ApplyMessageContext); + + shmq_res = shm_mq_receive(mqh, &len, &data, false); + + if (shmq_res != SHM_MQ_SUCCESS) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("lost connection to the main apply worker"))); + + if (len == 0) + break; + + s.cursor = 0; + s.maxlen = -1; + s.data = (char *) data; + s.len = len; + + /* + * We use first byte of message for additional communication between + * main Logical replication worker and apply background workers, so if + * it differs from 'w', then process it first. + */ + c = pq_getmsgbyte(&s); + switch (c) + { + /* End message of streaming chunk */ + case LOGICAL_REP_MSG_STREAM_STOP: + elog(DEBUG1, "[Apply BGW #%u] ended processing streaming chunk," + "waiting on shm_mq_receive", pst->n); + + in_streamed_transaction = false; + pgstat_report_activity(STATE_IDLEINTRANSACTION, NULL); + continue; + case 'w': + break; + default: + elog(ERROR, "[Apply BGW #%u] unexpected message \"%c\"", + pst->n, c); + break; + } + + /* Ignore statistics fields that have been updated. */ + s.cursor += IGNORE_SIZE_IN_MESSAGE; + + apply_dispatch(&s); + + if (ConfigReloadPending) + { + ConfigReloadPending = false; + ProcessConfigFile(PGC_SIGHUP); + } + + MemoryContextSwitchTo(oldctx); + MemoryContextReset(ApplyMessageContext); + } + + MemoryContextSwitchTo(TopMemoryContext); + MemoryContextReset(ApplyContext); + + /* Pop the error context stack */ + error_context_stack = errcallback.previous; + + elog(DEBUG1, "[Apply BGW #%u] exiting", pst->n); + + /* Signal main process that we are done. */ + SetLatch(®istrant->procLatch); +} + +/* + * Set the exit status so that the main apply worker can realize we have + * shutdown. + */ +static void +apply_bgworker_shutdown(int code, Datum arg) +{ + SpinLockAcquire(&MyParallelState->mutex); + MyParallelState->status = APPLY_BGWORKER_EXIT; + SpinLockRelease(&MyParallelState->mutex); + + dsm_detach((dsm_segment *) DatumGetPointer(arg)); +} + +/* + * Apply Background Worker entry point + */ +void +ApplyBgworkerMain(Datum main_arg) +{ + volatile ApplyBgworkerShared *pst; + + dsm_handle handle; + dsm_segment *seg; + shm_toc *toc; + shm_mq *mq; + shm_mq_handle *mqh; + MemoryContext oldcontext; + RepOriginId originid; + int worker_slot = DatumGetInt32(main_arg); + char originname[NAMEDATALEN]; + + MemoryContextSwitchTo(TopMemoryContext); + + /* Init the memory context for the apply background worker to work in. */ + ApplyContext = AllocSetContextCreate(TopMemoryContext, + "ApplyContext", + ALLOCSET_DEFAULT_SIZES); + + /* + * Init the ApplyMessageContext which we clean up after each replication + * protocol message. + */ + ApplyMessageContext = AllocSetContextCreate(ApplyContext, + "ApplyMessageContext", + ALLOCSET_DEFAULT_SIZES); + + /* Setup signal handling */ + pqsignal(SIGHUP, SignalHandlerForConfigReload); + pqsignal(SIGTERM, die); + BackgroundWorkerUnblockSignals(); + + /* + * Connect to the dynamic shared memory segment. + * + * The backend that registered this worker passed us the ID of a shared + * memory segment to which we must attach for further instructions. In + * order to attach to dynamic shared memory, we need a resource owner. + * Once we've mapped the segment in our address space, attach to the table + * of contents so we can locate the various data structures we'll need to + * find within the segment. + */ + CurrentResourceOwner = ResourceOwnerCreate(NULL, "Logical apply worker"); + memcpy(&handle, MyBgworkerEntry->bgw_extra, sizeof(dsm_handle)); + seg = dsm_attach(handle); + if (seg == NULL) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("unable to map dynamic shared memory segment"))); + toc = shm_toc_attach(PG_LOGICAL_APPLY_SHM_MAGIC, dsm_segment_address(seg)); + if (toc == NULL) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("bad magic number in dynamic shared memory segment"))); + + before_shmem_exit(apply_bgworker_shutdown, PointerGetDatum(seg)); + + /* Look up the parallel state. */ + pst = shm_toc_lookup(toc, APPLY_BGWORKER_KEY_SHARED, false); + MyParallelState = pst; + + /* + * Attach to the message queue. + */ + mq = shm_toc_lookup(toc, APPLY_BGWORKER_KEY_MQ, false); + shm_mq_set_receiver(mq, MyProc); + mqh = shm_mq_attach(mq, seg, NULL); + + /* Run as replica session replication role. */ + SetConfigOption("session_replication_role", "replica", + PGC_SUSET, PGC_S_OVERRIDE); + + /* + * Now, we have initialized DSM. Attach to slot. + */ + logicalrep_worker_attach(worker_slot); + + /* Connect to our database. */ + BackgroundWorkerInitializeConnectionByOid(MyLogicalRepWorker->dbid, + MyLogicalRepWorker->userid, + 0); + + /* + * Set always-secure search path, so malicious users can't redirect user + * code (e.g. pg_index.indexprs). + */ + SetConfigOption("search_path", "", PGC_SUSET, PGC_S_OVERRIDE); + + /* + * Set the client encoding to the database encoding, since that is what + * the leader will expect. + */ + SetClientEncoding(GetDatabaseEncoding()); + + stream_xid = pst->stream_xid; + + StartTransactionCommand(); + oldcontext = MemoryContextSwitchTo(ApplyContext); + + MySubscription = GetSubscription(MyLogicalRepWorker->subid, true); + if (!MySubscription) + { + ereport(LOG, + (errmsg("logical replication apply worker for subscription %u will not " + "start because the subscription was removed during startup", + MyLogicalRepWorker->subid))); + proc_exit(0); + } + + MySubscriptionValid = true; + MemoryContextSwitchTo(oldcontext); + + /* Setup synchronous commit according to the user's wishes */ + SetConfigOption("synchronous_commit", MySubscription->synccommit, + PGC_BACKEND, PGC_S_OVERRIDE); + + /* Keep us informed about subscription changes. */ + CacheRegisterSyscacheCallback(SUBSCRIPTIONOID, + subscription_change_cb, + (Datum) 0); + + CommitTransactionCommand(); + + /* Setup replication origin tracking. */ + StartTransactionCommand(); + snprintf(originname, sizeof(originname), "pg_%u", MySubscription->oid); + originid = replorigin_by_name(originname, true); + if (!OidIsValid(originid)) + originid = replorigin_create(originname); + + /* + * The apply background worker doesn't need to monopolize this replication + * origin which was already acquired by its leader process. + */ + replorigin_session_setup(originid, false); + replorigin_session_origin = originid; + CommitTransactionCommand(); + + /* + * Allocate the origin name in long-lived context for error context + * message. + */ + apply_error_callback_arg.origin_name = MemoryContextStrdup(ApplyContext, + originname); + + elog(DEBUG1, "[Apply BGW #%u] started", pst->n); + + LogicalApplyBgwLoop(mqh, pst); + + /* + * We're done. Explicitly detach the shared memory segment so that we + * don't get a resource leak warning at commit time. This will fire any + * on_dsm_detach callbacks we've registered, as well. Once that's done, + * we can go ahead and exit. + */ + dsm_detach(seg); + proc_exit(0); +} + +/* + * Set up a dynamic shared memory segment. + * + * We set up a control region that contains a ApplyBgworkerShared, + * plus one region per message queue. There are as many message queues as + * the number of workers. + */ +static void +apply_bgworker_setup_dsm(ApplyBgworkerState *wstate) +{ + shm_toc_estimator e; + Size segsize; + dsm_segment *seg; + shm_toc *toc; + ApplyBgworkerShared *pst; + shm_mq *mq; + int64 queue_size = DSM_QUEUE_SIZE; + + /* + * Estimate how much shared memory we need. + * + * Because the TOC machinery may choose to insert padding of oddly-sized + * requests, we must estimate each chunk separately. + * + * We need one key to register the location of the header, and we need + * another key to track the location of the message queue. + */ + shm_toc_initialize_estimator(&e); + shm_toc_estimate_chunk(&e, sizeof(ApplyBgworkerShared)); + shm_toc_estimate_chunk(&e, (Size) queue_size); + + shm_toc_estimate_keys(&e, 1 + 1); + segsize = shm_toc_estimate(&e); + + /* Create the shared memory segment and establish a table of contents. */ + seg = dsm_create(shm_toc_estimate(&e), 0); + toc = shm_toc_create(PG_LOGICAL_APPLY_SHM_MAGIC, dsm_segment_address(seg), + segsize); + + /* Set up the header region. */ + pst = shm_toc_allocate(toc, sizeof(ApplyBgworkerShared)); + SpinLockInit(&pst->mutex); + pst->status = APPLY_BGWORKER_BUSY; + pst->server_version = walrcv_server_version(LogRepWorkerWalRcvConn); + pst->stream_xid = stream_xid; + pst->n = list_length(ApplyWorkersList) + 1; + + shm_toc_insert(toc, APPLY_BGWORKER_KEY_SHARED, pst); + + /* Set up message queue for the worker. */ + mq = shm_mq_create(shm_toc_allocate(toc, (Size) queue_size), + (Size) queue_size); + shm_toc_insert(toc, APPLY_BGWORKER_KEY_MQ, mq); + shm_mq_set_sender(mq, MyProc); + + /* Attach the queue. */ + wstate->mq_handle = shm_mq_attach(mq, seg, NULL); + + /* Return results to caller. */ + wstate->dsm_seg = seg; + wstate->pstate = pst; +} + +/* + * Start apply background worker process and allocate shared memory for it. + */ +static ApplyBgworkerState * +apply_bgworker_setup(void) +{ + MemoryContext oldcontext; + bool launched; + ApplyBgworkerState *wstate; + int napplyworkers; + + elog(DEBUG1, "setting up apply worker #%u", list_length(ApplyWorkersList) + 1); + + /* Check If there are free worker slot(s) */ + LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); + napplyworkers = logicalrep_apply_bgworker_count(MyLogicalRepWorker->subid); + LWLockRelease(LogicalRepWorkerLock); + if (napplyworkers >= max_apply_bgworkers_per_subscription) + return NULL; + + oldcontext = MemoryContextSwitchTo(ApplyContext); + + wstate = (ApplyBgworkerState *) palloc0(sizeof(ApplyBgworkerState)); + + /* Setup shared memory */ + apply_bgworker_setup_dsm(wstate); + + launched = logicalrep_worker_launch(MyLogicalRepWorker->dbid, + MySubscription->oid, + MySubscription->name, + MyLogicalRepWorker->userid, + InvalidOid, + dsm_segment_handle(wstate->dsm_seg)); + + if (launched) + ApplyWorkersList = lappend(ApplyWorkersList, wstate); + else + { + dsm_detach(wstate->dsm_seg); + wstate->dsm_seg = NULL; + + pfree(wstate); + wstate = NULL; + } + + MemoryContextSwitchTo(oldcontext); + + return wstate; +} + +/* + * Send the data to the specified apply background worker via shared-memory queue. + */ +void +apply_bgworker_send_data(ApplyBgworkerState *wstate, Size nbytes, const void *data) +{ + shm_mq_result result; + + result = shm_mq_send(wstate->mq_handle, nbytes, data, false, true); + + if (result != SHM_MQ_SUCCESS) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("could not send tuples to shared-memory queue"))); +} + +/* + * Wait until the status of apply background worker reaches the + * 'wait_for_status' + */ +void +apply_bgworker_wait_for(ApplyBgworkerState *wstate, + ApplyBgworkerStatus wait_for_status) +{ + for (;;) + { + char status; + + SpinLockAcquire(&wstate->pstate->mutex); + status = wstate->pstate->status; + SpinLockRelease(&wstate->pstate->mutex); + + /* Done if already in correct status. */ + if (status == wait_for_status) + break; + + /* If any workers (or the postmaster) have died, we have failed. */ + if (status == APPLY_BGWORKER_EXIT) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("background worker %u failed to apply transaction %u", + wstate->pstate->n, wstate->pstate->stream_xid))); + + /* Wait to be signalled. */ + WaitLatch(MyLatch, WL_LATCH_SET | WL_EXIT_ON_PM_DEATH, 0, + WAIT_EVENT_LOGICAL_APPLY_WORKER_STATE_CHANGE); + + /* Reset the latch so we don't spin. */ + ResetLatch(MyLatch); + + /* An interrupt may have occurred while we were waiting. */ + CHECK_FOR_INTERRUPTS(); + } +} + +/* + * Check the status of workers and report an error if any apply background + * worker has exited unexpectedly. + */ +void +apply_bgworker_check_status(void) +{ + ListCell *lc; + + if (am_apply_bgworker() || MySubscription->stream != SUBSTREAM_PARALLEL) + return; + + foreach(lc, ApplyWorkersList) + { + ApplyBgworkerState *wstate = (ApplyBgworkerState *) lfirst(lc); + + /* + * We don't lock here as in the worst case we will just detect the + * failure of worker a bit later. + */ + if (wstate->pstate->status == APPLY_BGWORKER_EXIT) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("background worker %u exited unexpectedly", + wstate->pstate->n))); + } + + /* + * Exit if any relation is not in the READY state and if any worker is + * handling the streaming transaction at the same time. Because for + * streaming transactions that is being applied in apply background + * worker, we cannot decide whether to apply the change for a relation + * that is not in the READY state (see should_apply_changes_for_rel) as we + * won't know remote_final_lsn by that time. + */ + if (list_length(ApplyWorkersFreeList) != list_length(ApplyWorkersList) && + !AllTablesyncsReady()) + { + ereport(LOG, + (errmsg("logical replication apply workers for subscription \"%s\" will restart", + MySubscription->name), + errdetail("Cannot handle streamed replication transaction by apply " + "background workers until all tables are synchronized"))); + + proc_exit(0); + } +} + +/* Set the apply background worker status */ +void +apply_bgworker_set_status(ApplyBgworkerStatus status) +{ + if (!am_apply_bgworker()) + return; + + elog(DEBUG1, "[Apply BGW #%u] set status to %d", MyParallelState->n, status); + + SpinLockAcquire(&MyParallelState->mutex); + MyParallelState->status = status; + SpinLockRelease(&MyParallelState->mutex); +} + +/* + * Define a savepoint for a subxact in apply background worker if needed. + * + * Inside apply background worker we can figure out that new subtransaction was + * started if new change arrived with different xid. In that case we can define + * named savepoint, so that we were able to commit/rollback it separately + * later. + * Special case is if the first change comes from subtransaction, then + * we check that current_xid differs from stream_xid. + */ +void +apply_bgworker_subxact_info_add(TransactionId current_xid) +{ + if (current_xid != stream_xid && + !list_member_int(subxactlist, (int) current_xid)) + { + MemoryContext oldctx; + char spname[MAXPGPATH]; + + snprintf(spname, MAXPGPATH, "savepoint_for_xid_%u", current_xid); + + elog(DEBUG1, "[Apply BGW #%u] defining savepoint %s", + MyParallelState->n, spname); + + DefineSavepoint(spname); + CommitTransactionCommand(); + + oldctx = MemoryContextSwitchTo(ApplyContext); + subxactlist = lappend_int(subxactlist, (int) current_xid); + MemoryContextSwitchTo(oldctx); + } +} diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c index c5c6a2ba68..d4d5093a0b 100644 --- a/src/backend/replication/logical/decode.c +++ b/src/backend/replication/logical/decode.c @@ -651,9 +651,10 @@ DecodeCommit(LogicalDecodingContext *ctx, XLogRecordBuffer *buf, { for (i = 0; i < parsed->nsubxacts; i++) { - ReorderBufferForget(ctx->reorder, parsed->subxacts[i], buf->origptr); + ReorderBufferForget(ctx->reorder, parsed->subxacts[i], buf->origptr, + commit_time); } - ReorderBufferForget(ctx->reorder, xid, buf->origptr); + ReorderBufferForget(ctx->reorder, xid, buf->origptr, commit_time); return; } @@ -821,10 +822,11 @@ DecodeAbort(LogicalDecodingContext *ctx, XLogRecordBuffer *buf, for (i = 0; i < parsed->nsubxacts; i++) { ReorderBufferAbort(ctx->reorder, parsed->subxacts[i], - buf->record->EndRecPtr); + buf->record->EndRecPtr, abort_time); } - ReorderBufferAbort(ctx->reorder, xid, buf->record->EndRecPtr); + ReorderBufferAbort(ctx->reorder, xid, buf->record->EndRecPtr, + abort_time); } /* update the decoding stats */ diff --git a/src/backend/replication/logical/launcher.c b/src/backend/replication/logical/launcher.c index 2bdab53e19..d11d2361c6 100644 --- a/src/backend/replication/logical/launcher.c +++ b/src/backend/replication/logical/launcher.c @@ -54,6 +54,7 @@ int max_logical_replication_workers = 4; int max_sync_workers_per_subscription = 2; +int max_apply_bgworkers_per_subscription = 2; LogicalRepWorker *MyLogicalRepWorker = NULL; @@ -73,6 +74,7 @@ static void logicalrep_launcher_onexit(int code, Datum arg); static void logicalrep_worker_onexit(int code, Datum arg); static void logicalrep_worker_detach(void); static void logicalrep_worker_cleanup(LogicalRepWorker *worker); +static void logicalrep_worker_stop_internal(LogicalRepWorker *worker); static bool on_commit_launcher_wakeup = false; @@ -151,8 +153,10 @@ get_subscription_list(void) * * This is only needed for cleaning up the shared memory in case the worker * fails to attach. + * + * Return false if the attach fails. Otherwise return true. */ -static void +static bool WaitForReplicationWorkerAttach(LogicalRepWorker *worker, uint16 generation, BackgroundWorkerHandle *handle) @@ -168,11 +172,11 @@ WaitForReplicationWorkerAttach(LogicalRepWorker *worker, LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); - /* Worker either died or has started; no need to do anything. */ + /* Worker either died or has started. Return false if died. */ if (!worker->in_use || worker->proc) { LWLockRelease(LogicalRepWorkerLock); - return; + return worker->in_use; } LWLockRelease(LogicalRepWorkerLock); @@ -187,7 +191,7 @@ WaitForReplicationWorkerAttach(LogicalRepWorker *worker, if (generation == worker->generation) logicalrep_worker_cleanup(worker); LWLockRelease(LogicalRepWorkerLock); - return; + return false; } /* @@ -223,6 +227,13 @@ logicalrep_worker_find(Oid subid, Oid relid, bool only_running) { LogicalRepWorker *w = &LogicalRepCtx->workers[i]; + /* + * We are only interested in the main apply worker or table sync worker + * here. + */ + if (w->subworker) + continue; + if (w->in_use && w->subid == subid && w->relid == relid && (!only_running || w->proc)) { @@ -259,11 +270,11 @@ logicalrep_workers_find(Oid subid, bool only_running) } /* - * Start new apply background worker, if possible. + * Start new background worker, if possible. */ -void +bool logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, Oid userid, - Oid relid) + Oid relid, dsm_handle subworker_dsm) { BackgroundWorker bgw; BackgroundWorkerHandle *bgw_handle; @@ -273,6 +284,10 @@ logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, Oid userid, LogicalRepWorker *worker = NULL; int nsyncworkers; TimestampTz now; + bool is_subworker = (subworker_dsm != DSM_HANDLE_INVALID); + + /* Sanity check : we don't support table sync in subworker. */ + Assert(!(is_subworker && OidIsValid(relid))); ereport(DEBUG1, (errmsg_internal("starting logical replication worker for subscription \"%s\"", @@ -350,7 +365,7 @@ retry: if (OidIsValid(relid) && nsyncworkers >= max_sync_workers_per_subscription) { LWLockRelease(LogicalRepWorkerLock); - return; + return false; } /* @@ -364,7 +379,7 @@ retry: (errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED), errmsg("out of logical replication worker slots"), errhint("You might need to increase max_logical_replication_workers."))); - return; + return false; } /* Prepare the worker slot. */ @@ -379,6 +394,7 @@ retry: worker->relstate = SUBREL_STATE_UNKNOWN; worker->relstate_lsn = InvalidXLogRecPtr; worker->stream_fileset = NULL; + worker->subworker = is_subworker; worker->last_lsn = InvalidXLogRecPtr; TIMESTAMP_NOBEGIN(worker->last_send_time); TIMESTAMP_NOBEGIN(worker->last_recv_time); @@ -396,19 +412,31 @@ retry: BGWORKER_BACKEND_DATABASE_CONNECTION; bgw.bgw_start_time = BgWorkerStart_RecoveryFinished; snprintf(bgw.bgw_library_name, BGW_MAXLEN, "postgres"); - snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyWorkerMain"); + + if (is_subworker) + snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyBgworkerMain"); + else + snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyWorkerMain"); + if (OidIsValid(relid)) snprintf(bgw.bgw_name, BGW_MAXLEN, "logical replication worker for subscription %u sync %u", subid, relid); + else if (is_subworker) + snprintf(bgw.bgw_name, BGW_MAXLEN, + "logical replication apply background worker for subscription %u", subid); else snprintf(bgw.bgw_name, BGW_MAXLEN, "logical replication worker for subscription %u", subid); + snprintf(bgw.bgw_type, BGW_MAXLEN, "logical replication worker"); bgw.bgw_restart_time = BGW_NEVER_RESTART; bgw.bgw_notify_pid = MyProcPid; bgw.bgw_main_arg = Int32GetDatum(slot); + if (is_subworker) + memcpy(bgw.bgw_extra, &subworker_dsm, sizeof(dsm_handle)); + if (!RegisterDynamicBackgroundWorker(&bgw, &bgw_handle)) { /* Failed to start worker, so clean up the worker slot. */ @@ -421,11 +449,11 @@ retry: (errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED), errmsg("out of background worker slots"), errhint("You might need to increase max_worker_processes."))); - return; + return false; } /* Now wait until it attaches. */ - WaitForReplicationWorkerAttach(worker, generation, bgw_handle); + return WaitForReplicationWorkerAttach(worker, generation, bgw_handle); } /* @@ -436,18 +464,27 @@ void logicalrep_worker_stop(Oid subid, Oid relid) { LogicalRepWorker *worker; - uint16 generation; LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); worker = logicalrep_worker_find(subid, relid, false); - /* No worker, nothing to do. */ - if (!worker) - { - LWLockRelease(LogicalRepWorkerLock); - return; - } + if (worker) + logicalrep_worker_stop_internal(worker); + + LWLockRelease(LogicalRepWorkerLock); +} + +/* + * Workhorse for logicalrep_worker_stop() and logicalrep_worker_detach(). Stop + * the worker and wait for it to die. + */ +static void +logicalrep_worker_stop_internal(LogicalRepWorker *worker) +{ + uint16 generation; + + Assert(LWLockHeldByMe(LogicalRepWorkerLock)); /* * Remember which generation was our worker so we can check if what we see @@ -485,10 +522,7 @@ logicalrep_worker_stop(Oid subid, Oid relid) * different, meaning that a different worker has taken the slot. */ if (!worker->in_use || worker->generation != generation) - { - LWLockRelease(LogicalRepWorkerLock); return; - } /* Worker has assigned proc, so it has started. */ if (worker->proc) @@ -522,8 +556,6 @@ logicalrep_worker_stop(Oid subid, Oid relid) LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); } - - LWLockRelease(LogicalRepWorkerLock); } /* @@ -599,6 +631,29 @@ logicalrep_worker_attach(int slot) static void logicalrep_worker_detach(void) { + /* + * If we are the main apply worker, stop all the apply background workers + * we started before. + */ + if (!MyLogicalRepWorker->subworker) + { + List *workers; + ListCell *lc; + + LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); + + workers = logicalrep_workers_find(MyLogicalRepWorker->subid, true); + foreach(lc, workers) + { + LogicalRepWorker *w = (LogicalRepWorker *) lfirst(lc); + + if (w->subworker) + logicalrep_worker_stop_internal(w); + } + + LWLockRelease(LogicalRepWorkerLock); + } + /* Block concurrent access. */ LWLockAcquire(LogicalRepWorkerLock, LW_EXCLUSIVE); @@ -621,6 +676,7 @@ logicalrep_worker_cleanup(LogicalRepWorker *worker) worker->userid = InvalidOid; worker->subid = InvalidOid; worker->relid = InvalidOid; + worker->subworker = false; } /* @@ -679,6 +735,30 @@ logicalrep_sync_worker_count(Oid subid) return res; } +/* + * Count the number of registered (not necessarily running) apply background + * workers for a subscription. + */ +int +logicalrep_apply_bgworker_count(Oid subid) +{ + int i; + int res = 0; + + Assert(LWLockHeldByMe(LogicalRepWorkerLock)); + + /* Search for attached worker for a given subscription id. */ + for (i = 0; i < max_logical_replication_workers; i++) + { + LogicalRepWorker *w = &LogicalRepCtx->workers[i]; + + if (w->subid == subid && w->subworker) + res++; + } + + return res; +} + /* * ApplyLauncherShmemSize * Compute space needed for replication launcher shared memory @@ -868,7 +948,7 @@ ApplyLauncherMain(Datum main_arg) wait_time = wal_retrieve_retry_interval; logicalrep_worker_launch(sub->dbid, sub->oid, sub->name, - sub->owner, InvalidOid); + sub->owner, InvalidOid, DSM_HANDLE_INVALID); } } diff --git a/src/backend/replication/logical/origin.c b/src/backend/replication/logical/origin.c index 21937ab2d3..50c567fb6e 100644 --- a/src/backend/replication/logical/origin.c +++ b/src/backend/replication/logical/origin.c @@ -1063,12 +1063,21 @@ ReplicationOriginExitCleanup(int code, Datum arg) * array doesn't have to be searched when calling * replorigin_session_advance(). * - * Obviously only one such cached origin can exist per process and the current + * Normally only one such cached origin can exist per process and the current * cached value can only be set again after the previous value is torn down * with replorigin_session_reset(). + * + * However, if the function parameter 'must_acquire' is false, we allow the + * process to use the same slot already acquired by another process. It's safe + * because 1) The only caller (apply background workers) will maintain the + * commit order by allowing only one process to commit at a time, so no two + * workers will be operating on the same origin at the same time (see comments + * in logical/worker.c). 2) Even though we try to advance the session's origin + * concurrently, it's safe to do so as we change/advance the session_origin + * LSNs under replicate_state LWLock. */ void -replorigin_session_setup(RepOriginId node) +replorigin_session_setup(RepOriginId node, bool must_acquire) { static bool registered_cleanup; int i; @@ -1110,7 +1119,7 @@ replorigin_session_setup(RepOriginId node) if (curstate->roident != node) continue; - else if (curstate->acquired_by != 0) + else if (curstate->acquired_by != 0 && must_acquire) { ereport(ERROR, (errcode(ERRCODE_OBJECT_IN_USE), @@ -1141,7 +1150,14 @@ replorigin_session_setup(RepOriginId node) Assert(session_replication_state->roident != InvalidRepOriginId); - session_replication_state->acquired_by = MyProcPid; + if (must_acquire) + session_replication_state->acquired_by = MyProcPid; + else if (session_replication_state->acquired_by == 0) + ereport(ERROR, + (errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED), + errmsg("apply background worker could not find replication state slot for replication origin with OID %u", + node), + errdetail("There is no replication state slot set by its main apply worker."))); LWLockRelease(ReplicationOriginLock); @@ -1321,7 +1337,7 @@ pg_replication_origin_session_setup(PG_FUNCTION_ARGS) name = text_to_cstring((text *) DatumGetPointer(PG_GETARG_DATUM(0))); origin = replorigin_by_name(name, false); - replorigin_session_setup(origin); + replorigin_session_setup(origin, true); replorigin_session_origin = origin; diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c index ff8513e2d2..affd08cfa4 100644 --- a/src/backend/replication/logical/proto.c +++ b/src/backend/replication/logical/proto.c @@ -1163,31 +1163,56 @@ logicalrep_read_stream_commit(StringInfo in, LogicalRepCommitData *commit_data) /* * Write STREAM ABORT to the output stream. Note that xid and subxid will be * same for the top-level transaction abort. + * + * If write_abort_lsn is true, send the abort_lsn and abort_time fields. + * Otherwise not. */ void logicalrep_write_stream_abort(StringInfo out, TransactionId xid, - TransactionId subxid) + ReorderBufferTXN *txn, XLogRecPtr abort_lsn, + bool write_abort_lsn) { pq_sendbyte(out, LOGICAL_REP_MSG_STREAM_ABORT); - Assert(TransactionIdIsValid(xid) && TransactionIdIsValid(subxid)); + Assert(TransactionIdIsValid(xid) && TransactionIdIsValid(txn->xid)); /* transaction ID */ pq_sendint32(out, xid); - pq_sendint32(out, subxid); + pq_sendint32(out, txn->xid); + + if (write_abort_lsn) + { + pq_sendint64(out, abort_lsn); + pq_sendint64(out, txn->xact_time.abort_time); + } } /* * Read STREAM ABORT from the output stream. + * + * If read_abort_lsn is true, try to read the abort_lsn and abort_time fields. + * Otherwise not. */ void -logicalrep_read_stream_abort(StringInfo in, TransactionId *xid, - TransactionId *subxid) +logicalrep_read_stream_abort(StringInfo in, + LogicalRepStreamAbortData *abort_data, + bool read_abort_lsn) { - Assert(xid && subxid); + Assert(abort_data); - *xid = pq_getmsgint(in, 4); - *subxid = pq_getmsgint(in, 4); + abort_data->xid = pq_getmsgint(in, 4); + abort_data->subxid = pq_getmsgint(in, 4); + + if (read_abort_lsn) + { + abort_data->abort_lsn = pq_getmsgint64(in); + abort_data->abort_time = pq_getmsgint64(in); + } + else + { + abort_data->abort_lsn = InvalidXLogRecPtr; + abort_data->abort_time = 0; + } } /* diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c index 88a37fde72..8989328046 100644 --- a/src/backend/replication/logical/reorderbuffer.c +++ b/src/backend/replication/logical/reorderbuffer.c @@ -2826,7 +2826,8 @@ ReorderBufferFinishPrepared(ReorderBuffer *rb, TransactionId xid, * disk. */ void -ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn) +ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn, + TimestampTz abort_time) { ReorderBufferTXN *txn; @@ -2837,6 +2838,8 @@ ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn) if (txn == NULL) return; + txn->xact_time.abort_time = abort_time; + /* For streamed transactions notify the remote node about the abort. */ if (rbtxn_is_streamed(txn)) { @@ -2911,7 +2914,8 @@ ReorderBufferAbortOld(ReorderBuffer *rb, TransactionId oldestRunningXid) * to this xid might re-create the transaction incompletely. */ void -ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn) +ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn, + TimestampTz abort_time) { ReorderBufferTXN *txn; @@ -2922,6 +2926,8 @@ ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn) if (txn == NULL) return; + txn->xact_time.abort_time = abort_time; + /* For streamed transactions notify the remote node about the abort. */ if (rbtxn_is_streamed(txn)) rb->stream_abort(rb, txn, lsn); diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c index 670c6fcada..8ffba7e2e5 100644 --- a/src/backend/replication/logical/tablesync.c +++ b/src/backend/replication/logical/tablesync.c @@ -568,7 +568,8 @@ process_syncing_tables_for_apply(XLogRecPtr current_lsn) MySubscription->oid, MySubscription->name, MyLogicalRepWorker->userid, - rstate->relid); + rstate->relid, + DSM_HANDLE_INVALID); hentry->last_start_time = now; } } @@ -589,6 +590,9 @@ process_syncing_tables_for_apply(XLogRecPtr current_lsn) void process_syncing_tables(XLogRecPtr current_lsn) { + if (MyLogicalRepWorker->subworker) + return; + if (am_tablesync_worker()) process_syncing_tables_for_sync(current_lsn); else @@ -1273,7 +1277,7 @@ LogicalRepSyncTableStart(XLogRecPtr *origin_startpos) * time this tablesync was launched. */ originid = replorigin_by_name(originname, false); - replorigin_session_setup(originid); + replorigin_session_setup(originid, true); replorigin_session_origin = originid; *origin_startpos = replorigin_session_get_progress(false); @@ -1384,7 +1388,7 @@ LogicalRepSyncTableStart(XLogRecPtr *origin_startpos) true /* go backward */ , true /* WAL log */ ); UnlockRelationOid(ReplicationOriginRelationId, RowExclusiveLock); - replorigin_session_setup(originid); + replorigin_session_setup(originid, true); replorigin_session_origin = originid; } else diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c index 38e3b1c1b3..bbd0304a8d 100644 --- a/src/backend/replication/logical/worker.c +++ b/src/backend/replication/logical/worker.c @@ -22,8 +22,28 @@ * STREAMED TRANSACTIONS * --------------------- * Streamed transactions (large transactions exceeding a memory limit on the - * upstream) are not applied immediately, but instead, the data is written - * to temporary files and then applied at once when the final commit arrives. + * upstream) are applied using one of two approaches. + * + * 1) Separate background workers + * + * If streaming = parallel, We assign a new apply background worker (if + * available) as soon as the xact's first stream is received. The main apply + * worker will send changes to this new worker via shared memory. We keep this + * worker assigned till the transaction commit is received and also wait for + * the worker to finish at commit. This preserves commit ordering and avoids + * file I/O in most cases. We still need to spill to a file if there is no + * worker available. It is important to maintain commit order to avoid failures + * due to (a) transaction dependencies, say if we insert a row in the first + * transaction and update it in the second transaction then allowing to apply + * both in parallel can lead to failure in the update. (b) deadlocks, allowing + * transactions that update the same set of rows/tables in opposite order to be + * applied in parallel can lead to deadlocks. + * + * 2) Write to temporary files and apply when the final commit arrives + * + * If no worker is available to handle streamed transaction, the data is + * written to temporary files and then applied at once when the final commit + * arrives. * * Unlike the regular (non-streamed) case, handling streamed transactions has * to handle aborts of both the toplevel transaction and subtransactions. This @@ -219,20 +239,8 @@ typedef struct ApplyExecutionData PartitionTupleRouting *proute; /* partition routing info */ } ApplyExecutionData; -/* Struct for saving and restoring apply errcontext information */ -typedef struct ApplyErrorCallbackArg -{ - LogicalRepMsgType command; /* 0 if invalid */ - LogicalRepRelMapEntry *rel; - - /* Remote node information */ - int remote_attnum; /* -1 if invalid */ - TransactionId remote_xid; - XLogRecPtr finish_lsn; - char *origin_name; -} ApplyErrorCallbackArg; - -static ApplyErrorCallbackArg apply_error_callback_arg = +/* errcontext tracker */ +ApplyErrorCallbackArg apply_error_callback_arg = { .command = 0, .rel = NULL, @@ -242,7 +250,7 @@ static ApplyErrorCallbackArg apply_error_callback_arg = .origin_name = NULL, }; -static MemoryContext ApplyMessageContext = NULL; +MemoryContext ApplyMessageContext = NULL; MemoryContext ApplyContext = NULL; /* per stream context for streaming transactions */ @@ -251,27 +259,38 @@ static MemoryContext LogicalStreamingContext = NULL; WalReceiverConn *LogRepWorkerWalRcvConn = NULL; Subscription *MySubscription = NULL; -static bool MySubscriptionValid = false; +bool MySubscriptionValid = false; bool in_remote_transaction = false; static XLogRecPtr remote_final_lsn = InvalidXLogRecPtr; /* fields valid only when processing streamed transaction */ -static bool in_streamed_transaction = false; +bool in_streamed_transaction = false; + +TransactionId stream_xid = InvalidTransactionId; +static ApplyBgworkerState *stream_apply_worker = NULL; -static TransactionId stream_xid = InvalidTransactionId; +/* Check if we are applying the transaction in an apply background worker */ +#define apply_bgworker_active() (in_streamed_transaction && stream_apply_worker != NULL) + +/* + * The number of changes during one streaming block (only for apply background + * workers) + */ +static uint32 nchanges = 0; /* * We enable skipping all data modification changes (INSERT, UPDATE, etc.) for * the subscription if the remote transaction's finish LSN matches the subskiplsn. * Once we start skipping changes, we don't stop it until we skip all changes of * the transaction even if pg_subscription is updated and MySubscription->skiplsn - * gets changed or reset during that. Also, in streaming transaction cases, we - * don't skip receiving and spooling the changes since we decide whether or not + * gets changed or reset during that. Also, in streaming transaction cases (streaming = on), + * we don't skip receiving and spooling the changes since we decide whether or not * to skip applying the changes when starting to apply changes. The subskiplsn is * cleared after successfully skipping the transaction or applying non-empty * transaction. The latter prevents the mistakenly specified subskiplsn from - * being left. + * being left. Note that we cannot skip the streaming transaction in parallel + * mode, because we cannot get the finish LSN before applying the changes. */ static XLogRecPtr skip_xact_finish_lsn = InvalidXLogRecPtr; #define is_skipping_changes() (unlikely(!XLogRecPtrIsInvalid(skip_xact_finish_lsn))) @@ -324,9 +343,6 @@ static void maybe_reread_subscription(void); static void DisableSubscriptionAndExit(void); -/* prototype needed because of stream_commit */ -static void apply_dispatch(StringInfo s); - static void apply_handle_commit_internal(LogicalRepCommitData *commit_data); static void apply_handle_insert_internal(ApplyExecutionData *edata, ResultRelInfo *relinfo, @@ -359,7 +375,6 @@ static void stop_skipping_changes(void); static void clear_subscription_skip_lsn(XLogRecPtr finish_lsn); /* Functions for apply error callback */ -static void apply_error_callback(void *arg); static inline void set_apply_error_context_xact(TransactionId xid, XLogRecPtr lsn); static inline void reset_apply_error_context_info(void); @@ -426,40 +441,85 @@ end_replication_step(void) } /* - * Handle streamed transactions. + * Handle streamed transactions for both the main apply worker and the apply + * background workers. + * + * In streaming case (receiving a block of streamed transaction), for + * SUBSTREAM_ON mode, simply redirect it to a file for the proper toplevel + * transaction, and for SUBSTREAM_PARALLEL mode, send the changes to apply + * background workers (LOGICAL_REP_MSG_RELATION or LOGICAL_REP_MSG_TYPE changes + * will also be applied in main apply worker). * - * If in streaming mode (receiving a block of streamed transaction), we - * simply redirect it to a file for the proper toplevel transaction. + * For non-streamed transactions, returns false; + * For streamed transactions, returns true if in main apply worker, false + * otherwise. * - * Returns true for streamed transactions, false otherwise (regular mode). + * Exception: When parallel mode is applying streamed transaction in the main + * apply worker, (e.g. when addressing LOGICAL_REP_MSG_RELATION or + * LOGICAL_REP_MSG_TYPE changes), then return false. */ static bool handle_streamed_transaction(LogicalRepMsgType action, StringInfo s) { - TransactionId xid; + TransactionId current_xid = InvalidTransactionId; - /* not in streaming mode */ - if (!in_streamed_transaction) + /* Not in streaming mode */ + if (!(in_streamed_transaction || am_apply_bgworker())) return false; - Assert(stream_fd != NULL); Assert(TransactionIdIsValid(stream_xid)); /* * We should have received XID of the subxact as the first part of the * message, so extract it. */ - xid = pq_getmsgint(s, 4); + current_xid = pq_getmsgint(s, 4); - if (!TransactionIdIsValid(xid)) + if (!TransactionIdIsValid(current_xid)) ereport(ERROR, (errcode(ERRCODE_PROTOCOL_VIOLATION), errmsg_internal("invalid transaction ID in streamed replication transaction"))); - /* Add the new subxact to the array (unless already there). */ - subxact_info_add(xid); + if (am_apply_bgworker()) + { + /* Define a savepoint for a subxact if needed. */ + apply_bgworker_subxact_info_add(current_xid); + + return false; + } + + if (apply_bgworker_active()) + { + /* + * This is the main apply worker, but there is an apply background + * worker, so apply the changes of this transaction in that background + * worker. Pass the data to the worker. + */ + apply_bgworker_send_data(stream_apply_worker, s->len, s->data); + nchanges += 1; + + /* + * XXX The publisher side doesn't always send relation/type update + * messages after the streaming transaction, so also update the + * relation/type in main apply worker here. See function + * cleanup_rel_sync_cache. + */ + if (action == LOGICAL_REP_MSG_RELATION || + action == LOGICAL_REP_MSG_TYPE) + return false; + + return true; + } + + /* + * This is the main apply worker, but there is no apply background worker, + * so write to temporary files and apply when the final commit arrives. + * + * Add the new subxact to the array (unless already there). + */ + subxact_info_add(current_xid); - /* write the change to the current file */ + /* Write the change to the current file */ stream_write_change(action, s); return true; @@ -844,6 +904,9 @@ apply_handle_commit(StringInfo s) apply_handle_commit_internal(&commit_data); + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); + /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(commit_data.end_lsn); @@ -898,7 +961,9 @@ apply_handle_prepare_internal(LogicalRepPreparedTxnData *prepare_data) * BeginTransactionBlock is necessary to balance the EndTransactionBlock * called within the PrepareTransactionBlock below. */ - BeginTransactionBlock(); + if (!IsTransactionBlock()) + BeginTransactionBlock(); + CommitTransactionCommand(); /* Completes the preceding Begin command. */ /* @@ -950,6 +1015,9 @@ apply_handle_prepare(StringInfo s) in_remote_transaction = false; + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); + /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(prepare_data.end_lsn); @@ -1064,10 +1132,6 @@ apply_handle_rollback_prepared(StringInfo s) /* * Handle STREAM PREPARE. - * - * Logic is in two parts: - * 1. Replay all the spooled operations - * 2. Mark the transaction as prepared */ static void apply_handle_stream_prepare(StringInfo s) @@ -1088,24 +1152,76 @@ apply_handle_stream_prepare(StringInfo s) logicalrep_read_stream_prepare(s, &prepare_data); set_apply_error_context_xact(prepare_data.xid, prepare_data.prepare_lsn); - elog(DEBUG1, "received prepare for streamed transaction %u", prepare_data.xid); + if (am_apply_bgworker()) + { + /* Mark the transaction as prepared. */ + apply_handle_prepare_internal(&prepare_data); - /* Replay all the spooled operations. */ - apply_spooled_messages(prepare_data.xid, prepare_data.prepare_lsn); + CommitTransactionCommand(); - /* Mark the transaction as prepared. */ - apply_handle_prepare_internal(&prepare_data); + pgstat_report_stat(false); - CommitTransactionCommand(); + list_free(subxactlist); + subxactlist = NIL; - pgstat_report_stat(false); + apply_bgworker_set_status(APPLY_BGWORKER_FINISHED); + } + else + { + /* This is the main apply worker. */ + ApplyBgworkerState *wstate = apply_bgworker_find(prepare_data.xid); - store_flush_position(prepare_data.end_lsn); + elog(DEBUG1, "received prepare for streamed transaction %u", + prepare_data.xid); + + /* + * Check if we are processing this transaction in an apply background + * worker and if so, send the changes to that worker. + */ + if (wstate) + { + /* + * After sending the data to the apply background worker, wait for + * that worker to finish. This is necessary to maintain commit + * order which avoids failures due to transaction dependencies and + * deadlocks. + */ + apply_bgworker_send_data(wstate, s->len, s->data); + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_FINISHED); + apply_bgworker_free(wstate); + + pgstat_report_stat(false); + store_flush_position(prepare_data.end_lsn); + } + else + { + /* + * The transaction has been serialized to file, so replay all the + * spooled operations. + */ + apply_spooled_messages(prepare_data.xid, prepare_data.prepare_lsn); + + /* Mark the transaction as prepared. */ + apply_handle_prepare_internal(&prepare_data); + + CommitTransactionCommand(); + + pgstat_report_stat(false); + + store_flush_position(prepare_data.end_lsn); + + in_remote_transaction = false; + + /* Unlink the files with serialized changes and subxact info. */ + stream_cleanup_files(MyLogicalRepWorker->subid, prepare_data.xid); + } + } in_remote_transaction = false; + stream_apply_worker = NULL; - /* unlink the files with serialized changes and subxact info. */ - stream_cleanup_files(MyLogicalRepWorker->subid, prepare_data.xid); + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(prepare_data.end_lsn); @@ -1155,15 +1271,6 @@ apply_handle_stream_start(StringInfo s) (errcode(ERRCODE_PROTOCOL_VIOLATION), errmsg_internal("duplicate STREAM START message"))); - /* - * Start a transaction on stream start, this transaction will be committed - * on the stream stop unless it is a tablesync worker in which case it - * will be committed after processing all the messages. We need the - * transaction for handling the buffile, used for serializing the - * streaming data and subxact info. - */ - begin_replication_step(); - /* notify handle methods we're processing a remote transaction */ in_streamed_transaction = true; @@ -1177,36 +1284,93 @@ apply_handle_stream_start(StringInfo s) set_apply_error_context_xact(stream_xid, InvalidXLogRecPtr); - /* - * Initialize the worker's stream_fileset if we haven't yet. This will be - * used for the entire duration of the worker so create it in a permanent - * context. We create this on the very first streaming message from any - * transaction and then use it for this and other streaming transactions. - * Now, we could create a fileset at the start of the worker as well but - * then we won't be sure that it will ever be used. - */ - if (MyLogicalRepWorker->stream_fileset == NULL) + if (am_apply_bgworker()) { - MemoryContext oldctx; - - oldctx = MemoryContextSwitchTo(ApplyContext); + /* + * Make sure the handle apply_dispatch methods are aware we're in a + * remote transaction. + */ + in_remote_transaction = true; - MyLogicalRepWorker->stream_fileset = palloc(sizeof(FileSet)); - FileSetInit(MyLogicalRepWorker->stream_fileset); + /* Begin the transaction. */ + AcceptInvalidationMessages(); + maybe_reread_subscription(); - MemoryContextSwitchTo(oldctx); + StartTransactionCommand(); + BeginTransactionBlock(); + CommitTransactionCommand(); } + else + { + /* + * This is the main apply worker. Check if there is any free apply + * background worker we can use to process this transaction. + */ + if (first_segment) + stream_apply_worker = apply_bgworker_start(stream_xid); + else + stream_apply_worker = apply_bgworker_find(stream_xid); - /* open the spool file for this transaction */ - stream_open_file(MyLogicalRepWorker->subid, stream_xid, first_segment); + if (stream_apply_worker) + { + /* + * If we have found a free worker or if we are already applying this + * transaction in an apply background worker, then we pass the data to + * that worker. + */ + if (first_segment) + apply_bgworker_send_data(stream_apply_worker, s->len, s->data); - /* if this is not the first segment, open existing subxact file */ - if (!first_segment) - subxact_info_read(MyLogicalRepWorker->subid, stream_xid); + nchanges = 0; + elog(DEBUG1, "starting streaming of xid %u", stream_xid); + } + else + { + /* + * Since no apply background worker is available for the first + * stream start, serialize all the changes of the transaction. + * + * Start a transaction on stream start, this transaction will be + * committed on the stream stop unless it is a tablesync worker in + * which case it will be committed after processing all the + * messages. We need the transaction for handling the buffile, + * used for serializing the streaming data and subxact info. + */ + begin_replication_step(); - pgstat_report_activity(STATE_RUNNING, NULL); + /* + * Initialize the worker's stream_fileset if we haven't yet. This will + * be used for the entire duration of the worker so create it in a + * permanent context. We create this on the very first streaming + * message from any transaction and then use it for this and other + * streaming transactions. Now, we could create a fileset at the start + * of the worker as well but then we won't be sure that it will ever + * be used. + */ + if (MyLogicalRepWorker->stream_fileset == NULL) + { + MemoryContext oldctx; - end_replication_step(); + oldctx = MemoryContextSwitchTo(ApplyContext); + + MyLogicalRepWorker->stream_fileset = palloc(sizeof(FileSet)); + FileSetInit(MyLogicalRepWorker->stream_fileset); + + MemoryContextSwitchTo(oldctx); + } + + /* Open the spool file for this transaction. */ + stream_open_file(MyLogicalRepWorker->subid, stream_xid, first_segment); + + /* If this is not the first segment, open existing subxact file. */ + if (!first_segment) + subxact_info_read(MyLogicalRepWorker->subid, stream_xid); + + end_replication_step(); + } + } + + pgstat_report_activity(STATE_RUNNING, NULL); } /* @@ -1220,53 +1384,52 @@ apply_handle_stream_stop(StringInfo s) (errcode(ERRCODE_PROTOCOL_VIOLATION), errmsg_internal("STREAM STOP message without STREAM START"))); - /* - * Close the file with serialized changes, and serialize information about - * subxacts for the toplevel transaction. - */ - subxact_info_write(MyLogicalRepWorker->subid, stream_xid); - stream_close_file(); + if (apply_bgworker_active()) + { + char action = LOGICAL_REP_MSG_STREAM_STOP; - /* We must be in a valid transaction state */ - Assert(IsTransactionState()); + apply_bgworker_send_data(stream_apply_worker, 1, &action); - /* Commit the per-stream transaction */ - CommitTransactionCommand(); + elog(DEBUG1, "stopped streaming of xid %u, %u changes streamed", stream_xid, nchanges); + } + else + { + /* + * Close the file with serialized changes, and serialize information + * about subxacts for the toplevel transaction. + */ + subxact_info_write(MyLogicalRepWorker->subid, stream_xid); + stream_close_file(); - in_streamed_transaction = false; + /* We must be in a valid transaction state */ + Assert(IsTransactionState()); - /* Reset per-stream context */ - MemoryContextReset(LogicalStreamingContext); + /* Commit the per-stream transaction */ + CommitTransactionCommand(); + + /* Reset per-stream context */ + MemoryContextReset(LogicalStreamingContext); + } + + in_streamed_transaction = false; + stream_apply_worker = NULL; pgstat_report_activity(STATE_IDLE, NULL); reset_apply_error_context_info(); } /* - * Handle STREAM abort message. + * Handle STREAM ABORT message when the transaction was spilled to disk. */ static void -apply_handle_stream_abort(StringInfo s) +serialize_stream_abort(TransactionId xid, TransactionId subxid) { - TransactionId xid; - TransactionId subxid; - - if (in_streamed_transaction) - ereport(ERROR, - (errcode(ERRCODE_PROTOCOL_VIOLATION), - errmsg_internal("STREAM ABORT message without STREAM STOP"))); - - logicalrep_read_stream_abort(s, &xid, &subxid); - /* * If the two XIDs are the same, it's in fact abort of toplevel xact, so * just delete the files with serialized info. */ if (xid == subxid) - { - set_apply_error_context_xact(xid, InvalidXLogRecPtr); stream_cleanup_files(MyLogicalRepWorker->subid, xid); - } else { /* @@ -1290,8 +1453,6 @@ apply_handle_stream_abort(StringInfo s) bool found = false; char path[MAXPGPATH]; - set_apply_error_context_xact(subxid, InvalidXLogRecPtr); - subidx = -1; begin_replication_step(); subxact_info_read(MyLogicalRepWorker->subid, xid); @@ -1316,7 +1477,6 @@ apply_handle_stream_abort(StringInfo s) cleanup_subxact_info(); end_replication_step(); CommitTransactionCommand(); - reset_apply_error_context_info(); return; } @@ -1339,6 +1499,142 @@ apply_handle_stream_abort(StringInfo s) end_replication_step(); CommitTransactionCommand(); } +} + +/* + * Handle STREAM ABORT message. + */ +static void +apply_handle_stream_abort(StringInfo s) +{ + TransactionId xid; + TransactionId subxid; + LogicalRepStreamAbortData abort_data; + bool read_abort_lsn = false; + + if (in_streamed_transaction) + ereport(ERROR, + (errcode(ERRCODE_PROTOCOL_VIOLATION), + errmsg_internal("STREAM ABORT message without STREAM STOP"))); + + /* Check whether the publisher sends abort_lsn and abort_time. */ + if (am_apply_bgworker()) + read_abort_lsn = MyParallelState->server_version >= 160000; + + logicalrep_read_stream_abort(s, &abort_data, read_abort_lsn); + + xid = abort_data.xid; + subxid = abort_data.subxid; + + set_apply_error_context_xact(subxid, abort_data.abort_lsn); + + if (am_apply_bgworker()) + { + elog(DEBUG1, "[Apply BGW #%u] aborting current transaction xid=%u, subxid=%u", + MyParallelState->n, GetCurrentTransactionIdIfAny(), + GetCurrentSubTransactionId()); + + /* + * Update origin state so we can restart streaming from correct + * position in case of crash. + */ + if (read_abort_lsn) + { + replorigin_session_origin_lsn = abort_data.abort_lsn; + replorigin_session_origin_timestamp = abort_data.abort_time; + } + + /* + * If the two XIDs are the same, it's in fact abort of toplevel xact, + * so just free the subxactlist. + */ + if (subxid == xid) + { + AbortCurrentTransaction(); + + EndTransactionBlock(false); + CommitTransactionCommand(); + + in_remote_transaction = false; + pgstat_report_activity(STATE_IDLE, NULL); + + list_free(subxactlist); + subxactlist = NIL; + + apply_bgworker_set_status(APPLY_BGWORKER_FINISHED); + } + else + { + /* + * OK, so it's a subxact. Rollback to the savepoint. + * + * We also need to read the subxactlist, determine the offset + * tracked for the subxact, and truncate the list. + */ + int i; + bool found = false; + char spname[MAXPGPATH]; + + snprintf(spname, MAXPGPATH, "savepoint_for_xid_%u", subxid); + + elog(DEBUG1, "[Apply BGW #%u] rolling back to savepoint %s", + MyParallelState->n, spname); + + for (i = list_length(subxactlist) - 1; i >= 0; i--) + { + xid = (TransactionId) list_nth_int(subxactlist, i); + if (xid == subxid) + { + found = true; + break; + } + } + + if (found) + { + RollbackToSavepoint(spname); + CommitTransactionCommand(); + subxactlist = list_truncate(subxactlist, i + 1); + } + + pgstat_report_activity(STATE_IDLEINTRANSACTION, NULL); + } + } + else + { + /* This is the main apply worker. */ + ApplyBgworkerState *wstate = apply_bgworker_find(xid); + + /* + * Check if we are processing this transaction in an apply background + * worker and if so, send the changes to that worker. + */ + if (wstate) + { + /* Send STREAM ABORT message to the apply background worker. */ + apply_bgworker_send_data(wstate, s->len, s->data); + + /* + * After sending the data to the apply background worker, wait for + * that worker to finish. This is necessary to maintain commit + * order which avoids failures due to transaction dependencies and + * deadlocks. + */ + if (subxid == xid) + { + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_FINISHED); + apply_bgworker_free(wstate); + } + } + else + { + /* + * We are in main apply worker and the transaction has been + * serialized to file. + */ + serialize_stream_abort(xid, subxid); + } + } reset_apply_error_context_info(); } @@ -1468,8 +1764,8 @@ apply_spooled_messages(TransactionId xid, XLogRecPtr lsn) static void apply_handle_stream_commit(StringInfo s) { - TransactionId xid; LogicalRepCommitData commit_data; + TransactionId xid; if (in_streamed_transaction) ereport(ERROR, @@ -1479,14 +1775,81 @@ apply_handle_stream_commit(StringInfo s) xid = logicalrep_read_stream_commit(s, &commit_data); set_apply_error_context_xact(xid, commit_data.commit_lsn); - elog(DEBUG1, "received commit for streamed transaction %u", xid); + if (am_apply_bgworker()) + { + /* + * Update origin state so we can restart streaming from correct + * position in case of crash. + */ + replorigin_session_origin_lsn = commit_data.end_lsn; + replorigin_session_origin_timestamp = commit_data.committime; + + CommitTransactionCommand(); + EndTransactionBlock(false); + CommitTransactionCommand(); - apply_spooled_messages(xid, commit_data.commit_lsn); + in_remote_transaction = false; - apply_handle_commit_internal(&commit_data); + pgstat_report_stat(false); + + list_free(subxactlist); + subxactlist = NIL; + + apply_bgworker_set_status(APPLY_BGWORKER_FINISHED); + } + else + { + /* This is the main apply worker. */ + ApplyBgworkerState *wstate = apply_bgworker_find(xid); + + elog(DEBUG1, "received commit for streamed transaction %u", xid); + + /* + * Check if we are processing this transaction in an apply background + * worker and if so, send the changes to that worker. + */ + if (wstate) + { + /* Send STREAM COMMIT message to the apply background worker. */ + apply_bgworker_send_data(wstate, s->len, s->data); + + /* + * After sending the data to the apply background worker, wait for + * that worker to finish. This is necessary to maintain commit + * order which avoids failures due to transaction dependencies and + * deadlocks. + */ + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_FINISHED); - /* unlink the files with serialized changes and subxact info */ - stream_cleanup_files(MyLogicalRepWorker->subid, xid); + pgstat_report_stat(false); + store_flush_position(commit_data.end_lsn); + stop_skipping_changes(); + + apply_bgworker_free(wstate); + + /* + * The transaction is either non-empty or skipped, so we clear the + * subskiplsn. + */ + clear_subscription_skip_lsn(commit_data.commit_lsn); + } + else + { + /* + * The transaction has been serialized to file, so replay all the + * spooled operations. + */ + apply_spooled_messages(xid, commit_data.commit_lsn); + + apply_handle_commit_internal(&commit_data); + + /* Unlink the files with serialized changes and subxact info. */ + stream_cleanup_files(MyLogicalRepWorker->subid, xid); + } + } + + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(commit_data.end_lsn); @@ -2467,7 +2830,7 @@ apply_handle_truncate(StringInfo s) /* * Logical replication protocol message dispatcher. */ -static void +void apply_dispatch(StringInfo s) { LogicalRepMsgType action = pq_getmsgbyte(s); @@ -2636,6 +2999,10 @@ store_flush_position(XLogRecPtr remote_lsn) { FlushPosition *flushpos; + /* We only need to collect the LSN in main apply worker */ + if (am_apply_bgworker()) + return; + /* Need to do this in permanent context */ MemoryContextSwitchTo(ApplyContext); @@ -2650,7 +3017,7 @@ store_flush_position(XLogRecPtr remote_lsn) /* Update statistics of the worker. */ -static void +void UpdateWorkerStats(XLogRecPtr last_lsn, TimestampTz send_time, bool reply) { MyLogicalRepWorker->last_lsn = last_lsn; @@ -2812,6 +3179,9 @@ LogicalRepApplyLoop(XLogRecPtr last_received) AcceptInvalidationMessages(); maybe_reread_subscription(); + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); + /* Process any table synchronization changes. */ process_syncing_tables(last_received); } @@ -3113,7 +3483,7 @@ maybe_reread_subscription(void) /* * Callback from subscription syscache invalidation. */ -static void +void subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue) { MySubscriptionValid = false; @@ -3709,7 +4079,7 @@ ApplyWorkerMain(Datum main_arg) originid = replorigin_by_name(originname, true); if (!OidIsValid(originid)) originid = replorigin_create(originname); - replorigin_session_setup(originid); + replorigin_session_setup(originid, true); replorigin_session_origin = originid; origin_startpos = replorigin_session_get_progress(false); CommitTransactionCommand(); @@ -3750,13 +4120,14 @@ ApplyWorkerMain(Datum main_arg) server_version = walrcv_server_version(LogRepWorkerWalRcvConn); options.proto.logical.proto_version = + server_version >= 160000 ? LOGICALREP_PROTO_STREAM_PARALLEL_VERSION_NUM : server_version >= 150000 ? LOGICALREP_PROTO_TWOPHASE_VERSION_NUM : server_version >= 140000 ? LOGICALREP_PROTO_STREAM_VERSION_NUM : LOGICALREP_PROTO_VERSION_NUM; options.proto.logical.publication_names = MySubscription->publications; options.proto.logical.binary = MySubscription->binary; - options.proto.logical.streaming = MySubscription->stream; + options.proto.logical.streaming = (MySubscription->stream != SUBSTREAM_OFF); options.proto.logical.twophase = false; if (!am_tablesync_worker()) @@ -3914,7 +4285,8 @@ clear_subscription_skip_lsn(XLogRecPtr finish_lsn) XLogRecPtr myskiplsn = MySubscription->skiplsn; bool started_tx = false; - if (likely(XLogRecPtrIsInvalid(myskiplsn))) + if (likely(XLogRecPtrIsInvalid(myskiplsn)) || + am_apply_bgworker()) return; if (!IsTransactionState()) @@ -3986,7 +4358,7 @@ clear_subscription_skip_lsn(XLogRecPtr finish_lsn) } /* Error callback to give more context info about the change being applied */ -static void +void apply_error_callback(void *arg) { ApplyErrorCallbackArg *errarg = &apply_error_callback_arg; @@ -4014,23 +4386,47 @@ apply_error_callback(void *arg) errarg->remote_xid, LSN_FORMAT_ARGS(errarg->finish_lsn)); } - else if (errarg->remote_attnum < 0) - errcontext("processing remote data for replication origin \"%s\" during \"%s\" for replication target relation \"%s.%s\" in transaction %u finished at %X/%X", - errarg->origin_name, - logicalrep_message_type(errarg->command), - errarg->rel->remoterel.nspname, - errarg->rel->remoterel.relname, - errarg->remote_xid, - LSN_FORMAT_ARGS(errarg->finish_lsn)); else - errcontext("processing remote data for replication origin \"%s\" during \"%s\" for replication target relation \"%s.%s\" column \"%s\" in transaction %u finished at %X/%X", - errarg->origin_name, - logicalrep_message_type(errarg->command), - errarg->rel->remoterel.nspname, - errarg->rel->remoterel.relname, - errarg->rel->remoterel.attnames[errarg->remote_attnum], - errarg->remote_xid, - LSN_FORMAT_ARGS(errarg->finish_lsn)); + { + if (errarg->remote_attnum < 0) + { + if (XLogRecPtrIsInvalid(errarg->finish_lsn)) + errcontext("processing remote data for replication origin \"%s\" during \"%s\" for replication target relation \"%s.%s\" in transaction %u", + errarg->origin_name, + logicalrep_message_type(errarg->command), + errarg->rel->remoterel.nspname, + errarg->rel->remoterel.relname, + errarg->remote_xid); + else + errcontext("processing remote data for replication origin \"%s\" during \"%s\" for replication target relation \"%s.%s\" in transaction %u finished at %X/%X", + errarg->origin_name, + logicalrep_message_type(errarg->command), + errarg->rel->remoterel.nspname, + errarg->rel->remoterel.relname, + errarg->remote_xid, + LSN_FORMAT_ARGS(errarg->finish_lsn)); + } + else + { + if (XLogRecPtrIsInvalid(errarg->finish_lsn)) + errcontext("processing remote data for replication origin \"%s\" during \"%s\" for replication target relation \"%s.%s\" column \"%s\" in transaction %u", + errarg->origin_name, + logicalrep_message_type(errarg->command), + errarg->rel->remoterel.nspname, + errarg->rel->remoterel.relname, + errarg->rel->remoterel.attnames[errarg->remote_attnum], + errarg->remote_xid); + else + errcontext("processing remote data for replication origin \"%s\" during \"%s\" for replication target relation \"%s.%s\" column \"%s\" in transaction %u finished at %X/%X", + errarg->origin_name, + logicalrep_message_type(errarg->command), + errarg->rel->remoterel.nspname, + errarg->rel->remoterel.relname, + errarg->rel->remoterel.attnames[errarg->remote_attnum], + errarg->remote_xid, + LSN_FORMAT_ARGS(errarg->finish_lsn)); + } + } } /* Set transaction information of apply error callback */ diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c index 2cbca4a087..1aaca04982 100644 --- a/src/backend/replication/pgoutput/pgoutput.c +++ b/src/backend/replication/pgoutput/pgoutput.c @@ -1820,6 +1820,8 @@ pgoutput_stream_abort(struct LogicalDecodingContext *ctx, XLogRecPtr abort_lsn) { ReorderBufferTXN *toptxn; + bool write_abort_lsn = false; + PGOutputData *data = (PGOutputData *) ctx->output_plugin_private; /* * The abort should happen outside streaming block, even for streamed @@ -1832,8 +1834,13 @@ pgoutput_stream_abort(struct LogicalDecodingContext *ctx, Assert(rbtxn_is_streamed(toptxn)); + /* We only send abort_lsn and abort_time if the subscriber needs them. */ + if (data->protocol_version >= LOGICALREP_PROTO_STREAM_PARALLEL_VERSION_NUM) + write_abort_lsn = true; + OutputPluginPrepareWrite(ctx, true); - logicalrep_write_stream_abort(ctx->out, toptxn->xid, txn->xid); + logicalrep_write_stream_abort(ctx->out, toptxn->xid, txn, abort_lsn, + write_abort_lsn); OutputPluginWrite(ctx, true); cleanup_rel_sync_cache(toptxn->xid, false); diff --git a/src/backend/utils/activity/wait_event.c b/src/backend/utils/activity/wait_event.c index 87c15b9c6f..ba781e6f08 100644 --- a/src/backend/utils/activity/wait_event.c +++ b/src/backend/utils/activity/wait_event.c @@ -388,6 +388,9 @@ pgstat_get_wait_ipc(WaitEventIPC w) case WAIT_EVENT_HASH_GROW_BUCKETS_REINSERT: event_name = "HashGrowBucketsReinsert"; break; + case WAIT_EVENT_LOGICAL_APPLY_WORKER_STATE_CHANGE: + event_name = "LogicalApplyWorkerStateChange"; + break; case WAIT_EVENT_LOGICAL_SYNC_DATA: event_name = "LogicalSyncData"; break; diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c index 0328029d43..4284bcbcd1 100644 --- a/src/backend/utils/misc/guc.c +++ b/src/backend/utils/misc/guc.c @@ -3220,6 +3220,18 @@ static struct config_int ConfigureNamesInt[] = NULL, NULL, NULL }, + { + {"max_apply_bgworkers_per_subscription", + PGC_SIGHUP, + REPLICATION_SUBSCRIBERS, + gettext_noop("Maximum number of apply background workers per subscription."), + NULL, + }, + &max_apply_bgworkers_per_subscription, + 2, 0, MAX_BACKENDS, + NULL, NULL, NULL + }, + { {"log_rotation_age", PGC_SIGHUP, LOGGING_WHERE, gettext_noop("Sets the amount of time to wait before forcing " diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample index b4bc06e5f5..ad18710af4 100644 --- a/src/backend/utils/misc/postgresql.conf.sample +++ b/src/backend/utils/misc/postgresql.conf.sample @@ -360,6 +360,7 @@ #max_logical_replication_workers = 4 # taken from max_worker_processes # (change requires restart) #max_sync_workers_per_subscription = 2 # taken from max_logical_replication_workers +#max_apply_bgworkers_per_subscription = 2 # taken from max_logical_replication_workers #------------------------------------------------------------------------------ diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index f317f0a681..24927641b9 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -4449,7 +4449,7 @@ getSubscriptions(Archive *fout) if (fout->remoteVersion >= 140000) appendPQExpBufferStr(query, " s.substream,\n"); else - appendPQExpBufferStr(query, " false AS substream,\n"); + appendPQExpBufferStr(query, " 'f' AS substream,\n"); if (fout->remoteVersion >= 150000) appendPQExpBufferStr(query, @@ -4579,8 +4579,10 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo) if (strcmp(subinfo->subbinary, "t") == 0) appendPQExpBufferStr(query, ", binary = true"); - if (strcmp(subinfo->substream, "f") != 0) + if (strcmp(subinfo->substream, "t") == 0) appendPQExpBufferStr(query, ", streaming = on"); + else if (strcmp(subinfo->substream, "p") == 0) + appendPQExpBufferStr(query, ", streaming = parallel"); if (strcmp(subinfo->subtwophasestate, two_phase_disabled) != 0) appendPQExpBufferStr(query, ", two_phase = on"); diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h index d1260f590c..d54540f5f5 100644 --- a/src/include/catalog/pg_subscription.h +++ b/src/include/catalog/pg_subscription.h @@ -68,7 +68,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW bool subbinary; /* True if the subscription wants the * publisher to send data in binary */ - bool substream; /* Stream in-progress transactions. */ + char substream; /* Stream in-progress transactions. + * See SUBSTREAM_xxx constants. */ char subtwophasestate; /* Stream two-phase transactions */ @@ -109,7 +110,8 @@ typedef struct Subscription bool enabled; /* Indicates if the subscription is enabled */ bool binary; /* Indicates if the subscription wants data in * binary format */ - bool stream; /* Allow streaming in-progress transactions. */ + char stream; /* Allow streaming in-progress transactions. + * See SUBSTREAM_xxx constants. */ char twophasestate; /* Allow streaming two-phase transactions */ bool disableonerr; /* Indicates if the subscription should be * automatically disabled if a worker error @@ -120,6 +122,21 @@ typedef struct Subscription List *publications; /* List of publication names to subscribe to */ } Subscription; +/* Disallow streaming in-progress transactions */ +#define SUBSTREAM_OFF 'f' + +/* + * Streaming in-progress transactions are written to a temporary file and + * applied only after the transaction is committed on upstream. + */ +#define SUBSTREAM_ON 't' + +/* + * Streaming in-progress transactions are applied immediately via a background + * worker + */ +#define SUBSTREAM_PARALLEL 'p' + extern Subscription *GetSubscription(Oid subid, bool missing_ok); extern void FreeSubscription(Subscription *sub); extern void DisableSubscription(Oid subid); diff --git a/src/include/replication/logicallauncher.h b/src/include/replication/logicallauncher.h index f1e2821e25..ac8ef94381 100644 --- a/src/include/replication/logicallauncher.h +++ b/src/include/replication/logicallauncher.h @@ -14,6 +14,7 @@ extern PGDLLIMPORT int max_logical_replication_workers; extern PGDLLIMPORT int max_sync_workers_per_subscription; +extern PGDLLIMPORT int max_apply_bgworkers_per_subscription; extern void ApplyLauncherRegister(void); extern void ApplyLauncherMain(Datum main_arg); diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h index a771ab8ff3..0f74a3392b 100644 --- a/src/include/replication/logicalproto.h +++ b/src/include/replication/logicalproto.h @@ -32,12 +32,17 @@ * * LOGICALREP_PROTO_TWOPHASE_VERSION_NUM is the minimum protocol version with * support for two-phase commit decoding (at prepare time). Introduced in PG15. + * + * LOGICALREP_PROTO_STREAM_PARALLEL_VERSION_NUM is the minimum protocol version + * with support for streaming large transactions in apply background worker. + * Introduced in PG16. */ #define LOGICALREP_PROTO_MIN_VERSION_NUM 1 #define LOGICALREP_PROTO_VERSION_NUM 1 #define LOGICALREP_PROTO_STREAM_VERSION_NUM 2 #define LOGICALREP_PROTO_TWOPHASE_VERSION_NUM 3 -#define LOGICALREP_PROTO_MAX_VERSION_NUM LOGICALREP_PROTO_TWOPHASE_VERSION_NUM +#define LOGICALREP_PROTO_STREAM_PARALLEL_VERSION_NUM 4 +#define LOGICALREP_PROTO_MAX_VERSION_NUM LOGICALREP_PROTO_STREAM_PARALLEL_VERSION_NUM /* * Logical message types @@ -175,6 +180,17 @@ typedef struct LogicalRepRollbackPreparedTxnData char gid[GIDSIZE]; } LogicalRepRollbackPreparedTxnData; +/* + * Transaction protocol information for stream abort. + */ +typedef struct LogicalRepStreamAbortData +{ + TransactionId xid; + TransactionId subxid; + XLogRecPtr abort_lsn; + TimestampTz abort_time; +} LogicalRepStreamAbortData; + extern void logicalrep_write_begin(StringInfo out, ReorderBufferTXN *txn); extern void logicalrep_read_begin(StringInfo in, LogicalRepBeginData *begin_data); @@ -246,9 +262,12 @@ extern void logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn extern TransactionId logicalrep_read_stream_commit(StringInfo out, LogicalRepCommitData *commit_data); extern void logicalrep_write_stream_abort(StringInfo out, TransactionId xid, - TransactionId subxid); -extern void logicalrep_read_stream_abort(StringInfo in, TransactionId *xid, - TransactionId *subxid); + ReorderBufferTXN *txn, + XLogRecPtr abort_lsn, + bool write_abort_lsn); +extern void logicalrep_read_stream_abort(StringInfo in, + LogicalRepStreamAbortData *abort_data, + bool include_abort_lsn); extern char *logicalrep_message_type(LogicalRepMsgType action); #endif /* LOGICAL_PROTO_H */ diff --git a/src/include/replication/logicalworker.h b/src/include/replication/logicalworker.h index cd1b6e8afc..6a1af7f13c 100644 --- a/src/include/replication/logicalworker.h +++ b/src/include/replication/logicalworker.h @@ -13,6 +13,7 @@ #define LOGICALWORKER_H extern void ApplyWorkerMain(Datum main_arg); +extern void ApplyBgworkerMain(Datum main_arg); extern bool IsLogicalWorker(void); diff --git a/src/include/replication/origin.h b/src/include/replication/origin.h index 2d1b5e5c28..c7389b40a7 100644 --- a/src/include/replication/origin.h +++ b/src/include/replication/origin.h @@ -53,7 +53,7 @@ extern XLogRecPtr replorigin_get_progress(RepOriginId node, bool flush); extern void replorigin_session_advance(XLogRecPtr remote_commit, XLogRecPtr local_commit); -extern void replorigin_session_setup(RepOriginId node); +extern void replorigin_session_setup(RepOriginId node, bool must_acquire); extern void replorigin_session_reset(void); extern XLogRecPtr replorigin_session_get_progress(bool flush); diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h index d109d0baed..d2a80d79e5 100644 --- a/src/include/replication/reorderbuffer.h +++ b/src/include/replication/reorderbuffer.h @@ -301,6 +301,7 @@ typedef struct ReorderBufferTXN { TimestampTz commit_time; TimestampTz prepare_time; + TimestampTz abort_time; } xact_time; /* @@ -647,9 +648,11 @@ extern void ReorderBufferFinishPrepared(ReorderBuffer *rb, TransactionId xid, extern void ReorderBufferAssignChild(ReorderBuffer *, TransactionId, TransactionId, XLogRecPtr commit_lsn); extern void ReorderBufferCommitChild(ReorderBuffer *, TransactionId, TransactionId, XLogRecPtr commit_lsn, XLogRecPtr end_lsn); -extern void ReorderBufferAbort(ReorderBuffer *, TransactionId, XLogRecPtr lsn); +extern void ReorderBufferAbort(ReorderBuffer *, TransactionId, XLogRecPtr lsn, + TimestampTz abort_time); extern void ReorderBufferAbortOld(ReorderBuffer *, TransactionId xid); -extern void ReorderBufferForget(ReorderBuffer *, TransactionId, XLogRecPtr lsn); +extern void ReorderBufferForget(ReorderBuffer *, TransactionId, XLogRecPtr lsn, + TimestampTz abort_time); extern void ReorderBufferInvalidate(ReorderBuffer *, TransactionId, XLogRecPtr lsn); extern void ReorderBufferSetBaseSnapshot(ReorderBuffer *, TransactionId, XLogRecPtr lsn, struct SnapshotData *snap); diff --git a/src/include/replication/worker_internal.h b/src/include/replication/worker_internal.h index 901845abc2..5be8f5755e 100644 --- a/src/include/replication/worker_internal.h +++ b/src/include/replication/worker_internal.h @@ -17,8 +17,11 @@ #include "access/xlogdefs.h" #include "catalog/pg_subscription.h" #include "datatype/timestamp.h" +#include "replication/logicalrelation.h" #include "storage/fileset.h" #include "storage/lock.h" +#include "storage/shm_mq.h" +#include "storage/shm_toc.h" #include "storage/spin.h" @@ -60,6 +63,9 @@ typedef struct LogicalRepWorker */ FileSet *stream_fileset; + /* Indicates if this slot is used for an apply background worker. */ + bool subworker; + /* Stats. */ XLogRecPtr last_lsn; TimestampTz last_send_time; @@ -68,8 +74,68 @@ typedef struct LogicalRepWorker TimestampTz reply_time; } LogicalRepWorker; +/* Struct for saving and restoring apply errcontext information */ +typedef struct ApplyErrorCallbackArg +{ + LogicalRepMsgType command; /* 0 if invalid */ + LogicalRepRelMapEntry *rel; + + /* Remote node information */ + int remote_attnum; /* -1 if invalid */ + TransactionId remote_xid; + XLogRecPtr finish_lsn; + char *origin_name; +} ApplyErrorCallbackArg; + +/* + * Status of apply background worker. + */ +typedef enum ApplyBgworkerStatus +{ + APPLY_BGWORKER_BUSY = 0, /* assigned to a transaction */ + APPLY_BGWORKER_FINISHED, /* transaction is completed */ + APPLY_BGWORKER_EXIT /* exit */ +} ApplyBgworkerStatus; + +/* + * Struct for sharing information between apply main and apply background + * workers. + */ +typedef struct ApplyBgworkerShared +{ + slock_t mutex; + + /* Status of apply background worker. */ + ApplyBgworkerStatus status; + + /* server version of publisher. */ + int server_version; + + TransactionId stream_xid; + uint32 n; /* id of apply background worker */ +} ApplyBgworkerShared; + +/* + * Struct for maintaining an apply background worker. + */ +typedef struct ApplyBgworkerState +{ + shm_mq_handle *mq_handle; + dsm_segment *dsm_seg; + ApplyBgworkerShared volatile *pstate; +} ApplyBgworkerState; + /* Main memory context for apply worker. Permanent during worker lifetime. */ extern PGDLLIMPORT MemoryContext ApplyContext; +extern PGDLLIMPORT MemoryContext ApplyMessageContext; + +extern PGDLLIMPORT ApplyErrorCallbackArg apply_error_callback_arg; + +extern PGDLLIMPORT bool MySubscriptionValid; + +extern PGDLLIMPORT volatile ApplyBgworkerShared *MyParallelState; + +extern PGDLLIMPORT List *subxactlist; /* libpqreceiver connection */ extern PGDLLIMPORT struct WalReceiverConn *LogRepWorkerWalRcvConn; @@ -79,18 +145,22 @@ extern PGDLLIMPORT Subscription *MySubscription; extern PGDLLIMPORT LogicalRepWorker *MyLogicalRepWorker; extern PGDLLIMPORT bool in_remote_transaction; +extern PGDLLIMPORT bool in_streamed_transaction; +extern PGDLLIMPORT TransactionId stream_xid; extern void logicalrep_worker_attach(int slot); extern LogicalRepWorker *logicalrep_worker_find(Oid subid, Oid relid, bool only_running); extern List *logicalrep_workers_find(Oid subid, bool only_running); -extern void logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, - Oid userid, Oid relid); +extern bool logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, + Oid userid, Oid relid, + dsm_handle subworker_dsm); extern void logicalrep_worker_stop(Oid subid, Oid relid); extern void logicalrep_worker_wakeup(Oid subid, Oid relid); extern void logicalrep_worker_wakeup_ptr(LogicalRepWorker *worker); extern int logicalrep_sync_worker_count(Oid subid); +extern int logicalrep_apply_bgworker_count(Oid subid); extern void ReplicationOriginNameForTablesync(Oid suboid, Oid relid, char *originname, int szorgname); @@ -103,10 +173,38 @@ extern void process_syncing_tables(XLogRecPtr current_lsn); extern void invalidate_syncing_table_states(Datum arg, int cacheid, uint32 hashvalue); +extern void UpdateWorkerStats(XLogRecPtr last_lsn, TimestampTz send_time, + bool reply); + +extern void apply_dispatch(StringInfo s); + +/* Function for apply error callback */ +extern void apply_error_callback(void *arg); + +extern void subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue); + +/* Apply background worker setup and interactions */ +extern ApplyBgworkerState *apply_bgworker_start(TransactionId xid); +extern ApplyBgworkerState *apply_bgworker_find(TransactionId xid); +extern void apply_bgworker_wait_for(ApplyBgworkerState *wstate, + ApplyBgworkerStatus wait_for_status); +extern void apply_bgworker_send_data(ApplyBgworkerState *wstate, Size nbytes, + const void *data); +extern void apply_bgworker_free(ApplyBgworkerState *wstate); +extern void apply_bgworker_check_status(void); +extern void apply_bgworker_set_status(ApplyBgworkerStatus status); +extern void apply_bgworker_subxact_info_add(TransactionId current_xid); + static inline bool am_tablesync_worker(void) { return OidIsValid(MyLogicalRepWorker->relid); } +static inline bool +am_apply_bgworker(void) +{ + return MyLogicalRepWorker->subworker; +} + #endif /* WORKER_INTERNAL_H */ diff --git a/src/include/utils/wait_event.h b/src/include/utils/wait_event.h index b578e2ec75..c2d2a114d7 100644 --- a/src/include/utils/wait_event.h +++ b/src/include/utils/wait_event.h @@ -105,6 +105,7 @@ typedef enum WAIT_EVENT_HASH_GROW_BUCKETS_ALLOCATE, WAIT_EVENT_HASH_GROW_BUCKETS_ELECT, WAIT_EVENT_HASH_GROW_BUCKETS_REINSERT, + WAIT_EVENT_LOGICAL_APPLY_WORKER_STATE_CHANGE, WAIT_EVENT_LOGICAL_SYNC_DATA, WAIT_EVENT_LOGICAL_SYNC_STATE_CHANGE, WAIT_EVENT_MQ_INTERNAL, diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out index 5db7146e06..919266ae06 100644 --- a/src/test/regress/expected/subscription.out +++ b/src/test/regress/expected/subscription.out @@ -197,7 +197,7 @@ ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); DROP SUBSCRIPTION regress_testsub; -- fail - streaming must be boolean CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = foo); -ERROR: streaming requires a Boolean value +ERROR: streaming requires a Boolean value or "parallel" -- now it works CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = true); WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 34a76ceb60..4137dc77b4 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -116,6 +116,10 @@ Append AppendPath AppendRelInfo AppendState +ApplyBgworkerEntry +ApplyBgworkerShared +ApplyBgworkerState +ApplyBgworkerStatus ApplyErrorCallbackArg ApplyExecutionData ApplySubXactData @@ -1485,6 +1489,7 @@ LogicalRepRelId LogicalRepRelMapEntry LogicalRepRelation LogicalRepRollbackPreparedTxnData +LogicalRepStreamAbortData LogicalRepTupleData LogicalRepTyp LogicalRepWorker -- 2.23.0.windows.1 [application/octet-stream] v16-0002-Test-streaming-parallel-option-in-tap-test.patch (69.1K, ../../OS3PR01MB627594D75E870BBB45E2E80A9E839@OS3PR01MB6275.jpnprd01.prod.outlook.com/3-v16-0002-Test-streaming-parallel-option-in-tap-test.patch) download | inline diff: From 882c87682d3a149ed27f4141c283f4262bc790f3 Mon Sep 17 00:00:00 2001 From: "shiy.fnst" <[email protected]> Date: Fri, 13 May 2022 14:50:30 +0800 Subject: [PATCH v16 2/4] Test streaming parallel option in tap test Change all TAP tests using the SUBSCRIPTION "streaming" parameter, so they now test both 'on' and 'parallel' values. --- src/test/subscription/t/015_stream.pl | 199 ++++--- src/test/subscription/t/016_stream_subxact.pl | 119 +++-- src/test/subscription/t/017_stream_ddl.pl | 188 ++++--- .../t/018_stream_subxact_abort.pl | 195 ++++--- .../t/019_stream_subxact_ddl_abort.pl | 110 +++- .../subscription/t/022_twophase_cascade.pl | 363 +++++++------ .../subscription/t/023_twophase_stream.pl | 498 ++++++++++-------- 7 files changed, 1035 insertions(+), 637 deletions(-) diff --git a/src/test/subscription/t/015_stream.pl b/src/test/subscription/t/015_stream.pl index 6561b189de..0bdd234935 100644 --- a/src/test/subscription/t/015_stream.pl +++ b/src/test/subscription/t/015_stream.pl @@ -8,6 +8,116 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" +# parameter so the same code can be run both for the streaming=on and +# streaming=parallel cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_parallel) = @_; + + # Interleave a pair of transactions, each exceeding the 64kB limit. + my $in = ''; + my $out = ''; + + my $timer = IPC::Run::timeout($PostgreSQL::Test::Utils::timeout_default); + + my $h = $node_publisher->background_psql('postgres', \$in, \$out, $timer, + on_error_stop => 0); + + # Check that a background worker starts if "streaming" parameter is + # specified as "parallel". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_parallel) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + $in .= q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + }; + $h->pump_nb; + + if ($is_parallel) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001, 9999) s(i); + DELETE FROM test_tab WHERE a > 5000; + COMMIT; + }); + + $in .= q{ + COMMIT; + \q + }; + $h->finish; # errors make the next test fail, so ignore them here + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), + 'check extra columns contain local defaults'); + + # Test the streaming in binary mode + $node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET (binary = on)"); + + # Insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001, 10000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + COMMIT; + }); + + $node_publisher->wait_for_catchup($appname); + + $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(6667|6667|6667), + 'check extra columns contain local defaults'); + + # Change the local values of the extra columns on the subscriber, + # update publisher, and check that subscriber retains the expected + # values. This is to ensure that non-streaming transactions behave + # properly after a streaming transaction. + $node_subscriber->safe_psql('postgres', + "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'" + ); + $node_publisher->safe_psql('postgres', + "UPDATE test_tab SET b = md5(a::text)"); + + $node_publisher->wait_for_catchup($appname); + + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab" + ); + is($result, qq(6667|6667|6667), + 'check extra columns contain locally changed data'); + + # Cleanup the test data + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE (a > 2)"); + $node_publisher->wait_for_catchup($appname); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -37,6 +147,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -54,82 +168,25 @@ my $result = "SELECT count(*), count(c), count(d = 999) FROM test_tab"); is($result, qq(2|2|2), 'check initial data was copied to subscriber'); -# Interleave a pair of transactions, each exceeding the 64kB limit. -my $in = ''; -my $out = ''; - -my $timer = IPC::Run::timeout($PostgreSQL::Test::Utils::timeout_default); - -my $h = $node_publisher->background_psql('postgres', \$in, \$out, $timer, - on_error_stop => 0); - -$in .= q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -}; -$h->pump_nb; - -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001, 9999) s(i); -DELETE FROM test_tab WHERE a > 5000; -COMMIT; -}); - -$in .= q{ -COMMIT; -\q -}; -$h->finish; # errors make the next test fail, so ignore them here - -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), 'check extra columns contain local defaults'); - -# Test the streaming in binary mode -$node_subscriber->safe_psql('postgres', - "ALTER SUBSCRIPTION tap_sub SET (binary = on)"); - -# Insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001, 10000) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -COMMIT; -}); - -$node_publisher->wait_for_catchup($appname); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(6667|6667|6667), 'check extra columns contain local defaults'); +###################################### +# Test using streaming mode 'parallel' +###################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" +); -# Change the local values of the extra columns on the subscriber, -# update publisher, and check that subscriber retains the expected -# values. This is to ensure that non-streaming transactions behave -# properly after a streaming transaction. $node_subscriber->safe_psql('postgres', - "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'" -); -$node_publisher->safe_psql('postgres', - "UPDATE test_tab SET b = md5(a::text)"); + "ALTER SUBSCRIPTION tap_sub SET(streaming = parallel, binary = off)"); -$node_publisher->wait_for_catchup($appname); +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab" -); -is($result, qq(6667|6667|6667), - 'check extra columns contain locally changed data'); +test_streaming($node_publisher, $node_subscriber, $appname, 1); $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/016_stream_subxact.pl b/src/test/subscription/t/016_stream_subxact.pl index f27f1694f2..45429dddba 100644 --- a/src/test/subscription/t/016_stream_subxact.pl +++ b/src/test/subscription/t/016_stream_subxact.pl @@ -8,6 +8,72 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" +# parameter so the same code can be run both for the streaming=on and +# streaming=parallel cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_parallel) = @_; + + # Check that a background worker starts if "streaming" parameter is + # specified as "parallel". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_parallel) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + # Insert, update and delete enough rows to exceed 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series( 3, 500) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501, 1000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001, 1500) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + SAVEPOINT s3; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501, 2000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + SAVEPOINT s4; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001, 2500) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + COMMIT; + }); + + if ($is_parallel) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(1667|1667|1667), + 'check data was copied to subscriber in streaming mode and extra columns contain local defaults' + ); + + # Cleanup the test data + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE (a > 2)"); + $node_publisher->wait_for_catchup($appname); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -37,6 +103,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -54,41 +124,26 @@ my $result = "SELECT count(*), count(c), count(d = 999) FROM test_tab"); is($result, qq(2|2|2), 'check initial data was copied to subscriber'); -# Insert, update and delete enough rows to exceed 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series( 3, 500) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501, 1000) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001, 1500) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -SAVEPOINT s3; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501, 2000) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -SAVEPOINT s4; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001, 2500) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -COMMIT; -}); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(1667|1667|1667), - 'check data was copied to subscriber in streaming mode and extra columns contain local defaults' +###################################### +# Test using streaming mode 'parallel' +###################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" ); +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = parallel)"); + +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; + +test_streaming($node_publisher, $node_subscriber, $appname, 1); + $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/017_stream_ddl.pl b/src/test/subscription/t/017_stream_ddl.pl index 0bce63b716..52dfef4780 100644 --- a/src/test/subscription/t/017_stream_ddl.pl +++ b/src/test/subscription/t/017_stream_ddl.pl @@ -8,6 +8,111 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" +# parameter so the same code can be run both for the streaming=on and +# streaming=parallel cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_parallel) = @_; + + # a small (non-streamed) transaction with DDL and DML + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab VALUES (3, md5(3::text)); + ALTER TABLE test_tab ADD COLUMN c INT; + SAVEPOINT s1; + INSERT INTO test_tab VALUES (4, md5(4::text), -4); + COMMIT; + }); + + # Check that a background worker starts if "streaming" parameter is + # specified as "parallel". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_parallel) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + # large (streamed) transaction with DDL and DML + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(5, 1000) s(i); + ALTER TABLE test_tab ADD COLUMN d INT; + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001, 2000) s(i); + COMMIT; + }); + + if ($is_parallel) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + # a small (non-streamed) transaction with DDL and DML + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab VALUES (2001, md5(2001::text), -2001, 2*2001); + ALTER TABLE test_tab ADD COLUMN e INT; + SAVEPOINT s1; + INSERT INTO test_tab VALUES (2002, md5(2002::text), -2002, 2*2002, -3*2002); + COMMIT; + }); + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d), count(e) FROM test_tab"); + is($result, qq(2002|1999|1002|1), + 'check data was copied to subscriber in streaming mode and extra columns contain local defaults' + ); + + # A large (streamed) transaction with DDL and DML. One of the DDL is performed + # after DML to ensure that we invalidate the schema sent for test_tab so that + # the next transaction has to send the schema again. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(2003,5000) s(i); + ALTER TABLE test_tab ADD COLUMN f INT; + COMMIT; + }); + + # A small transaction that won't get streamed. This is just to ensure that we + # send the schema again to reflect the last column added in the previous test. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i, 4*i FROM generate_series(5001,5005) s(i); + COMMIT; + }); + + $node_publisher->wait_for_catchup($appname); + + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d), count(e), count(f) FROM test_tab" + ); + is($result, qq(5005|5002|4005|3004|5), + 'check data was copied to subscriber for both streaming and non-streaming transactions' + ); + + # Cleanup the test data + $node_publisher->safe_psql( + 'postgres', q{ + DELETE FROM test_tab WHERE (a > 2); + ALTER TABLE test_tab DROP COLUMN c, DROP COLUMN d, DROP COLUMN e, DROP COLUMN f; + }); + $node_publisher->wait_for_catchup($appname); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -37,6 +142,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -54,76 +163,25 @@ my $result = "SELECT count(*), count(c), count(d = 999) FROM test_tab"); is($result, qq(2|0|0), 'check initial data was copied to subscriber'); -# a small (non-streamed) transaction with DDL and DML -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab VALUES (3, md5(3::text)); -ALTER TABLE test_tab ADD COLUMN c INT; -SAVEPOINT s1; -INSERT INTO test_tab VALUES (4, md5(4::text), -4); -COMMIT; -}); - -# large (streamed) transaction with DDL and DML -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(5, 1000) s(i); -ALTER TABLE test_tab ADD COLUMN d INT; -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001, 2000) s(i); -COMMIT; -}); - -# a small (non-streamed) transaction with DDL and DML -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab VALUES (2001, md5(2001::text), -2001, 2*2001); -ALTER TABLE test_tab ADD COLUMN e INT; -SAVEPOINT s1; -INSERT INTO test_tab VALUES (2002, md5(2002::text), -2002, 2*2002, -3*2002); -COMMIT; -}); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d), count(e) FROM test_tab"); -is($result, qq(2002|1999|1002|1), - 'check data was copied to subscriber in streaming mode and extra columns contain local defaults' +###################################### +# Test using streaming mode 'parallel' +###################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" ); -# A large (streamed) transaction with DDL and DML. One of the DDL is performed -# after DML to ensure that we invalidate the schema sent for test_tab so that -# the next transaction has to send the schema again. -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(2003,5000) s(i); -ALTER TABLE test_tab ADD COLUMN f INT; -COMMIT; -}); - -# A small transaction that won't get streamed. This is just to ensure that we -# send the schema again to reflect the last column added in the previous test. -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i, 4*i FROM generate_series(5001,5005) s(i); -COMMIT; -}); +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = parallel)"); -$node_publisher->wait_for_catchup($appname); +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d), count(e), count(f) FROM test_tab"); -is($result, qq(5005|5002|4005|3004|5), - 'check data was copied to subscriber for both streaming and non-streaming transactions' -); +test_streaming($node_publisher, $node_subscriber, $appname, 1); $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/018_stream_subxact_abort.pl b/src/test/subscription/t/018_stream_subxact_abort.pl index 7155442e76..68f0e4b0d1 100644 --- a/src/test/subscription/t/018_stream_subxact_abort.pl +++ b/src/test/subscription/t/018_stream_subxact_abort.pl @@ -8,6 +8,113 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" +# parameter so the same code can be run both for the streaming=on and +# streaming=parallel cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_parallel) = @_; + + # Check that a background worker starts if "streaming" parameter is + # specified as "parallel". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_parallel) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + # large (streamed) transaction with DDL, DML and ROLLBACKs + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i); + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501,1000) s(i); + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001,1500) s(i); + SAVEPOINT s3; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501,2000) s(i); + ROLLBACK TO s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001,2500) s(i); + ROLLBACK TO s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2501,3000) s(i); + SAVEPOINT s4; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3001,3500) s(i); + SAVEPOINT s5; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3501,4000) s(i); + COMMIT; + }); + + if ($is_parallel) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c) FROM test_tab"); + is($result, qq(2000|0), + 'check rollback to savepoint was reflected on subscriber and extra columns contain local defaults' + ); + + # large (streamed) transaction with subscriber receiving out of order + # subtransaction ROLLBACKs + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(4001,4500) s(i); + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001,5500) s(i); + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(6001,6500) s(i); + SAVEPOINT s3; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(7001,7500) s(i); + RELEASE s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(8001,8500) s(i); + ROLLBACK TO s1; + COMMIT; + }); + + $node_publisher->wait_for_catchup($appname); + + $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c) FROM test_tab"); + is($result, qq(2500|0), + 'check rollback to savepoint was reflected on subscriber'); + + # large (streamed) transaction with subscriber receiving rollback + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(8501,9000) s(i); + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(9001,9500) s(i); + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(9501,10000) s(i); + ROLLBACK; + }); + + $node_publisher->wait_for_catchup($appname); + + $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c) FROM test_tab"); + is($result, qq(2500|0), 'check rollback was reflected on subscriber'); + + # Cleanup the test data + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE (a > 2)"); + $node_publisher->wait_for_catchup($appname); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -36,6 +143,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -53,81 +164,25 @@ my $result = "SELECT count(*), count(c) FROM test_tab"); is($result, qq(2|0), 'check initial data was copied to subscriber'); -# large (streamed) transaction with DDL, DML and ROLLBACKs -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i); -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501,1000) s(i); -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001,1500) s(i); -SAVEPOINT s3; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501,2000) s(i); -ROLLBACK TO s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001,2500) s(i); -ROLLBACK TO s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2501,3000) s(i); -SAVEPOINT s4; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3001,3500) s(i); -SAVEPOINT s5; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3501,4000) s(i); -COMMIT; -}); - -$node_publisher->wait_for_catchup($appname); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c) FROM test_tab"); -is($result, qq(2000|0), - 'check rollback to savepoint was reflected on subscriber and extra columns contain local defaults' +###################################### +# Test using streaming mode 'parallel' +###################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" ); -# large (streamed) transaction with subscriber receiving out of order -# subtransaction ROLLBACKs -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(4001,4500) s(i); -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001,5500) s(i); -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(6001,6500) s(i); -SAVEPOINT s3; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(7001,7500) s(i); -RELEASE s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(8001,8500) s(i); -ROLLBACK TO s1; -COMMIT; -}); - -$node_publisher->wait_for_catchup($appname); +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = parallel)"); -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c) FROM test_tab"); -is($result, qq(2500|0), - 'check rollback to savepoint was reflected on subscriber'); - -# large (streamed) transaction with subscriber receiving rollback -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(8501,9000) s(i); -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(9001,9500) s(i); -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(9501,10000) s(i); -ROLLBACK; -}); +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c) FROM test_tab"); -is($result, qq(2500|0), 'check rollback was reflected on subscriber'); +test_streaming($node_publisher, $node_subscriber, $appname, 1); $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/019_stream_subxact_ddl_abort.pl b/src/test/subscription/t/019_stream_subxact_ddl_abort.pl index dbd0fca4d1..b276063721 100644 --- a/src/test/subscription/t/019_stream_subxact_ddl_abort.pl +++ b/src/test/subscription/t/019_stream_subxact_ddl_abort.pl @@ -9,6 +9,69 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" +# parameter so the same code can be run both for the streaming=on and +# streaming=parallel cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_parallel) = @_; + + # Check that a background worker starts if "streaming" parameter is + # specified as "parallel". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_parallel) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + # large (streamed) transaction with DDL, DML and ROLLBACKs + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i); + ALTER TABLE test_tab ADD COLUMN c INT; + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(501,1000) s(i); + ALTER TABLE test_tab ADD COLUMN d INT; + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001,1500) s(i); + ALTER TABLE test_tab ADD COLUMN e INT; + SAVEPOINT s3; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(1501,2000) s(i); + ALTER TABLE test_tab DROP COLUMN c; + ROLLBACK TO s1; + INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(501,1000) s(i); + COMMIT; + }); + + if ($is_parallel) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c) FROM test_tab"); + is($result, qq(1000|500), + 'check rollback to savepoint was reflected on subscriber and extra columns contain local defaults' + ); + + # Cleanup the test data + $node_publisher->safe_psql( + 'postgres', q{ + DELETE FROM test_tab WHERE (a > 2); + ALTER TABLE test_tab DROP COLUMN c; + }); + $node_publisher->wait_for_catchup($appname); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -37,6 +100,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -54,35 +121,26 @@ my $result = "SELECT count(*), count(c) FROM test_tab"); is($result, qq(2|0), 'check initial data was copied to subscriber'); -# large (streamed) transaction with DDL, DML and ROLLBACKs -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i); -ALTER TABLE test_tab ADD COLUMN c INT; -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(501,1000) s(i); -ALTER TABLE test_tab ADD COLUMN d INT; -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001,1500) s(i); -ALTER TABLE test_tab ADD COLUMN e INT; -SAVEPOINT s3; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(1501,2000) s(i); -ALTER TABLE test_tab DROP COLUMN c; -ROLLBACK TO s1; -INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(501,1000) s(i); -COMMIT; -}); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c) FROM test_tab"); -is($result, qq(1000|500), - 'check rollback to savepoint was reflected on subscriber and extra columns contain local defaults' +###################################### +# Test using streaming mode 'parallel' +###################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" ); +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = parallel)"); + +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; + +test_streaming($node_publisher, $node_subscriber, $appname, 1); + $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/022_twophase_cascade.pl b/src/test/subscription/t/022_twophase_cascade.pl index 7a797f37ba..0a4152d3be 100644 --- a/src/test/subscription/t/022_twophase_cascade.pl +++ b/src/test/subscription/t/022_twophase_cascade.pl @@ -11,6 +11,208 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" parameter +# so the same code can be run both for the streaming=on and streaming=parallel +# cases. +sub test_streaming +{ + my ($node_A, $node_B, $node_C, $appname_B, $appname_C, $streaming_mode) = + @_; + + my $oldpid_B = $node_A->safe_psql( + 'postgres', " + SELECT pid FROM pg_stat_replication + WHERE application_name = '$appname_B' AND state = 'streaming';"); + my $oldpid_C = $node_B->safe_psql( + 'postgres', " + SELECT pid FROM pg_stat_replication + WHERE application_name = '$appname_C' AND state = 'streaming';"); + + # Setup logical replication streaming mode + + $node_B->safe_psql( + 'postgres', " + ALTER SUBSCRIPTION tap_sub_B + SET (streaming = $streaming_mode);"); + $node_C->safe_psql( + 'postgres', " + ALTER SUBSCRIPTION tap_sub_C + SET (streaming = $streaming_mode)"); + + # Wait for subscribers to finish initialization + + $node_A->poll_query_until( + 'postgres', " + SELECT pid != $oldpid_B FROM pg_stat_replication + WHERE application_name = '$appname_B' AND state = 'streaming';" + ) or die "Timed out while waiting for apply to restart"; + $node_B->poll_query_until( + 'postgres', " + SELECT pid != $oldpid_C FROM pg_stat_replication + WHERE application_name = '$appname_C' AND state = 'streaming';" + ) or die "Timed out while waiting for apply to restart"; + + ############################### + # Test 2PC PREPARE / COMMIT PREPARED. + # 1. Data is streamed as a 2PC transaction. + # 2. Then do commit prepared. + # + # Expect all data is replicated on subscriber(s) after the commit. + ############################### + + # Check that a background worker starts if "streaming" optparameterion is + # specified as "parallel". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($streaming_mode eq 'parallel') + { + $node_B->append_conf('postgresql.conf', "log_min_messages = debug1"); + $node_B->reload; + + $node_C->append_conf('postgresql.conf', "log_min_messages = debug1"); + $node_C->reload; + } + + # Insert, update and delete enough rows to exceed the 64kB limit. + # Then 2PC PREPARE + $node_A->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + if ($streaming_mode eq 'parallel') + { + $node_B->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_B->append_conf('postgresql.conf', "log_min_messages = warning"); + $node_B->reload; + + $node_C->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_C->append_conf('postgresql.conf', "log_min_messages = warning"); + $node_C->reload; + } + + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); + + # check the transaction state is prepared on subscriber(s) + my $result = + $node_B->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber B'); + $result = + $node_C->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber C'); + + # 2PC COMMIT + $node_A->safe_psql('postgres', "COMMIT PREPARED 'test_prepared_tab';"); + + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); + + # check that transaction was committed on subscriber(s) + $result = $node_B->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), + 'Rows inserted by 2PC have committed on subscriber B, and extra columns have local defaults' + ); + $result = $node_C->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), + 'Rows inserted by 2PC have committed on subscriber C, and extra columns have local defaults' + ); + + # check the transaction state is ended on subscriber(s) + $result = + $node_B->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is committed on subscriber B'); + $result = + $node_C->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is committed on subscriber C'); + + ############################### + # Test 2PC PREPARE with a nested ROLLBACK TO SAVEPOINT. + # 0. Cleanup from previous test leaving only 2 rows. + # 1. Insert one more row. + # 2. Record a SAVEPOINT. + # 3. Data is streamed using 2PC. + # 4. Do rollback to SAVEPOINT prior to the streamed inserts. + # 5. Then COMMIT PREPARED. + # + # Expect data after the SAVEPOINT is aborted leaving only 3 rows (= 2 original + 1 from step 1). + ############################### + + # First, delete the data except for 2 rows (delete will be replicated) + $node_A->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); + + # 2PC PREPARE with a nested ROLLBACK TO SAVEPOINT + $node_A->safe_psql( + 'postgres', " + BEGIN; + INSERT INTO test_tab VALUES (9999, 'foobar'); + SAVEPOINT sp_inner; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + ROLLBACK TO SAVEPOINT sp_inner; + PREPARE TRANSACTION 'outer'; + "); + + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); + + # check the transaction state prepared on subscriber(s) + $result = + $node_B->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber B'); + $result = + $node_C->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber C'); + + # 2PC COMMIT + $node_A->safe_psql('postgres', "COMMIT PREPARED 'outer';"); + + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); + + # check the transaction state is ended on subscriber + $result = + $node_B->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is ended on subscriber B'); + $result = + $node_C->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is ended on subscriber C'); + + # check inserts are visible at subscriber(s). + # All the streamed data (prior to the SAVEPOINT) should be rolled back. + # (9999, 'foobar') should be committed. + $result = $node_B->safe_psql('postgres', + "SELECT count(*) FROM test_tab where b = 'foobar';"); + is($result, qq(1), 'Rows committed are present on subscriber B'); + $result = + $node_B->safe_psql('postgres', "SELECT count(*) FROM test_tab;"); + is($result, qq(3), 'Rows committed are present on subscriber B'); + $result = $node_C->safe_psql('postgres', + "SELECT count(*) FROM test_tab where b = 'foobar';"); + is($result, qq(1), 'Rows committed are present on subscriber C'); + $result = + $node_C->safe_psql('postgres', "SELECT count(*) FROM test_tab;"); + is($result, qq(3), 'Rows committed are present on subscriber C'); + + # Cleanup the test data + $node_A->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); +} + ############################### # Setup a cascade of pub/sub nodes. # node_A -> node_B -> node_C @@ -260,160 +462,15 @@ is($result, qq(21), 'Rows committed are present on subscriber C'); # 2PC + STREAMING TESTS # --------------------- -my $oldpid_B = $node_A->safe_psql( - 'postgres', " - SELECT pid FROM pg_stat_replication - WHERE application_name = '$appname_B' AND state = 'streaming';"); -my $oldpid_C = $node_B->safe_psql( - 'postgres', " - SELECT pid FROM pg_stat_replication - WHERE application_name = '$appname_C' AND state = 'streaming';"); - -# Setup logical replication (streaming = on) - -$node_B->safe_psql( - 'postgres', " - ALTER SUBSCRIPTION tap_sub_B - SET (streaming = on);"); -$node_C->safe_psql( - 'postgres', " - ALTER SUBSCRIPTION tap_sub_C - SET (streaming = on)"); - -# Wait for subscribers to finish initialization - -$node_A->poll_query_until( - 'postgres', " - SELECT pid != $oldpid_B FROM pg_stat_replication - WHERE application_name = '$appname_B' AND state = 'streaming';" -) or die "Timed out while waiting for apply to restart"; -$node_B->poll_query_until( - 'postgres', " - SELECT pid != $oldpid_C FROM pg_stat_replication - WHERE application_name = '$appname_C' AND state = 'streaming';" -) or die "Timed out while waiting for apply to restart"; - -############################### -# Test 2PC PREPARE / COMMIT PREPARED. -# 1. Data is streamed as a 2PC transaction. -# 2. Then do commit prepared. -# -# Expect all data is replicated on subscriber(s) after the commit. -############################### - -# Insert, update and delete enough rows to exceed the 64kB limit. -# Then 2PC PREPARE -$node_A->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_A->wait_for_catchup($appname_B); -$node_B->wait_for_catchup($appname_C); +################################ +# Test using streaming mode 'on' +################################ +test_streaming($node_A, $node_B, $node_C, $appname_B, $appname_C, 'on'); -# check the transaction state is prepared on subscriber(s) -$result = - $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber B'); -$result = - $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber C'); - -# 2PC COMMIT -$node_A->safe_psql('postgres', "COMMIT PREPARED 'test_prepared_tab';"); - -$node_A->wait_for_catchup($appname_B); -$node_B->wait_for_catchup($appname_C); - -# check that transaction was committed on subscriber(s) -$result = $node_B->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), - 'Rows inserted by 2PC have committed on subscriber B, and extra columns have local defaults' -); -$result = $node_C->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), - 'Rows inserted by 2PC have committed on subscriber C, and extra columns have local defaults' -); - -# check the transaction state is ended on subscriber(s) -$result = - $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is committed on subscriber B'); -$result = - $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is committed on subscriber C'); - -############################### -# Test 2PC PREPARE with a nested ROLLBACK TO SAVEPOINT. -# 0. Cleanup from previous test leaving only 2 rows. -# 1. Insert one more row. -# 2. Record a SAVEPOINT. -# 3. Data is streamed using 2PC. -# 4. Do rollback to SAVEPOINT prior to the streamed inserts. -# 5. Then COMMIT PREPARED. -# -# Expect data after the SAVEPOINT is aborted leaving only 3 rows (= 2 original + 1 from step 1). -############################### - -# First, delete the data except for 2 rows (delete will be replicated) -$node_A->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); - -# 2PC PREPARE with a nested ROLLBACK TO SAVEPOINT -$node_A->safe_psql( - 'postgres', " - BEGIN; - INSERT INTO test_tab VALUES (9999, 'foobar'); - SAVEPOINT sp_inner; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - ROLLBACK TO SAVEPOINT sp_inner; - PREPARE TRANSACTION 'outer'; - "); - -$node_A->wait_for_catchup($appname_B); -$node_B->wait_for_catchup($appname_C); - -# check the transaction state prepared on subscriber(s) -$result = - $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber B'); -$result = - $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber C'); - -# 2PC COMMIT -$node_A->safe_psql('postgres', "COMMIT PREPARED 'outer';"); - -$node_A->wait_for_catchup($appname_B); -$node_B->wait_for_catchup($appname_C); - -# check the transaction state is ended on subscriber -$result = - $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is ended on subscriber B'); -$result = - $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is ended on subscriber C'); - -# check inserts are visible at subscriber(s). -# All the streamed data (prior to the SAVEPOINT) should be rolled back. -# (9999, 'foobar') should be committed. -$result = $node_B->safe_psql('postgres', - "SELECT count(*) FROM test_tab where b = 'foobar';"); -is($result, qq(1), 'Rows committed are present on subscriber B'); -$result = $node_B->safe_psql('postgres', "SELECT count(*) FROM test_tab;"); -is($result, qq(3), 'Rows committed are present on subscriber B'); -$result = $node_C->safe_psql('postgres', - "SELECT count(*) FROM test_tab where b = 'foobar';"); -is($result, qq(1), 'Rows committed are present on subscriber C'); -$result = $node_C->safe_psql('postgres', "SELECT count(*) FROM test_tab;"); -is($result, qq(3), 'Rows committed are present on subscriber C'); +###################################### +# Test using streaming mode 'parallel' +###################################### +test_streaming($node_A, $node_B, $node_C, $appname_B, $appname_C, 'parallel'); ############################### # check all the cleanup diff --git a/src/test/subscription/t/023_twophase_stream.pl b/src/test/subscription/t/023_twophase_stream.pl index d8475d25a4..b89414ab74 100644 --- a/src/test/subscription/t/023_twophase_stream.pl +++ b/src/test/subscription/t/023_twophase_stream.pl @@ -8,6 +8,266 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" +# parameter so the same code can be run both for the streaming=on and +# streaming=parallel cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_parallel) = @_; + + ############################### + # Test 2PC PREPARE / COMMIT PREPARED. + # 1. Data is streamed as a 2PC transaction. + # 2. Then do commit prepared. + # + # Expect all data is replicated on subscriber side after the commit. + ############################### + + # Check that a background worker starts if "streaming" parameter is + # specified as "parallel". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_parallel) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + # check that 2PC gets replicated to subscriber + # Insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + if ($is_parallel) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is in prepared state on subscriber + my $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber'); + + # 2PC transaction gets committed + $node_publisher->safe_psql('postgres', + "COMMIT PREPARED 'test_prepared_tab';"); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is committed on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), + 'Rows inserted by 2PC have committed on subscriber, and extra columns contain local defaults' + ); + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is committed on subscriber'); + + ############################### + # Test 2PC PREPARE / ROLLBACK PREPARED. + # 1. Table is deleted back to 2 rows which are replicated on subscriber. + # 2. Data is streamed using 2PC. + # 3. Do rollback prepared. + # + # Expect data rolls back leaving only the original 2 rows. + ############################### + + # First, delete the data except for 2 rows (will be replicated) + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE a > 2;"); + + # Then insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is in prepared state on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber'); + + # 2PC transaction gets aborted + $node_publisher->safe_psql('postgres', + "ROLLBACK PREPARED 'test_prepared_tab';"); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is aborted on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(2|2|2), + 'Rows inserted by 2PC are rolled back, leaving only the original 2 rows' + ); + + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is aborted on subscriber'); + + ############################### + # Check that 2PC COMMIT PREPARED is decoded properly on crash restart. + # 1. insert, update and delete enough rows to exceed the 64kB limit. + # 2. Then server crashes before the 2PC transaction is committed. + # 3. After servers are restarted the pending transaction is committed. + # + # Expect all data is replicated on subscriber side after the commit. + # Note: both publisher and subscriber do crash/restart. + ############################### + + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + $node_subscriber->stop('immediate'); + $node_publisher->stop('immediate'); + + $node_publisher->start; + $node_subscriber->start; + + # commit post the restart + $node_publisher->safe_psql('postgres', + "COMMIT PREPARED 'test_prepared_tab';"); + $node_publisher->wait_for_catchup($appname); + + # check inserts are visible + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), + 'Rows inserted by 2PC have committed on subscriber, and extra columns contain local defaults' + ); + + ############################### + # Do INSERT after the PREPARE but before ROLLBACK PREPARED. + # 1. Table is deleted back to 2 rows which are replicated on subscriber. + # 2. Data is streamed using 2PC. + # 3. A single row INSERT is done which is after the PREPARE. + # 4. Then do a ROLLBACK PREPARED. + # + # Expect the 2PC data rolls back leaving only 3 rows on the subscriber + # (the original 2 + inserted 1). + ############################### + + # First, delete the data except for 2 rows (will be replicated) + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE a > 2;"); + + # Then insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is in prepared state on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber'); + + # Insert a different record (now we are outside of the 2PC transaction) + # Note: the 2PC transaction still holds row locks so make sure this insert is for a separate primary key + $node_publisher->safe_psql('postgres', + "INSERT INTO test_tab VALUES (99999, 'foobar')"); + + # 2PC transaction gets aborted + $node_publisher->safe_psql('postgres', + "ROLLBACK PREPARED 'test_prepared_tab';"); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is aborted on subscriber, + # but the extra INSERT outside of the 2PC still was replicated + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3|3|3), + 'check the outside insert was copied to subscriber'); + + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is aborted on subscriber'); + + ############################### + # Do INSERT after the PREPARE but before COMMIT PREPARED. + # 1. Table is deleted back to 2 rows which are replicated on subscriber. + # 2. Data is streamed using 2PC. + # 3. A single row INSERT is done which is after the PREPARE. + # 4. Then do a COMMIT PREPARED. + # + # Expect 2PC data + the extra row are on the subscriber + # (the 3334 + inserted 1 = 3335). + ############################### + + # First, delete the data except for 2 rows (will be replicated) + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE a > 2;"); + + # Then insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is in prepared state on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber'); + + # Insert a different record (now we are outside of the 2PC transaction) + # Note: the 2PC transaction still holds row locks so make sure this insert is for a separare primary key + $node_publisher->safe_psql('postgres', + "INSERT INTO test_tab VALUES (99999, 'foobar')"); + + # 2PC transaction gets committed + $node_publisher->safe_psql('postgres', + "COMMIT PREPARED 'test_prepared_tab';"); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is committed on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3335|3335|3335), + 'Rows inserted by 2PC (as well as outside insert) have committed on subscriber, and extra columns contain local defaults' + ); + + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is committed on subscriber'); + + # Cleanup the test data + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE a > 2;"); + $node_publisher->wait_for_catchup($appname); +} + ############################### # Setup ############################### @@ -48,6 +308,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql( 'postgres', " CREATE SUBSCRIPTION tap_sub @@ -70,236 +334,30 @@ my $twophase_query = $node_subscriber->poll_query_until('postgres', $twophase_query) or die "Timed out while waiting for subscriber to enable twophase"; -############################### # Check initial data was copied to subscriber -############################### my $result = $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab"); is($result, qq(2|2|2), 'check initial data was copied to subscriber'); -############################### -# Test 2PC PREPARE / COMMIT PREPARED. -# 1. Data is streamed as a 2PC transaction. -# 2. Then do commit prepared. -# -# Expect all data is replicated on subscriber side after the commit. -############################### - -# check that 2PC gets replicated to subscriber -# Insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is in prepared state on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber'); - -# 2PC transaction gets committed -$node_publisher->safe_psql('postgres', - "COMMIT PREPARED 'test_prepared_tab';"); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is committed on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), - 'Rows inserted by 2PC have committed on subscriber, and extra columns contain local defaults' -); -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is committed on subscriber'); - -############################### -# Test 2PC PREPARE / ROLLBACK PREPARED. -# 1. Table is deleted back to 2 rows which are replicated on subscriber. -# 2. Data is streamed using 2PC. -# 3. Do rollback prepared. -# -# Expect data rolls back leaving only the original 2 rows. -############################### - -# First, delete the data except for 2 rows (will be replicated) -$node_publisher->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); - -# Then insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is in prepared state on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber'); - -# 2PC transaction gets aborted -$node_publisher->safe_psql('postgres', - "ROLLBACK PREPARED 'test_prepared_tab';"); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is aborted on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(2|2|2), - 'Rows inserted by 2PC are rolled back, leaving only the original 2 rows'); - -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is aborted on subscriber'); - -############################### -# Check that 2PC COMMIT PREPARED is decoded properly on crash restart. -# 1. insert, update and delete enough rows to exceed the 64kB limit. -# 2. Then server crashes before the 2PC transaction is committed. -# 3. After servers are restarted the pending transaction is committed. -# -# Expect all data is replicated on subscriber side after the commit. -# Note: both publisher and subscriber do crash/restart. -############################### - -$node_publisher->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_subscriber->stop('immediate'); -$node_publisher->stop('immediate'); - -$node_publisher->start; -$node_subscriber->start; - -# commit post the restart -$node_publisher->safe_psql('postgres', - "COMMIT PREPARED 'test_prepared_tab';"); -$node_publisher->wait_for_catchup($appname); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -# check inserts are visible -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), - 'Rows inserted by 2PC have committed on subscriber, and extra columns contain local defaults' +###################################### +# Test using streaming mode 'parallel' +###################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" ); -############################### -# Do INSERT after the PREPARE but before ROLLBACK PREPARED. -# 1. Table is deleted back to 2 rows which are replicated on subscriber. -# 2. Data is streamed using 2PC. -# 3. A single row INSERT is done which is after the PREPARE. -# 4. Then do a ROLLBACK PREPARED. -# -# Expect the 2PC data rolls back leaving only 3 rows on the subscriber -# (the original 2 + inserted 1). -############################### - -# First, delete the data except for 2 rows (will be replicated) -$node_publisher->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); - -# Then insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is in prepared state on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber'); - -# Insert a different record (now we are outside of the 2PC transaction) -# Note: the 2PC transaction still holds row locks so make sure this insert is for a separate primary key -$node_publisher->safe_psql('postgres', - "INSERT INTO test_tab VALUES (99999, 'foobar')"); - -# 2PC transaction gets aborted -$node_publisher->safe_psql('postgres', - "ROLLBACK PREPARED 'test_prepared_tab';"); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is aborted on subscriber, -# but the extra INSERT outside of the 2PC still was replicated -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3|3|3), 'check the outside insert was copied to subscriber'); - -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is aborted on subscriber'); - -############################### -# Do INSERT after the PREPARE but before COMMIT PREPARED. -# 1. Table is deleted back to 2 rows which are replicated on subscriber. -# 2. Data is streamed using 2PC. -# 3. A single row INSERT is done which is after the PREPARE. -# 4. Then do a COMMIT PREPARED. -# -# Expect 2PC data + the extra row are on the subscriber -# (the 3334 + inserted 1 = 3335). -############################### - -# First, delete the data except for 2 rows (will be replicated) -$node_publisher->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); - -# Then insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is in prepared state on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber'); - -# Insert a different record (now we are outside of the 2PC transaction) -# Note: the 2PC transaction still holds row locks so make sure this insert is for a separare primary key -$node_publisher->safe_psql('postgres', - "INSERT INTO test_tab VALUES (99999, 'foobar')"); - -# 2PC transaction gets committed -$node_publisher->safe_psql('postgres', - "COMMIT PREPARED 'test_prepared_tab';"); - -$node_publisher->wait_for_catchup($appname); +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = parallel)"); -# check that transaction is committed on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3335|3335|3335), - 'Rows inserted by 2PC (as well as outside insert) have committed on subscriber, and extra columns contain local defaults' -); +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is committed on subscriber'); +test_streaming($node_publisher, $node_subscriber, $appname, 1); ############################### # check all the cleanup -- 2.23.0.windows.1 [application/octet-stream] v16-0003-Add-some-checks-before-using-apply-background-wo.patch (36.6K, ../../OS3PR01MB627594D75E870BBB45E2E80A9E839@OS3PR01MB6275.jpnprd01.prod.outlook.com/4-v16-0003-Add-some-checks-before-using-apply-background-wo.patch) download | inline diff: From fae06ce145c7f239eb27293999afcc44b04eebfb Mon Sep 17 00:00:00 2001 From: wangw <[email protected]> Date: Tue, 14 Jun 2022 11:23:52 +0800 Subject: [PATCH v16 3/4] Add some checks before using apply background worker to apply changes. streaming=parallel mode has two requirements: 1) The unique column in the relation on the subscriber-side should also be the unique column on the publisher-side; 2) There cannot be any non-immutable functions in the subscriber-side replicated table. Look for functions in the following places: * a. Trigger functions * b. Column default value expressions and domain constraints * c. Constraint expressions * d. Foreign keys --- doc/src/sgml/ref/create_subscription.sgml | 4 + .../replication/logical/applybgworker.c | 42 ++ src/backend/replication/logical/proto.c | 66 ++- src/backend/replication/logical/relation.c | 199 +++++++++ src/backend/replication/logical/tablesync.c | 1 + src/backend/replication/logical/worker.c | 23 +- src/backend/utils/cache/typcache.c | 17 + src/include/replication/logicalproto.h | 1 + src/include/replication/logicalrelation.h | 15 + src/include/replication/worker_internal.h | 1 + src/include/utils/typcache.h | 2 + .../subscription/t/022_twophase_cascade.pl | 6 + .../subscription/t/032_streaming_apply.pl | 391 ++++++++++++++++++ src/tools/pgindent/typedefs.list | 1 + 14 files changed, 759 insertions(+), 10 deletions(-) create mode 100644 src/test/subscription/t/032_streaming_apply.pl diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml index 71dd4aca81..270e3d382e 100644 --- a/doc/src/sgml/ref/create_subscription.sgml +++ b/doc/src/sgml/ref/create_subscription.sgml @@ -240,6 +240,10 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl transaction is committed. Note that if an error happens when applying changes in a background worker, the finish LSN of the remote transaction might not be reported in the server log. + Parallel mode has two requirements: 1) the unique column in the + relation on the subscriber-side should also be the unique column on + the publisher-side; 2) there cannot be any non-immutable functions + in the subscriber-side replicated table. </para> </listitem> </varlistentry> diff --git a/src/backend/replication/logical/applybgworker.c b/src/backend/replication/logical/applybgworker.c index bb2b180e29..3be238223b 100644 --- a/src/backend/replication/logical/applybgworker.c +++ b/src/backend/replication/logical/applybgworker.c @@ -784,3 +784,45 @@ apply_bgworker_subxact_info_add(TransactionId current_xid) MemoryContextSwitchTo(oldctx); } } + +/* + * Check if changes on this relation can be applied by an apply background + * worker. + * + * Although the commit order is maintained only allowing one process to commit + * at a time, the access order to the relation has changed. This could cause + * unexpected problems if the unique column on the replicated table is + * inconsistent with the publisher-side or contains non-immutable functions + * when applying transactions in the apply background worker. + */ +void +apply_bgworker_relation_check(LogicalRepRelMapEntry *rel) +{ + /* Skip check if not an apply background worker. */ + if (!am_apply_bgworker()) + return; + + /* + * Partition table checks are done later in function + * apply_handle_tuple_routing. + */ + if (rel->localrel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + return; + + /* + * Return if changes on this relation can be applied by an apply background + * worker. + */ + if (rel->parallel == PARALLEL_APPLY_SAFE) + return; + + /* We are in error mode and should give user correct error. */ + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot replicate target relation \"%s.%s\" in parallel " + "mode", rel->remoterel.nspname, rel->remoterel.relname), + errdetail("The unique column on subscriber is not the unique " + "column on publisher or there is at least one " + "non-immutable function."), + errhint("Please change the streaming option to 'on' instead of 'parallel'."))); +} diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c index affd08cfa4..12a9799f5c 100644 --- a/src/backend/replication/logical/proto.c +++ b/src/backend/replication/logical/proto.c @@ -23,7 +23,8 @@ /* * Protocol message flags. */ -#define LOGICALREP_IS_REPLICA_IDENTITY 1 +#define ATTR_IS_REPLICA_IDENTITY (1 << 0) +#define ATTR_IS_UNIQUE (1 << 1) #define MESSAGE_TRANSACTIONAL (1<<0) #define TRUNCATE_CASCADE (1<<0) @@ -933,11 +934,55 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns) TupleDesc desc; int i; uint16 nliveatts = 0; - Bitmapset *idattrs = NULL; + Bitmapset *idattrs = NULL, + *attunique = NULL; bool replidentfull; desc = RelationGetDescr(rel); + if (rel->rd_rel->relhasindex) + { + List *indexoidlist = RelationGetIndexList(rel); + ListCell *indexoidscan; + + foreach(indexoidscan, indexoidlist) + { + Oid indexoid = lfirst_oid(indexoidscan); + Relation indexRel; + + /* Look up the description for index */ + indexRel = RelationIdGetRelation(indexoid); + + if (!RelationIsValid(indexRel)) + elog(ERROR, "could not open relation with OID %u", indexoid); + + if (indexRel->rd_index->indisunique) + { + int i; + + /* Add referenced attributes to idindexattrs */ + for (i = 0; i < indexRel->rd_index->indnatts; i++) + { + int attrnum = indexRel->rd_index->indkey.values[i]; + + /* + * We don't include non-key columns into idindexattrs + * bitmaps. See RelationGetIndexAttrBitmap. + */ + if (attrnum != 0) + { + if (i < indexRel->rd_index->indnkeyatts && + !bms_is_member(attrnum - FirstLowInvalidHeapAttributeNumber, attunique)) + attunique = bms_add_member(attunique, + attrnum - FirstLowInvalidHeapAttributeNumber); + } + } + } + RelationClose(indexRel); + } + list_free(indexoidlist); + } + /* send number of live attributes */ for (i = 0; i < desc->natts; i++) { @@ -974,7 +1019,11 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns) if (replidentfull || bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber, idattrs)) - flags |= LOGICALREP_IS_REPLICA_IDENTITY; + flags |= ATTR_IS_REPLICA_IDENTITY; + + if (bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber, + attunique)) + flags |= ATTR_IS_UNIQUE; pq_sendbyte(out, flags); @@ -1001,7 +1050,8 @@ logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel) int natts; char **attnames; Oid *atttyps; - Bitmapset *attkeys = NULL; + Bitmapset *attkeys = NULL, + *attunique = NULL; natts = pq_getmsgint(in, 2); attnames = palloc(natts * sizeof(char *)); @@ -1012,11 +1062,14 @@ logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel) { uint8 flags; - /* Check for replica identity column */ + /* Check for replica identity and unique column */ flags = pq_getmsgbyte(in); - if (flags & LOGICALREP_IS_REPLICA_IDENTITY) + if (flags & ATTR_IS_REPLICA_IDENTITY) attkeys = bms_add_member(attkeys, i); + if (flags & ATTR_IS_UNIQUE) + attunique = bms_add_member(attunique, i); + /* attribute name */ attnames[i] = pstrdup(pq_getmsgstring(in)); @@ -1030,6 +1083,7 @@ logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel) rel->attnames = attnames; rel->atttyps = atttyps; rel->attkeys = attkeys; + rel->attunique = attunique; rel->natts = natts; } diff --git a/src/backend/replication/logical/relation.c b/src/backend/replication/logical/relation.c index e989047681..f4f5531e4d 100644 --- a/src/backend/replication/logical/relation.c +++ b/src/backend/replication/logical/relation.c @@ -19,12 +19,19 @@ #include "access/table.h" #include "catalog/namespace.h" +#include "catalog/pg_proc.h" #include "catalog/pg_subscription_rel.h" +#include "commands/trigger.h" #include "executor/executor.h" #include "nodes/makefuncs.h" +#include "optimizer/optimizer.h" #include "replication/logicalrelation.h" #include "replication/worker_internal.h" +#include "rewrite/rewriteHandler.h" #include "utils/inval.h" +#include "utils/lsyscache.h" +#include "utils/syscache.h" +#include "utils/typcache.h" static MemoryContext LogicalRepRelMapContext = NULL; @@ -91,6 +98,26 @@ logicalrep_relmap_invalidate_cb(Datum arg, Oid reloid) } } +/* + * Relcache invalidation callback to reset parallel flag. + */ +static void +logicalrep_relmap_reset_parallel_cb(Datum arg, int cacheid, uint32 hashvalue) +{ + HASH_SEQ_STATUS hash_seq; + LogicalRepRelMapEntry *entry; + + if (LogicalRepRelMap == NULL) + return; + + hash_seq_init(&hash_seq, LogicalRepRelMap); + while ((entry = hash_seq_search(&hash_seq)) != NULL) + { + entry->parallel = PARALLEL_APPLY_UNKNOWN; + entry->localrelvalid = false; + } +} + /* * Initialize the relation map cache. */ @@ -116,6 +143,9 @@ logicalrep_relmap_init(void) /* Watch for invalidation events. */ CacheRegisterRelcacheCallback(logicalrep_relmap_invalidate_cb, (Datum) 0); + CacheRegisterSyscacheCallback(PROCOID, + logicalrep_relmap_reset_parallel_cb, + (Datum) 0); } /* @@ -142,6 +172,7 @@ logicalrep_relmap_free_entry(LogicalRepRelMapEntry *entry) pfree(remoterel->atttyps); } bms_free(remoterel->attkeys); + bms_free(remoterel->attunique); if (entry->attrmap) free_attrmap(entry->attrmap); @@ -190,6 +221,7 @@ logicalrep_relmap_update(LogicalRepRelation *remoterel) } entry->remoterel.replident = remoterel->replident; entry->remoterel.attkeys = bms_copy(remoterel->attkeys); + entry->remoterel.attunique = bms_copy(remoterel->attunique); MemoryContextSwitchTo(oldctx); } @@ -310,6 +342,166 @@ logicalrep_rel_mark_updatable(LogicalRepRelMapEntry *entry) } } +/* + * Check if changes on one relation can be applied by an apply background + * worker and assign the 'parallel' flag. + * + * There are two requirements for applying changes in an apply background + * worker: 1) The unique column in the relation on the subscriber-side should + * also be the unique column on the publisher-side; 2) There cannot be any + * non-immutable functions in the subscriber-side. + * + * We just mark the relation entry as 'PARALLEL_APPLY_UNSAFE' here if changes + * on one relation can not be applied by an apply background worker and leave + * it to apply_bgworker_relation_check() to throw the actual error if needed. + */ +static void +logicalrep_rel_mark_parallel(LogicalRepRelMapEntry *entry) +{ + Bitmapset *ukey; + int i; + TupleDesc tupdesc; + int attnum; + List *fkeys = NIL; + + /* Fast path if we marked 'parallel' flag. */ + if (entry->parallel != PARALLEL_APPLY_UNKNOWN) + return; + + /* Initialize the flag. */ + entry->parallel = PARALLEL_APPLY_SAFE; + + /* + * First, we check if the unique column in the relation on the + * subscriber-side is also the unique column on the publisher-side. + */ + ukey = RelationGetIndexAttrBitmap(entry->localrel, + INDEX_ATTR_BITMAP_KEY); + + if (ukey) + { + i = -1; + while ((i = bms_next_member(ukey, i)) >= 0) + { + attnum = AttrNumberGetAttrOffset(i + FirstLowInvalidHeapAttributeNumber); + + if (entry->attrmap->attnums[attnum] < 0 || + !bms_is_member(entry->attrmap->attnums[attnum], entry->remoterel.attunique)) + { + entry->parallel = PARALLEL_APPLY_UNSAFE; + return; + } + } + } + + /* + * Then, We check if there is any non-immutable function in the local + * table. Look for functions in the following places: + * a. trigger functions; + * b. Column default value expressions and domain constraints; + * c. Constraint expressions; + * d. Foreign keys. + */ + /* Check the trigger functions. */ + if (entry->localrel->trigdesc != NULL) + { + for (i = 0; i < entry->localrel->trigdesc->numtriggers; i++) + { + Trigger *trig = entry->localrel->trigdesc->triggers + i; + + if (trig->tgenabled != TRIGGER_FIRES_ALWAYS && + trig->tgenabled != TRIGGER_FIRES_ON_REPLICA) + continue; + + if (func_volatile(trig->tgfoid) != PROVOLATILE_IMMUTABLE) + { + entry->parallel = PARALLEL_APPLY_UNSAFE; + return; + } + } + } + + /* Check the columns. */ + tupdesc = RelationGetDescr(entry->localrel); + for (attnum = 0; attnum < tupdesc->natts; attnum++) + { + Form_pg_attribute att = TupleDescAttr(tupdesc, attnum); + + /* We don't need info for dropped or generated attributes */ + if (att->attisdropped || att->attgenerated) + continue; + + /* + * We don't need to check columns that only exist on the + * subscriber + */ + if (entry->attrmap->attnums[attnum] < 0) + continue; + + if (att->atthasdef) + { + Node *defaultexpr; + + defaultexpr = build_column_default(entry->localrel, attnum + 1); + if (contain_mutable_functions(defaultexpr)) + { + entry->parallel = PARALLEL_APPLY_UNSAFE; + return; + } + } + + /* + * If the column is of a DOMAIN type, determine whether + * that domain has any CHECK expressions that are not + * immutable. + */ + if (get_typtype(att->atttypid) == TYPTYPE_DOMAIN) + { + List *domain_constraints; + ListCell *lc; + + domain_constraints = GetDomainConstraints(att->atttypid); + + foreach(lc, domain_constraints) + { + DomainConstraintState *con = (DomainConstraintState *) lfirst(lc); + + if (con->check_expr && contain_mutable_functions((Node *) con->check_expr)) + { + entry->parallel = PARALLEL_APPLY_UNSAFE; + return; + } + } + } + } + + /* Check the constraints. */ + if (tupdesc->constr) + { + ConstrCheck *check = tupdesc->constr->check; + + /* + * Determine if there are any CHECK constraints which + * contains non-immutable function. + */ + for (i = 0; i < tupdesc->constr->num_check; i++) + { + Expr *check_expr = stringToNode(check[i].ccbin); + + if (contain_mutable_functions((Node *) check_expr)) + { + entry->parallel = PARALLEL_APPLY_UNSAFE; + return; + } + } + } + + /* Check the foreign keys. */ + fkeys = RelationGetFKeyList(entry->localrel); + if (fkeys) + entry->parallel = PARALLEL_APPLY_UNSAFE; +} + /* * Open the local relation associated with the remote one. * @@ -438,6 +630,9 @@ logicalrep_rel_open(LogicalRepRelId remoteid, LOCKMODE lockmode) */ logicalrep_rel_mark_updatable(entry); + /* Set if changes could be applied in the apply background worker. */ + logicalrep_rel_mark_parallel(entry); + entry->localrelvalid = true; } @@ -653,6 +848,7 @@ logicalrep_partition_open(LogicalRepRelMapEntry *root, } entry->remoterel.replident = remoterel->replident; entry->remoterel.attkeys = bms_copy(remoterel->attkeys); + entry->remoterel.attunique = bms_copy(remoterel->attunique); } entry->localrel = partrel; @@ -696,6 +892,9 @@ logicalrep_partition_open(LogicalRepRelMapEntry *root, /* Set if the table's replica identity is enough to apply update/delete. */ logicalrep_rel_mark_updatable(entry); + /* Set if changes could be applied in the apply background worker. */ + logicalrep_rel_mark_parallel(entry); + entry->localrelvalid = true; /* state and statelsn are left set to 0. */ diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c index 8ffba7e2e5..3cdbf8b457 100644 --- a/src/backend/replication/logical/tablesync.c +++ b/src/backend/replication/logical/tablesync.c @@ -884,6 +884,7 @@ fetch_remote_table_info(char *nspname, char *relname, lrel->attnames = palloc0(MaxTupleAttributeNumber * sizeof(char *)); lrel->atttyps = palloc0(MaxTupleAttributeNumber * sizeof(Oid)); lrel->attkeys = NULL; + lrel->attunique = NULL; /* * Store the columns as a list of names. Ignore those that are not diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c index bbd0304a8d..a46eb7dfab 100644 --- a/src/backend/replication/logical/worker.c +++ b/src/backend/replication/logical/worker.c @@ -1388,6 +1388,14 @@ apply_handle_stream_stop(StringInfo s) { char action = LOGICAL_REP_MSG_STREAM_STOP; + /* + * Unlike stream_commit, we don't need to wait here for stream_stop to + * finish. Allowing the other transaction to be applied before + * stream_stop is finished can lead to failures if the unique + * index/constraint is different between publisher and subscriber. But + * for such cases, we don't allow streamed transactions to be applied + * in parallel. See apply_bgworker_relation_check. + */ apply_bgworker_send_data(stream_apply_worker, 1, &action); elog(DEBUG1, "stopped streaming of xid %u, %u changes streamed", stream_xid, nchanges); @@ -2040,6 +2048,8 @@ apply_handle_insert(StringInfo s) /* Set relation for error callback */ apply_error_callback_arg.rel = rel; + apply_bgworker_relation_check(rel); + /* Initialize the executor state. */ edata = create_edata_for_relation(rel); estate = edata->estate; @@ -2183,6 +2193,8 @@ apply_handle_update(StringInfo s) /* Check if we can do the update. */ check_relation_updatable(rel); + apply_bgworker_relation_check(rel); + /* Initialize the executor state. */ edata = create_edata_for_relation(rel); estate = edata->estate; @@ -2351,6 +2363,8 @@ apply_handle_delete(StringInfo s) /* Check if we can do the delete. */ check_relation_updatable(rel); + apply_bgworker_relation_check(rel); + /* Initialize the executor state. */ edata = create_edata_for_relation(rel); estate = edata->estate; @@ -2536,13 +2550,14 @@ apply_handle_tuple_routing(ApplyExecutionData *edata, } MemoryContextSwitchTo(oldctx); + part_entry = logicalrep_partition_open(relmapentry, partrel, + attrmap); + /* Check if we can do the update or delete on the leaf partition. */ if (operation == CMD_UPDATE || operation == CMD_DELETE) - { - part_entry = logicalrep_partition_open(relmapentry, partrel, - attrmap); check_relation_updatable(part_entry); - } + + apply_bgworker_relation_check(part_entry); switch (operation) { diff --git a/src/backend/utils/cache/typcache.c b/src/backend/utils/cache/typcache.c index 808f9ebd0d..b248899d82 100644 --- a/src/backend/utils/cache/typcache.c +++ b/src/backend/utils/cache/typcache.c @@ -2540,6 +2540,23 @@ compare_values_of_enum(TypeCacheEntry *tcache, Oid arg1, Oid arg2) return 0; } +/* + * GetDomainConstraints --- get DomainConstraintState list of specified domain type + */ +List * +GetDomainConstraints(Oid type_id) +{ + TypeCacheEntry *typentry; + List *constraints = NIL; + + typentry = lookup_type_cache(type_id, TYPECACHE_DOMAIN_CONSTR_INFO); + + if(typentry->domainData != NULL) + constraints = typentry->domainData->constraints; + + return constraints; +} + /* * Load (or re-load) the enumData member of the typcache entry. */ diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h index 0f74a3392b..130c4eb82f 100644 --- a/src/include/replication/logicalproto.h +++ b/src/include/replication/logicalproto.h @@ -113,6 +113,7 @@ typedef struct LogicalRepRelation char replident; /* replica identity */ char relkind; /* remote relation kind */ Bitmapset *attkeys; /* Bitmap of key columns */ + Bitmapset *attunique; /* Bitmap of unique columns */ } LogicalRepRelation; /* Type mapping info */ diff --git a/src/include/replication/logicalrelation.h b/src/include/replication/logicalrelation.h index 78cd7e77f5..7a1e9f762a 100644 --- a/src/include/replication/logicalrelation.h +++ b/src/include/replication/logicalrelation.h @@ -15,6 +15,19 @@ #include "access/attmap.h" #include "replication/logicalproto.h" +/* + * States to determine if changes on one relation can be applied by an apply + * background worker. + */ +typedef enum RelParallel +{ + PARALLEL_APPLY_UNKNOWN = 0, /* unknown */ + PARALLEL_APPLY_SAFE, /* Can apply changes in an apply background + worker */ + PARALLEL_APPLY_UNSAFE /* Can not apply changes in an apply background + worker */ +} RelParallel; + typedef struct LogicalRepRelMapEntry { LogicalRepRelation remoterel; /* key is remoterel.remoteid */ @@ -31,6 +44,8 @@ typedef struct LogicalRepRelMapEntry Relation localrel; /* relcache entry (NULL when closed) */ AttrMap *attrmap; /* map of local attributes to remote ones */ bool updatable; /* Can apply updates/deletes? */ + RelParallel parallel; /* Can apply changes in an apply + background worker? */ /* Sync state. */ char state; diff --git a/src/include/replication/worker_internal.h b/src/include/replication/worker_internal.h index 5be8f5755e..b28f4ab977 100644 --- a/src/include/replication/worker_internal.h +++ b/src/include/replication/worker_internal.h @@ -194,6 +194,7 @@ extern void apply_bgworker_free(ApplyBgworkerState *wstate); extern void apply_bgworker_check_status(void); extern void apply_bgworker_set_status(ApplyBgworkerStatus status); extern void apply_bgworker_subxact_info_add(TransactionId current_xid); +extern void apply_bgworker_relation_check(LogicalRepRelMapEntry *rel); static inline bool am_tablesync_worker(void) diff --git a/src/include/utils/typcache.h b/src/include/utils/typcache.h index 431ad7f1b3..ed7c2e7f48 100644 --- a/src/include/utils/typcache.h +++ b/src/include/utils/typcache.h @@ -199,6 +199,8 @@ extern uint64 assign_record_type_identifier(Oid type_id, int32 typmod); extern int compare_values_of_enum(TypeCacheEntry *tcache, Oid arg1, Oid arg2); +extern List *GetDomainConstraints(Oid type_id); + extern size_t SharedRecordTypmodRegistryEstimate(void); extern void SharedRecordTypmodRegistryInit(SharedRecordTypmodRegistry *, diff --git a/src/test/subscription/t/022_twophase_cascade.pl b/src/test/subscription/t/022_twophase_cascade.pl index 0a4152d3be..30a01f7305 100644 --- a/src/test/subscription/t/022_twophase_cascade.pl +++ b/src/test/subscription/t/022_twophase_cascade.pl @@ -39,6 +39,12 @@ sub test_streaming ALTER SUBSCRIPTION tap_sub_C SET (streaming = $streaming_mode)"); + if ($streaming_mode eq 'parallel') + { + $node_C->safe_psql( + 'postgres', "ALTER TABLE test_tab ALTER c DROP DEFAULT"); + } + # Wait for subscribers to finish initialization $node_A->poll_query_until( diff --git a/src/test/subscription/t/032_streaming_apply.pl b/src/test/subscription/t/032_streaming_apply.pl new file mode 100644 index 0000000000..eca4328676 --- /dev/null +++ b/src/test/subscription/t/032_streaming_apply.pl @@ -0,0 +1,391 @@ +# Copyright (c) 2022, PostgreSQL Global Development Group + +# Test the restrictions of streaming mode "parallel" in logical replication + +use strict; +use warnings; +use PostgreSQL::Test::Cluster; +use PostgreSQL::Test::Utils; +use Test::More; + +my $offset = 0; + +# Create publisher node +my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); +$node_publisher->init(allows_streaming => 'logical'); +$node_publisher->append_conf('postgresql.conf', + 'logical_decoding_work_mem = 64kB'); +$node_publisher->start; + +# Create subscriber node +my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber'); +$node_subscriber->init; +$node_subscriber->start; + +# Setup structure on publisher +$node_publisher->safe_psql('postgres', + "CREATE TABLE test_tab (a int primary key, b varchar)"); +$node_publisher->safe_psql('postgres', + "CREATE TABLE test_tab_partitioned (a int primary key, b varchar)"); + +# Setup structure on subscriber +# We need to test normal table and partition table. +$node_subscriber->safe_psql('postgres', + "CREATE TABLE test_tab (a int primary key, b varchar)"); +$node_subscriber->safe_psql('postgres', + "CREATE TABLE test_tab_partitioned (a int primary key, b varchar) PARTITION BY RANGE(a)"); +$node_subscriber->safe_psql('postgres', + "CREATE TABLE test_tab_partition (LIKE test_tab_partitioned)"); +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partitioned ATTACH PARTITION test_tab_partition DEFAULT" +); + +# Setup logical replication +my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres'; +$node_publisher->safe_psql('postgres', + "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); +$node_publisher->safe_psql('postgres', + "CREATE PUBLICATION tap_pub_partitioned FOR TABLE test_tab_partitioned"); + +my $appname = 'tap_sub'; +$node_subscriber->safe_psql( + 'postgres', " + CREATE SUBSCRIPTION tap_sub + CONNECTION '$publisher_connstr application_name=$appname' + PUBLICATION tap_pub, tap_pub_partitioned + WITH (streaming = parallel, copy_data = false)"); + +$node_publisher->wait_for_catchup($appname); + +# It is not allowed that the unique index on the publisher and the subscriber +# is different. Check the error reported by background worker in this case. +# First we check the unique index on normal table. +$node_subscriber->safe_psql('postgres', + "CREATE UNIQUE INDEX test_tab_b_idx ON test_tab (b)"); + +# Check that a background worker starts if "streaming" option is specified as +# "parallel". We have to look for the DEBUG1 log messages about that, so +# temporarily bump up the log verbosity. +$node_subscriber->append_conf('postgresql.conf', "log_min_messages = debug1"); +$node_subscriber->reload; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); +$node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); +$node_subscriber->reload; + +$node_subscriber->wait_for_log( + qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab" in parallel mode/, + $offset); + +# Drop the unique index on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', "DROP INDEX test_tab_b_idx"); + +$node_publisher->wait_for_catchup($appname); + +my $result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); +is($result, qq(5000), 'data replicated to subscriber after dropping index'); + +# Then we check the unique index on partition table. +$node_subscriber->safe_psql('postgres', + "CREATE UNIQUE INDEX test_tab_b_partition_idx ON test_tab_partition (b)"); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab_partitioned SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log( + qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab_partitioned" in parallel mode/, + $offset); + +# Drop the unique index on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', "DROP INDEX test_tab_b_partition_idx"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); +is($result, qq(5000), 'data replicated to subscriber after dropping index'); + +# Triggers which execute non-immutable function are not allowed on the +# subscriber side. Check the error reported by background worker in this case. +# First we check the trigger function on normal table. +$node_subscriber->safe_psql( + 'postgres', qq{ +CREATE FUNCTION trigger_func() RETURNS TRIGGER AS \$\$ + BEGIN + RETURN NULL; + END +\$\$ language plpgsql; +CREATE TRIGGER insert_trig +BEFORE INSERT ON test_tab +FOR EACH ROW EXECUTE PROCEDURE trigger_func(); +ALTER TABLE test_tab ENABLE REPLICA TRIGGER insert_trig; +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', "DELETE FROM test_tab"); + +$node_subscriber->wait_for_log( + qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab" in parallel mode/, + $offset); + +# Drop the trigger on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "DROP TRIGGER insert_trig ON test_tab"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); +is($result, qq(0), 'data replicated to subscriber after dropping trigger'); + +# Then we check the trigger function on partition table. +$node_subscriber->safe_psql( + 'postgres', qq{ +CREATE TRIGGER insert_trig +BEFORE INSERT ON test_tab_partition +FOR EACH ROW EXECUTE PROCEDURE trigger_func(); +ALTER TABLE test_tab_partition ENABLE REPLICA TRIGGER insert_trig; +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', "DELETE FROM test_tab_partitioned"); + +$node_subscriber->wait_for_log( + qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab_partitioned" in parallel mode/, + $offset); + +# Drop the trigger on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "DROP TRIGGER insert_trig ON test_tab_partition"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); +is($result, qq(0), 'data replicated to subscriber after dropping trigger'); + +# It is not allowed that column default value expression contains a +# non-immutable function on the subscriber side. Check the error reported by +# background worker in this case. +# First we check the column default value expression on normal table. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab ALTER COLUMN b SET DEFAULT random()"); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log( + qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab" in parallel mode/, + $offset); + +# Drop default value on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab ALTER COLUMN b DROP DEFAULT"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); +is($result, qq(5000), + 'data replicated to subscriber after dropping default value expression'); + +# Then we check the column default value expression on partition table. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partition ALTER COLUMN b SET DEFAULT random()"); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab_partitioned SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log( + qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab_partitioned" in parallel mode/, + $offset); + +# Drop default value on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partition ALTER COLUMN b DROP DEFAULT"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); +is($result, qq(5000), + 'data replicated to subscriber after dropping default value expression'); + +# It is not allowed that domain constraint expression contains a non-immutable +# function on the subscriber side. Check the error reported by background +# worker in this case. +# Because the column type of the partition table must be the same as its parent +# table, only test normal table here. +$node_subscriber->safe_psql( + 'postgres', qq{ +CREATE DOMAIN test_domain AS int CHECK(VALUE > random()); +ALTER TABLE test_tab ALTER COLUMN a TYPE test_domain; +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', "DELETE FROM test_tab"); + +$node_subscriber->wait_for_log( + qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab" in parallel mode/, + $offset); + +# Drop domain constraint expression on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab ALTER COLUMN a TYPE int"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); +is($result, qq(0), + 'data replicated to subscriber after dropping domain constraint expression' +); + +# It is not allowed that constraint expression contains a non-immutable function +# on the subscriber side. Check the error reported by background worker in this +# case. +# First we check the constraint expression on normal table. +$node_subscriber->safe_psql( + 'postgres', qq{ +ALTER TABLE test_tab ADD CONSTRAINT test_tab_con check (a > random()); +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log( + qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab" in parallel mode/, + $offset); + +# Drop constraint on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab DROP CONSTRAINT test_tab_con"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); +is($result, qq(5000), + 'data replicated to subscriber after dropping constraint expression'); + +# Then we check the constraint expression on partition table. +$node_subscriber->safe_psql( + 'postgres', qq{ +ALTER TABLE test_tab_partition ADD CONSTRAINT test_tab_con check (a > random()); +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', "DELETE FROM test_tab_partitioned"); + +$node_subscriber->wait_for_log( + qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab_partitioned" in parallel mode/, + $offset); + +# Drop constraint on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partition DROP CONSTRAINT test_tab_con"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); +is($result, qq(0), + 'data replicated to subscriber after dropping constraint expression'); + +# It is not allowed that foreign key on the subscriber side. Check the error +# reported by background worker in this case. +# First we check the foreign key on normal table. +$node_publisher->safe_psql('postgres', "DELETE FROM test_tab"); +$node_publisher->wait_for_catchup($appname); +$node_subscriber->safe_psql( + 'postgres', qq{ +CREATE TABLE test_tab_f (a int primary key); +ALTER TABLE test_tab ADD CONSTRAINT test_tabfk FOREIGN KEY(a) REFERENCES test_tab_f(a); +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log( + qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab" in parallel mode/, + $offset); + +# Drop the foreign key constraint on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab DROP CONSTRAINT test_tabfk"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); +is($result, qq(5000), + 'data replicated to subscriber after dropping the foreign key'); + +# Then we check the foreign key on partition table. +$node_publisher->wait_for_catchup($appname); +$node_subscriber->safe_psql( + 'postgres', qq{ +CREATE TABLE test_tab_partition_f (a int primary key); +ALTER TABLE test_tab_partition ADD CONSTRAINT test_tab_patition_fk FOREIGN KEY(a) REFERENCES test_tab_partition_f(a); +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab_partitioned SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log( + qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab_partitioned" in parallel mode/, + $offset); + +# Drop the foreign key constraint on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partition DROP CONSTRAINT test_tab_patition_fk"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); +is($result, qq(5000), + 'data replicated to subscriber after dropping the foreign key'); + +$node_subscriber->stop; +$node_publisher->stop; + +done_testing(); diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 4137dc77b4..ae7cd99159 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -2271,6 +2271,7 @@ RelMapFile RelMapping RelOptInfo RelOptKind +RelParallel RelToCheck RelToCluster RelabelType -- 2.23.0.windows.1 [application/octet-stream] v16-0004-Retry-to-apply-streaming-xact-only-in-apply-work.patch (26.6K, ../../OS3PR01MB627594D75E870BBB45E2E80A9E839@OS3PR01MB6275.jpnprd01.prod.outlook.com/5-v16-0004-Retry-to-apply-streaming-xact-only-in-apply-work.patch) download | inline diff: From abcc64adb1cf721c8828ad0722e7cc1e25e44963 Mon Sep 17 00:00:00 2001 From: wangw <[email protected]> Date: Wed, 15 Jun 2022 11:02:08 +0800 Subject: [PATCH v16 4/4] Retry to apply streaming xact only in apply worker If the user sets the subscription_parameter "streaming" to "parallel", when applying a streaming transaction, we will try to apply this transaction in apply background worker. However, when the changes in this transaction cannot be applied in apply background worker, the background worker will exit with an error. In this case, we can retry applying this streaming transaction in "on" mode. In this way, we may avoid blocking logical replication here. So we introduce field "subretry" in catalog "pg_subscription". When the subscriber exit with an error, we will try to set this flag to true, and when the transaction is applied successfully, we will try to set this flag to false. Then when we try to apply a streaming transaction in apply background worker, we can see if this transaction has failed before based on the "subretry" field. --- doc/src/sgml/catalogs.sgml | 9 + doc/src/sgml/ref/create_subscription.sgml | 5 + src/backend/catalog/pg_subscription.c | 1 + src/backend/catalog/system_views.sql | 4 +- src/backend/commands/subscriptioncmds.c | 1 + .../replication/logical/applybgworker.c | 15 +- src/backend/replication/logical/worker.c | 89 ++++++++++ src/bin/pg_dump/pg_dump.c | 5 +- src/include/catalog/pg_subscription.h | 4 + .../subscription/t/032_streaming_apply.pl | 154 +++++++++++------- 10 files changed, 221 insertions(+), 66 deletions(-) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index 815cae6082..28f3d121d9 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -7907,6 +7907,15 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>subretry</structfield> <type>bool</type> + </para> + <para> + True if the previous apply change failed and a retry was required. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>subconninfo</structfield> <type>text</type> diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml index 270e3d382e..bd5361991e 100644 --- a/doc/src/sgml/ref/create_subscription.sgml +++ b/doc/src/sgml/ref/create_subscription.sgml @@ -244,6 +244,11 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl relation on the subscriber-side should also be the unique column on the publisher-side; 2) there cannot be any non-immutable functions in the subscriber-side replicated table. + When applying a streaming transaction, if either requirement is not + met, the background worker will exit with an error. + <literal>parallel</literal> mode is disregarded when retrying; + instead the transaction will be applied using <literal>on</literal> + mode. </para> </listitem> </varlistentry> diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c index 8856ce3b50..9b7f09653d 100644 --- a/src/backend/catalog/pg_subscription.c +++ b/src/backend/catalog/pg_subscription.c @@ -71,6 +71,7 @@ GetSubscription(Oid subid, bool missing_ok) sub->stream = subform->substream; sub->twophasestate = subform->subtwophasestate; sub->disableonerr = subform->subdisableonerr; + sub->retry = subform->subretry; /* Get conninfo */ datum = SysCacheGetAttr(SUBSCRIPTIONOID, diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql index fedaed533b..10f4dd6785 100644 --- a/src/backend/catalog/system_views.sql +++ b/src/backend/catalog/system_views.sql @@ -1298,8 +1298,8 @@ REVOKE ALL ON pg_replication_origin_status FROM public; -- All columns of pg_subscription except subconninfo are publicly readable. REVOKE ALL ON pg_subscription FROM public; GRANT SELECT (oid, subdbid, subskiplsn, subname, subowner, subenabled, - subbinary, substream, subtwophasestate, subdisableonerr, subslotname, - subsynccommit, subpublications) + subbinary, substream, subtwophasestate, subdisableonerr, + subretry, subslotname, subsynccommit, subpublications) ON pg_subscription TO public; CREATE VIEW pg_stat_subscription_stats AS diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c index 5f349067cc..33aef31b30 100644 --- a/src/backend/commands/subscriptioncmds.c +++ b/src/backend/commands/subscriptioncmds.c @@ -662,6 +662,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt, LOGICALREP_TWOPHASE_STATE_PENDING : LOGICALREP_TWOPHASE_STATE_DISABLED); values[Anum_pg_subscription_subdisableonerr - 1] = BoolGetDatum(opts.disableonerr); + values[Anum_pg_subscription_subretry - 1] = BoolGetDatum(false); values[Anum_pg_subscription_subconninfo - 1] = CStringGetTextDatum(conninfo); if (opts.slot_name) diff --git a/src/backend/replication/logical/applybgworker.c b/src/backend/replication/logical/applybgworker.c index 3be238223b..5cfa8d1dbc 100644 --- a/src/backend/replication/logical/applybgworker.c +++ b/src/backend/replication/logical/applybgworker.c @@ -106,6 +106,18 @@ apply_bgworker_can_start(TransactionId xid) if (!XLogRecPtrIsInvalid(MySubscription->skiplsn)) return false; + /* + * Don't use apply background workers for retries, because it is possible + * that the last time we tried to apply a transaction using an apply + * background worker the checks failed (see function + * apply_bgworker_relation_check). + */ + if (MySubscription->retry) + { + elog(DEBUG1, "apply background workers are not used for retries"); + return false; + } + /* * For streaming transactions that are being applied in apply background * worker, we cannot decide whether to apply the change for a relation @@ -823,6 +835,5 @@ apply_bgworker_relation_check(LogicalRepRelMapEntry *rel) "mode", rel->remoterel.nspname, rel->remoterel.relname), errdetail("The unique column on subscriber is not the unique " "column on publisher or there is at least one " - "non-immutable function."), - errhint("Please change the streaming option to 'on' instead of 'parallel'."))); + "non-immutable function."))); } diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c index a46eb7dfab..0daeb4f9c5 100644 --- a/src/backend/replication/logical/worker.c +++ b/src/backend/replication/logical/worker.c @@ -378,6 +378,8 @@ static void clear_subscription_skip_lsn(XLogRecPtr finish_lsn); static inline void set_apply_error_context_xact(TransactionId xid, XLogRecPtr lsn); static inline void reset_apply_error_context_info(void); +static void set_subscription_retry(bool retry); + /* * Should this worker apply changes for given relation. * @@ -904,6 +906,9 @@ apply_handle_commit(StringInfo s) apply_handle_commit_internal(&commit_data); + /* Reset the retry flag. */ + set_subscription_retry(false); + /* Check the status of apply background worker if any. */ apply_bgworker_check_status(); @@ -1015,6 +1020,9 @@ apply_handle_prepare(StringInfo s) in_remote_transaction = false; + /* Reset the retry flag. */ + set_subscription_retry(false); + /* Check the status of apply background worker if any. */ apply_bgworker_check_status(); @@ -1068,6 +1076,9 @@ apply_handle_commit_prepared(StringInfo s) store_flush_position(prepare_data.end_lsn); in_remote_transaction = false; + /* Reset the retry flag. */ + set_subscription_retry(false); + /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(prepare_data.end_lsn); @@ -1123,6 +1134,9 @@ apply_handle_rollback_prepared(StringInfo s) store_flush_position(rollback_data.rollback_end_lsn); in_remote_transaction = false; + /* Reset the retry flag. */ + set_subscription_retry(false); + /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(rollback_data.rollback_end_lsn); @@ -1215,6 +1229,9 @@ apply_handle_stream_prepare(StringInfo s) /* Unlink the files with serialized changes and subxact info. */ stream_cleanup_files(MyLogicalRepWorker->subid, prepare_data.xid); } + + /* Reset the retry flag. */ + set_subscription_retry(false); } in_remote_transaction = false; @@ -1642,6 +1659,9 @@ apply_handle_stream_abort(StringInfo s) */ serialize_stream_abort(xid, subxid); } + + /* Reset the retry flag. */ + set_subscription_retry(false); } reset_apply_error_context_info(); @@ -1854,6 +1874,9 @@ apply_handle_stream_commit(StringInfo s) /* Unlink the files with serialized changes and subxact info. */ stream_cleanup_files(MyLogicalRepWorker->subid, xid); } + + /* Reset the retry flag. */ + set_subscription_retry(false); } /* Check the status of apply background worker if any. */ @@ -3897,6 +3920,9 @@ start_table_sync(XLogRecPtr *origin_startpos, char **myslotname) } PG_CATCH(); { + /* Set the retry flag. */ + set_subscription_retry(true); + if (MySubscription->disableonerr) DisableSubscriptionAndExit(); else @@ -3935,6 +3961,9 @@ start_apply(XLogRecPtr origin_startpos) } PG_CATCH(); { + /* Set the retry flag. */ + set_subscription_retry(true); + if (MySubscription->disableonerr) DisableSubscriptionAndExit(); else @@ -4461,3 +4490,63 @@ reset_apply_error_context_info(void) apply_error_callback_arg.remote_attnum = -1; set_apply_error_context_xact(InvalidTransactionId, InvalidXLogRecPtr); } + +/* + * Set subretry of pg_subscription catalog. + * + * If retry is true, subscriber is about to exit with an error. Otherwise, it + * means that the transaction was applied successfully. + */ +static void +set_subscription_retry(bool retry) +{ + Relation rel; + HeapTuple tup; + bool started_tx = false; + bool nulls[Natts_pg_subscription]; + bool replaces[Natts_pg_subscription]; + Datum values[Natts_pg_subscription]; + + if (MySubscription->retry == retry || + am_apply_bgworker()) + return; + + if (!IsTransactionState()) + { + StartTransactionCommand(); + started_tx = true; + } + + /* Look up the subscription in the catalog */ + rel = table_open(SubscriptionRelationId, RowExclusiveLock); + tup = SearchSysCacheCopy1(SUBSCRIPTIONOID, + ObjectIdGetDatum(MySubscription->oid)); + + if (!HeapTupleIsValid(tup)) + elog(ERROR, "subscription \"%s\" does not exist", MySubscription->name); + + LockSharedObject(SubscriptionRelationId, MySubscription->oid, 0, + AccessShareLock); + + /* Form a new tuple. */ + memset(values, 0, sizeof(values)); + memset(nulls, false, sizeof(nulls)); + memset(replaces, false, sizeof(replaces)); + + /* reset subretry */ + values[Anum_pg_subscription_subretry - 1] = BoolGetDatum(retry); + replaces[Anum_pg_subscription_subretry - 1] = true; + + tup = heap_modify_tuple(tup, RelationGetDescr(rel), values, nulls, + replaces); + + /* Update the catalog. */ + CatalogTupleUpdate(rel, &tup->t_self, tup); + + /* Cleanup. */ + heap_freetuple(tup); + table_close(rel, NoLock); + + if (started_tx) + CommitTransactionCommand(); +} diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 24927641b9..9ce774fc39 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -4471,8 +4471,9 @@ getSubscriptions(Archive *fout) ntups = PQntuples(res); /* - * Get subscription fields. We don't include subskiplsn in the dump as - * after restoring the dump this value may no longer be relevant. + * Get subscription fields. We don't include subskiplsn and subretry in + * the dump as after restoring the dump this value may no longer be + * relevant. */ i_tableoid = PQfnumber(res, "tableoid"); i_oid = PQfnumber(res, "oid"); diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h index d54540f5f5..5f4e058ec1 100644 --- a/src/include/catalog/pg_subscription.h +++ b/src/include/catalog/pg_subscription.h @@ -76,6 +76,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW bool subdisableonerr; /* True if a worker error should cause the * subscription to be disabled */ + bool subretry BKI_DEFAULT(f); /* True if the previous apply change failed. */ + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* Connection string to the publisher */ text subconninfo BKI_FORCE_NOT_NULL; @@ -116,6 +118,8 @@ typedef struct Subscription bool disableonerr; /* Indicates if the subscription should be * automatically disabled if a worker error * occurs */ + bool retry; /* Indicates if the previous apply change + * failed. */ char *conninfo; /* Connection string to the publisher */ char *slotname; /* Name of the replication slot */ char *synccommit; /* Synchronous commit setting for worker */ diff --git a/src/test/subscription/t/032_streaming_apply.pl b/src/test/subscription/t/032_streaming_apply.pl index eca4328676..1bcb4c65b7 100644 --- a/src/test/subscription/t/032_streaming_apply.pl +++ b/src/test/subscription/t/032_streaming_apply.pl @@ -57,8 +57,13 @@ $node_subscriber->safe_psql( $node_publisher->wait_for_catchup($appname); +# ============================================================================ # It is not allowed that the unique index on the publisher and the subscriber -# is different. Check the error reported by background worker in this case. +# is different. Check the error reported by background worker in this case. And +# after retrying in apply worker, we check if the data is replicated +# successfully. +# ============================================================================ + # First we check the unique index on normal table. $node_subscriber->safe_psql('postgres', "CREATE UNIQUE INDEX test_tab_b_idx ON test_tab (b)"); @@ -82,14 +87,15 @@ $node_subscriber->wait_for_log( qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab" in parallel mode/, $offset); -# Drop the unique index on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', "DROP INDEX test_tab_b_idx"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); my $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); -is($result, qq(5000), 'data replicated to subscriber after dropping index'); +is($result, qq(5000), 'data replicated to subscribers after retrying because of unique index'); + +# Drop the unique index on the subscriber. +$node_subscriber->safe_psql('postgres', "DROP INDEX test_tab_b_idx"); # Then we check the unique index on partition table. $node_subscriber->safe_psql('postgres', @@ -106,17 +112,20 @@ $node_subscriber->wait_for_log( qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab_partitioned" in parallel mode/, $offset); -# Drop the unique index on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', "DROP INDEX test_tab_b_partition_idx"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); -is($result, qq(5000), 'data replicated to subscriber after dropping index'); +is($result, qq(5000), 'data replicated to subscribers after retrying because of unique index'); + +# Drop the unique index on the subscriber. +$node_subscriber->safe_psql('postgres', "DROP INDEX test_tab_b_partition_idx"); # Triggers which execute non-immutable function are not allowed on the # subscriber side. Check the error reported by background worker in this case. +# And after retrying in apply worker, we check if the data is replicated +# successfully. # First we check the trigger function on normal table. $node_subscriber->safe_psql( 'postgres', qq{ @@ -140,15 +149,16 @@ $node_subscriber->wait_for_log( qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab" in parallel mode/, $offset); -# Drop the trigger on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "DROP TRIGGER insert_trig ON test_tab"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); -is($result, qq(0), 'data replicated to subscriber after dropping trigger'); +is($result, qq(0), 'data replicated to subscribers after retrying because of trigger'); + +# Drop the trigger on the subscriber. +$node_subscriber->safe_psql('postgres', + "DROP TRIGGER insert_trig ON test_tab"); # Then we check the trigger function on partition table. $node_subscriber->safe_psql( @@ -168,19 +178,24 @@ $node_subscriber->wait_for_log( qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab_partitioned" in parallel mode/, $offset); -# Drop the trigger on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "DROP TRIGGER insert_trig ON test_tab_partition"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); -is($result, qq(0), 'data replicated to subscriber after dropping trigger'); +is($result, qq(0), 'data replicated to subscribers after retrying because of trigger'); +# Drop the trigger on the subscriber. +$node_subscriber->safe_psql('postgres', + "DROP TRIGGER insert_trig ON test_tab_partition"); + +# ============================================================================ # It is not allowed that column default value expression contains a # non-immutable function on the subscriber side. Check the error reported by -# background worker in this case. +# background worker in this case. And after retrying in apply worker, we check +# if the data is replicated successfully. +# ============================================================================ + # First we check the column default value expression on normal table. $node_subscriber->safe_psql('postgres', "ALTER TABLE test_tab ALTER COLUMN b SET DEFAULT random()"); @@ -196,16 +211,17 @@ $node_subscriber->wait_for_log( qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab" in parallel mode/, $offset); -# Drop default value on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "ALTER TABLE test_tab ALTER COLUMN b DROP DEFAULT"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); is($result, qq(5000), - 'data replicated to subscriber after dropping default value expression'); + 'data replicated to subscribers after retrying because of column default value'); + +# Drop default value on the subscriber. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab ALTER COLUMN b DROP DEFAULT"); # Then we check the column default value expression on partition table. $node_subscriber->safe_psql('postgres', @@ -222,20 +238,25 @@ $node_subscriber->wait_for_log( qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab_partitioned" in parallel mode/, $offset); -# Drop default value on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "ALTER TABLE test_tab_partition ALTER COLUMN b DROP DEFAULT"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); is($result, qq(5000), - 'data replicated to subscriber after dropping default value expression'); + 'data replicated to subscribers after retrying because of column default value'); + +# Drop default value on the subscriber. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partition ALTER COLUMN b DROP DEFAULT"); +# ============================================================================ # It is not allowed that domain constraint expression contains a non-immutable # function on the subscriber side. Check the error reported by background -# worker in this case. +# worker in this case. And after retrying in apply worker, we check if the data +# is replicated successfully. +# ============================================================================ + # Because the column type of the partition table must be the same as its parent # table, only test normal table here. $node_subscriber->safe_psql( @@ -253,21 +274,26 @@ $node_subscriber->wait_for_log( qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab" in parallel mode/, $offset); -# Drop domain constraint expression on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "ALTER TABLE test_tab ALTER COLUMN a TYPE int"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); is($result, qq(0), - 'data replicated to subscriber after dropping domain constraint expression' + 'data replicated to subscribers after retrying because of domain' ); -# It is not allowed that constraint expression contains a non-immutable function -# on the subscriber side. Check the error reported by background worker in this -# case. +# Drop domain constraint expression on the subscriber. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab ALTER COLUMN a TYPE int"); + +# ============================================================================ +# It is not allowed that constraint expression contains a non-immutable +# function on the subscriber side. Check the error reported by background +# worker in this case. And after retrying in apply worker, we check if the data +# is replicated successfully. +# ============================================================================ + # First we check the constraint expression on normal table. $node_subscriber->safe_psql( 'postgres', qq{ @@ -285,16 +311,17 @@ $node_subscriber->wait_for_log( qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab" in parallel mode/, $offset); -# Drop constraint on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "ALTER TABLE test_tab DROP CONSTRAINT test_tab_con"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); is($result, qq(5000), - 'data replicated to subscriber after dropping constraint expression'); + 'data replicated to subscribers after retrying because of constraint'); + +# Drop constraint on the subscriber. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab DROP CONSTRAINT test_tab_con"); # Then we check the constraint expression on partition table. $node_subscriber->safe_psql( @@ -311,19 +338,24 @@ $node_subscriber->wait_for_log( qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab_partitioned" in parallel mode/, $offset); -# Drop constraint on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "ALTER TABLE test_tab_partition DROP CONSTRAINT test_tab_con"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); is($result, qq(0), - 'data replicated to subscriber after dropping constraint expression'); + 'data replicated to subscribers after retrying because of constraint'); +# Drop constraint on the subscriber. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partition DROP CONSTRAINT test_tab_con"); + +# ============================================================================ # It is not allowed that foreign key on the subscriber side. Check the error -# reported by background worker in this case. +# reported by background worker in this case. And after retrying in apply +# worker, we check if the data is replicated successfully. +# ============================================================================ + # First we check the foreign key on normal table. $node_publisher->safe_psql('postgres', "DELETE FROM test_tab"); $node_publisher->wait_for_catchup($appname); @@ -344,16 +376,17 @@ $node_subscriber->wait_for_log( qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab" in parallel mode/, $offset); -# Drop the foreign key constraint on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "ALTER TABLE test_tab DROP CONSTRAINT test_tabfk"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); is($result, qq(5000), - 'data replicated to subscriber after dropping the foreign key'); + 'data replicated to subscribers after retrying because of foreign key'); + +# Drop the foreign key constraint on the subscriber. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab DROP CONSTRAINT test_tabfk"); # Then we check the foreign key on partition table. $node_publisher->wait_for_catchup($appname); @@ -374,16 +407,17 @@ $node_subscriber->wait_for_log( qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab_partitioned" in parallel mode/, $offset); -# Drop the foreign key constraint on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "ALTER TABLE test_tab_partition DROP CONSTRAINT test_tab_patition_fk"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); is($result, qq(5000), - 'data replicated to subscriber after dropping the foreign key'); + 'data replicated to subscribers after retrying because of foreign key'); + +# Drop the foreign key constraint on the subscriber. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partition DROP CONSTRAINT test_tab_patition_fk"); $node_subscriber->stop; $node_publisher->stop; -- 2.23.0.windows.1 ^ permalink raw reply [nested|flat] 66+ messages in thread
* Re: Perform streaming logical transactions by background workers and parallel apply @ 2022-07-13 04:33 Peter Smith <[email protected]> parent: [email protected] <[email protected]> 0 siblings, 1 reply; 66+ messages in thread From: Peter Smith @ 2022-07-13 04:33 UTC (permalink / raw) To: [email protected] <[email protected]>; +Cc: Amit Kapila <[email protected]>; Masahiko Sawada <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; PostgreSQL Hackers <[email protected]> Below are my review comments for the v16* patch set: ======== v16-0001 ======== 1.0 <general> There are places (comments, docs, errmsgs, etc) in the patch referring to "parallel mode". I think every one of those references should be found and renamed to "parallel streaming mode" or "streaming=parallel" or at the very least match sure that "streaming" is in the same sentence. IMO it's too vague just saying "parallel" without also saying the context is for the "streaming" parameter. I have commented on some of those examples below, but please search everything anyway (including the docs) to catch the ones I haven't explicitly mentioned. ====== 1.1 src/backend/commands/subscriptioncmds.c +defGetStreamingMode(DefElem *def) +{ + /* + * If no value given, assume "true" is meant. + */ Please fix this comment to identical to this pushed patch [1] ====== 1.2 .../replication/logical/applybgworker.c - apply_bgworker_start + if (list_length(ApplyWorkersFreeList) > 0) + { + wstate = (ApplyBgworkerState *) llast(ApplyWorkersFreeList); + ApplyWorkersFreeList = list_delete_last(ApplyWorkersFreeList); + Assert(wstate->pstate->status == APPLY_BGWORKER_FINISHED); + } The Assert that the entries in the free-list are FINISHED seems like unnecessary checking. IIUC, code is already doing the Assert that entries are FINISHED before allowing them into the free-list in the first place. ~~~ 1.3 .../replication/logical/applybgworker.c - apply_bgworker_find + if (found) + { + char status = entry->wstate->pstate->status; + + /* If any workers (or the postmaster) have died, we have failed. */ + if (status == APPLY_BGWORKER_EXIT) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("background worker %u failed to apply transaction %u", + entry->wstate->pstate->n, + entry->wstate->pstate->stream_xid))); + + Assert(status == APPLY_BGWORKER_BUSY); + + return entry->wstate; + } Why not remove that Assert but change the condition to be: if (status != APPLY_BGWORKER_BUSY) ereport(...) ====== 1.4 src/backend/replication/logical/proto.c - logicalrep_write_stream_abort @@ -1163,31 +1163,56 @@ logicalrep_read_stream_commit(StringInfo in, LogicalRepCommitData *commit_data) /* * Write STREAM ABORT to the output stream. Note that xid and subxid will be * same for the top-level transaction abort. + * + * If write_abort_lsn is true, send the abort_lsn and abort_time fields. + * Otherwise not. */ "Otherwise not." -> ", otherwise don't." ~~~ 1.5 src/backend/replication/logical/proto.c - logicalrep_read_stream_abort + * + * If read_abort_lsn is true, try to read the abort_lsn and abort_time fields. + * Otherwise not. */ void -logicalrep_read_stream_abort(StringInfo in, TransactionId *xid, - TransactionId *subxid) +logicalrep_read_stream_abort(StringInfo in, + LogicalRepStreamAbortData *abort_data, + bool read_abort_lsn) "Otherwise not." -> ", otherwise don't." ====== 1.6 src/backend/replication/logical/worker.c - file comment + * If streaming = parallel, We assign a new apply background worker (if + * available) as soon as the xact's first stream is received. The main apply "We" -> "we" ... or maybe better just remove it completely. ~~~ 1.7 src/backend/replication/logical/worker.c - apply_handle_stream_prepare + /* + * After sending the data to the apply background worker, wait for + * that worker to finish. This is necessary to maintain commit + * order which avoids failures due to transaction dependencies and + * deadlocks. + */ + apply_bgworker_send_data(wstate, s->len, s->data); + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_FINISHED); + apply_bgworker_free(wstate); The comment should be changed how you had suggested [2], so that it will be formatted the same way as a couple of other similar comments. ~~~ 1.8 src/backend/replication/logical/worker.c - apply_handle_stream_abort + /* Check whether the publisher sends abort_lsn and abort_time. */ + if (am_apply_bgworker()) + read_abort_lsn = MyParallelState->server_version >= 160000; This is handling decisions about read/write of the protocol bytes. I think feel like it will be better to be checking the server *protocol* version (not the server postgres version) to make this decision – e.g. this code should be using the new macro you introduced so it will end up looking much like how the pgoutput_stream_abort code is doing it. ~~~ 1.9 src/backend/replication/logical/worker.c - store_flush_position @@ -2636,6 +2999,10 @@ store_flush_position(XLogRecPtr remote_lsn) { FlushPosition *flushpos; + /* We only need to collect the LSN in main apply worker */ + if (am_apply_bgworker()) + return; + SUGGESTION /* Skip if not the main apply worker */ ====== 1.10 src/backend/replication/pgoutput/pgoutput.c @@ -1820,6 +1820,8 @@ pgoutput_stream_abort(struct LogicalDecodingContext *ctx, XLogRecPtr abort_lsn) { ReorderBufferTXN *toptxn; + bool write_abort_lsn = false; + PGOutputData *data = (PGOutputData *) ctx->output_plugin_private; /* * The abort should happen outside streaming block, even for streamed @@ -1832,8 +1834,13 @@ pgoutput_stream_abort(struct LogicalDecodingContext *ctx, Assert(rbtxn_is_streamed(toptxn)); + /* We only send abort_lsn and abort_time if the subscriber needs them. */ + if (data->protocol_version >= LOGICALREP_PROTO_STREAM_PARALLEL_VERSION_NUM) + write_abort_lsn = true; + IMO it's simpler to remove the declaration default assignment, and instead this code can be written as: write_abort_lsn = data->protocol_version >= LOGICALREP_PROTO_STREAM_PARALLEL_VERSION_NUM; ====== 1.11 src/include/replication/logicalproto.h + * + * LOGICALREP_PROTO_STREAM_PARALLEL_VERSION_NUM is the minimum protocol version + * with support for streaming large transactions in apply background worker. + * Introduced in PG16. "in apply background worker" -> "using apply background workers" ~~~ 1.12 +extern void logicalrep_read_stream_abort(StringInfo in, + LogicalRepStreamAbortData *abort_data, + bool include_abort_lsn); I think the "include_abort_lsn" is now renamed to "include_abort_lsn". ======== v16-0002 ======== No comments. ======== v16-0003 ======== 3.0 <general> Same comment about "parallel mode" as in comment #1.0 ====== 3.1 doc/src/sgml/ref/create_subscription.sgml + the publisher-side; 2) there cannot be any non-immutable functions + in the subscriber-side replicated table. The functions are not table data so maybe it's better to say "functions in the ..." -> "functions used by the ...". If you change this then there are equivalent comments and commit messages that should change to match it. ====== 3.2 .../replication/logical/applybgworker.c - apply_bgworker_relation_check + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot replicate target relation \"%s.%s\" in parallel " + "mode", rel->remoterel.nspname, rel->remoterel.relname), + errdetail("The unique column on subscriber is not the unique " + "column on publisher or there is at least one " + "non-immutable function."), + errhint("Please change the streaming option to 'on' instead of 'parallel'."))); 3.2a SUGGESTED errmsg "cannot replicate target relation \"%s.%s\" using subscription parameter streaming=parallel" 3.2b SUGGESTED errhint "Please change to use subscription parameter streaming=on" 3.3 The errcode seems the wrong one. Perhaps it should be ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE. ====== 3.4 src/backend/replication/logical/proto.c - logicalrep_write_attrs In [3] you wrote: I think the file relcache.c should contain cache-build operations, and the code I added doesn't have this operation. So I didn't change. But I only gave relcache.c as an example. It can also be a new static function in this same file, but anyway I still think this big slab of code might be better if not done inline in logicalrep_write_attrs. ~~~ 3.5 src/backend/replication/logical/proto.c - logicalrep_read_attrs @@ -1012,11 +1062,14 @@ logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel) { uint8 flags; - /* Check for replica identity column */ + /* Check for replica identity and unique column */ flags = pq_getmsgbyte(in); - if (flags & LOGICALREP_IS_REPLICA_IDENTITY) + if (flags & ATTR_IS_REPLICA_IDENTITY) attkeys = bms_add_member(attkeys, i); + if (flags & ATTR_IS_UNIQUE) + attunique = bms_add_member(attunique, i); The code comment really applies to all 3 statements so maybe better not to have the blank line here. ====== 3.6 src/backend/replication/logical/relation.c - logicalrep_rel_mark_parallel 3.6.a + /* Fast path if we marked 'parallel' flag. */ + if (entry->parallel != PARALLEL_APPLY_UNKNOWN) + return; SUGGESTED Fast path if 'parallel' flag is already known. ~ 3.6.b + /* Initialize the flag. */ + entry->parallel = PARALLEL_APPLY_SAFE; I think it makes more sense if assigning SAFE is the very *last* thing this function does – not the first thing. ~ 3.6.c + /* + * First, we check if the unique column in the relation on the + * subscriber-side is also the unique column on the publisher-side. + */ "First, we check..." -> "First, check..." ~ 3.6.d + /* + * Then, We check if there is any non-immutable function in the local + * table. Look for functions in the following places: "Then, We check..." -> "Then, check" ~~~ 3.7 src/backend/replication/logical/relation.c - logicalrep_rel_mark_parallel ^ permalink raw reply [nested|flat] 66+ messages in thread
* RE: Perform streaming logical transactions by background workers and parallel apply @ 2022-07-13 05:48 [email protected] <[email protected]> parent: [email protected] <[email protected]> 0 siblings, 0 replies; 66+ messages in thread From: [email protected] @ 2022-07-13 05:48 UTC (permalink / raw) To: [email protected] <[email protected]>; Amit Kapila <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Peter Smith <[email protected]>; [email protected] <[email protected]>; PostgreSQL Hackers <[email protected]> On Fri, Jul 7, 2022 at 11:32 AM Shi, Yu/侍 雨 <[email protected]> wrote: > Thanks for updating the patch. > > Here are some comments. Thanks for your comments. > 0001 patch > ============== > 1. > + /* Check If there are free worker slot(s) */ > + LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); > > I think "Check If" should be "Check if". Fixed. > 0003 patch > ============== > 1. > Should we call apply_bgworker_relation_check() in apply_handle_truncate()? Because TRUNCATE blocks all other operations on the table, I think that when two transactions on the publisher-side operate on the same table, at least one of them will be blocked. So I think for this case the blocking will happen on the publisher-side. > 0004 patch > ============== > 1. > @@ -3932,6 +3958,9 @@ start_apply(XLogRecPtr origin_startpos) > } > PG_CATCH(); > { > + /* Set the flag that we will retry later. */ > + set_subscription_retry(true); > + > if (MySubscription->disableonerr) > DisableSubscriptionAndExit(); > Else > > I think we need to emit the error and recover from the error state before > setting the retry flag, like what we do in DisableSubscriptionAndExit(). > Otherwise if an error is detected when setting the retry flag, we won't get the > error message reported by the apply worker. You are right. I fixed this point as you suggested. (I moved the operations you mentioned from the function DisableSubscriptionAndExit to before setting the retry flag.) I also made a similar modification in the function start_table_sync. Attach the news patches. Regards, Wang wei Attachments: [application/octet-stream] v17-0001-Perform-streaming-logical-transactions-by-backgr.patch (105.9K, ../../OSZPR01MB6278023241CEDDE233C8386F9E899@OSZPR01MB6278.jpnprd01.prod.outlook.com/2-v17-0001-Perform-streaming-logical-transactions-by-backgr.patch) download | inline diff: From 7b6823065f3b8fee99a986a3910931d43493c494 Mon Sep 17 00:00:00 2001 From: "houzj.fnst" <[email protected]> Date: Wed, 20 Apr 2022 16:45:07 +0800 Subject: [PATCH v17 1/4] Perform streaming logical transactions by background workers Currently, for large transactions, the publisher sends the data in multiple streams (changes divided into chunks depending upon logical_decoding_work_mem), and then on the subscriber-side, the apply worker writes the changes into temporary files and once it receives the commit, it reads from the file and applies the entire transaction. To improve the performance of such transactions, we can instead allow them to be applied via background workers. In this approach, we assign a new apply background worker (if available) as soon as the xact's first stream is received and the main apply worker will send changes to this new worker via shared memory. The apply background worker will directly apply the change instead of writing it to temporary files. We keep this worker assigned till the transaction commit is received and also wait for the worker to finish at commit. This preserves commit ordering and avoids writing to and reading from file in most cases. We still need to spill if there is no worker available. This patch also extends the SUBSCRIPTION 'streaming' parameter so that the user can control whether to apply the streaming transaction in an apply background worker or spill the change to disk. The user can set the streaming parameter to 'on/off', 'parallel'. The parameter value 'parallel' means the streaming will be applied via an apply background worker, if available. The parameter value 'on' means the streaming transaction will be spilled to disk. The default value is 'off' (same as current behaviour). --- doc/src/sgml/catalogs.sgml | 10 +- doc/src/sgml/config.sgml | 25 + doc/src/sgml/logical-replication.sgml | 10 + doc/src/sgml/protocol.sgml | 19 + doc/src/sgml/ref/create_subscription.sgml | 24 +- src/backend/access/transam/xact.c | 13 + src/backend/commands/subscriptioncmds.c | 66 +- src/backend/postmaster/bgworker.c | 3 + src/backend/replication/logical/Makefile | 1 + .../replication/logical/applybgworker.c | 786 ++++++++++++++++++ src/backend/replication/logical/decode.c | 10 +- src/backend/replication/logical/launcher.c | 130 ++- src/backend/replication/logical/origin.c | 26 +- src/backend/replication/logical/proto.c | 41 +- .../replication/logical/reorderbuffer.c | 10 +- src/backend/replication/logical/tablesync.c | 10 +- src/backend/replication/logical/worker.c | 688 +++++++++++---- src/backend/replication/pgoutput/pgoutput.c | 9 +- src/backend/utils/activity/wait_event.c | 3 + src/backend/utils/misc/guc.c | 12 + src/backend/utils/misc/postgresql.conf.sample | 1 + src/bin/pg_dump/pg_dump.c | 6 +- src/include/catalog/pg_subscription.h | 21 +- src/include/replication/logicallauncher.h | 1 + src/include/replication/logicalproto.h | 27 +- src/include/replication/logicalworker.h | 1 + src/include/replication/origin.h | 2 +- src/include/replication/reorderbuffer.h | 7 +- src/include/replication/worker_internal.h | 102 ++- src/include/utils/wait_event.h | 1 + src/test/regress/expected/subscription.out | 2 +- src/tools/pgindent/typedefs.list | 5 + 32 files changed, 1852 insertions(+), 220 deletions(-) create mode 100644 src/backend/replication/logical/applybgworker.c diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index 4f3f375a84..815cae6082 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -7873,11 +7873,15 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l <row> <entry role="catalog_table_entry"><para role="column_definition"> - <structfield>substream</structfield> <type>bool</type> + <structfield>substream</structfield> <type>char</type> </para> <para> - If true, the subscription will allow streaming of in-progress - transactions + Controls how to handle the streaming of in-progress transactions: + <literal>f</literal> = disallow streaming of in-progress transactions, + <literal>t</literal> = spill the changes of in-progress transactions to + disk and apply at once after the transaction is committed on the + publisher, + <literal>p</literal> = apply changes directly using a background worker </para></entry> </row> diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml index 37fd80388c..6bbd986195 100644 --- a/doc/src/sgml/config.sgml +++ b/doc/src/sgml/config.sgml @@ -4970,6 +4970,31 @@ ANY <replaceable class="parameter">num_sync</replaceable> ( <replaceable class=" </listitem> </varlistentry> + <varlistentry id="guc-max-apply-bgworkers-per-subscription" xreflabel="max_apply_bgworkers_per_subscription"> + <term><varname>max_apply_bgworkers_per_subscription</varname> (<type>integer</type>) + <indexterm> + <primary><varname>max_apply_bgworkers_per_subscription</varname> configuration parameter</primary> + </indexterm> + </term> + <listitem> + <para> + Maximum number of apply background workers per subscription. This + parameter controls the amount of parallelism of the streaming of + in-progress transactions when subscription parameter + <literal>streaming = parallel</literal>. + </para> + <para> + The apply background workers are taken from the pool defined by + <varname>max_logical_replication_workers</varname>. + </para> + <para> + The default value is 2. This parameter can only be set in the + <filename>postgresql.conf</filename> file or on the server command + line. + </para> + </listitem> + </varlistentry> + </variablelist> </sect2> diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml index bdf1e7b727..92997f9299 100644 --- a/doc/src/sgml/logical-replication.sgml +++ b/doc/src/sgml/logical-replication.sgml @@ -1153,6 +1153,16 @@ CONTEXT: processing remote data for replication origin "pg_16395" during "INSER might not violate any constraint. This can easily make the subscriber inconsistent. </para> + + <para> + When the streaming mode is <literal>parallel</literal>, the finish LSN of + failed transactions may not be logged. In that case, it may be necessary to + change the streaming mode to <literal>on</literal> and cause the same + conflicts again so the finish LSN of the failed transaction will be written + to the server log. For the usage of finish LSN, please refer to <link + linkend="sql-altersubscription"><command>ALTER SUBSCRIPTION ... + SKIP</command></link>. + </para> </sect1> <sect1 id="logical-replication-restrictions"> diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml index c0b89a3c01..7e88ba9631 100644 --- a/doc/src/sgml/protocol.sgml +++ b/doc/src/sgml/protocol.sgml @@ -6809,6 +6809,25 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;" </listitem> </varlistentry> + <varlistentry> + <term>Int64 (XLogRecPtr)</term> + <listitem> + <para> + The LSN of the abort. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Int64 (TimestampTz)</term> + <listitem> + <para> + Abort timestamp of the transaction. The value is in number + of microseconds since PostgreSQL epoch (2000-01-01). + </para> + </listitem> + </varlistentry> + <varlistentry> <term>Int32 (TransactionId)</term> <listitem> diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml index 34b3264b26..71dd4aca81 100644 --- a/doc/src/sgml/ref/create_subscription.sgml +++ b/doc/src/sgml/ref/create_subscription.sgml @@ -217,13 +217,29 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl </varlistentry> <varlistentry> - <term><literal>streaming</literal> (<type>boolean</type>)</term> + <term><literal>streaming</literal> (<type>enum</type>)</term> <listitem> <para> Specifies whether to enable streaming of in-progress transactions - for this subscription. By default, all transactions - are fully decoded on the publisher and only then sent to the - subscriber as a whole. + for this subscription. The default value is <literal>off</literal>, + meaning all transactions are fully decoded on the publisher and only + then sent to the subscriber as a whole. + </para> + + <para> + If set to <literal>on</literal>, the incoming changes are written to + temporary files and then applied only after the transaction is + committed on the publisher. + </para> + + <para> + If set to <literal>parallel</literal>, incoming changes are directly + applied via one of the apply background workers, if available. If no + background worker is free to handle streaming transaction then the + changes are written to temporary files and applied after the + transaction is committed. Note that if an error happens when + applying changes in a background worker, the finish LSN of the + remote transaction might not be reported in the server log. </para> </listitem> </varlistentry> diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c index 116de1175b..3e61a57b50 100644 --- a/src/backend/access/transam/xact.c +++ b/src/backend/access/transam/xact.c @@ -1711,6 +1711,7 @@ RecordTransactionAbort(bool isSubXact) int nchildren; TransactionId *children; TimestampTz xact_time; + bool replorigin; /* * If we haven't been assigned an XID, nobody will care whether we aborted @@ -1741,6 +1742,13 @@ RecordTransactionAbort(bool isSubXact) elog(PANIC, "cannot abort transaction %u, it was already committed", xid); + /* + * Are we using the replication origins feature? Or, in other words, + * are we replaying remote actions? + */ + replorigin = (replorigin_session_origin != InvalidRepOriginId && + replorigin_session_origin != DoNotReplicateId); + /* Fetch the data we need for the abort record */ nrels = smgrGetPendingDeletes(false, &rels); nchildren = xactGetCommittedChildren(&children); @@ -1765,6 +1773,11 @@ RecordTransactionAbort(bool isSubXact) MyXactFlags, InvalidTransactionId, NULL); + if (replorigin) + /* Move LSNs forward for this replication origin */ + replorigin_session_advance(replorigin_session_origin_lsn, + XactLastRecEnd); + /* * Report the latest async abort LSN, so that the WAL writer knows to * flush this abort. There's nothing to be gained by delaying this, since diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c index bdc1208724..d4b2616ee6 100644 --- a/src/backend/commands/subscriptioncmds.c +++ b/src/backend/commands/subscriptioncmds.c @@ -83,7 +83,7 @@ typedef struct SubOpts bool copy_data; bool refresh; bool binary; - bool streaming; + char streaming; bool twophase; bool disableonerr; XLogRecPtr lsn; @@ -95,6 +95,62 @@ static List *merge_publications(List *oldpublist, List *newpublist, bool addpub, static void ReportSlotConnectionError(List *rstates, Oid subid, char *slotname, char *err); +/* + * Extract the streaming mode value from a DefElem. This is like + * defGetBoolean() but also accepts the special value of "parallel". + */ +static char +defGetStreamingMode(DefElem *def) +{ + /* + * If no parameter value given, assume "true" is meant. + */ + if (def->arg == NULL) + return SUBSTREAM_ON; + + /* + * Allow 0, 1, "false", "true", "off", "on" or "parallel". + */ + switch (nodeTag(def->arg)) + { + case T_Integer: + switch (intVal(def->arg)) + { + case 0: + return SUBSTREAM_OFF; + case 1: + return SUBSTREAM_ON; + default: + /* otherwise, error out below */ + break; + } + break; + default: + { + char *sval = defGetString(def); + + /* + * The set of strings accepted here should match up with the + * grammar's opt_boolean_or_string production. + */ + if (pg_strcasecmp(sval, "false") == 0 || + pg_strcasecmp(sval, "off") == 0) + return SUBSTREAM_OFF; + if (pg_strcasecmp(sval, "true") == 0 || + pg_strcasecmp(sval, "on") == 0) + return SUBSTREAM_ON; + if (pg_strcasecmp(sval, "parallel") == 0) + return SUBSTREAM_PARALLEL; + } + break; + } + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s requires a Boolean value or \"parallel\"", + def->defname))); + return SUBSTREAM_OFF; /* keep compiler quiet */ +} + /* * Common option parsing function for CREATE and ALTER SUBSCRIPTION commands. * @@ -132,7 +188,7 @@ parse_subscription_options(ParseState *pstate, List *stmt_options, if (IsSet(supported_opts, SUBOPT_BINARY)) opts->binary = false; if (IsSet(supported_opts, SUBOPT_STREAMING)) - opts->streaming = false; + opts->streaming = SUBSTREAM_OFF; if (IsSet(supported_opts, SUBOPT_TWOPHASE_COMMIT)) opts->twophase = false; if (IsSet(supported_opts, SUBOPT_DISABLE_ON_ERR)) @@ -233,7 +289,7 @@ parse_subscription_options(ParseState *pstate, List *stmt_options, errorConflictingDefElem(defel, pstate); opts->specified_opts |= SUBOPT_STREAMING; - opts->streaming = defGetBoolean(defel); + opts->streaming = defGetStreamingMode(defel); } else if (strcmp(defel->defname, "two_phase") == 0) { @@ -600,7 +656,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt, values[Anum_pg_subscription_subowner - 1] = ObjectIdGetDatum(owner); values[Anum_pg_subscription_subenabled - 1] = BoolGetDatum(opts.enabled); values[Anum_pg_subscription_subbinary - 1] = BoolGetDatum(opts.binary); - values[Anum_pg_subscription_substream - 1] = BoolGetDatum(opts.streaming); + values[Anum_pg_subscription_substream - 1] = CharGetDatum(opts.streaming); values[Anum_pg_subscription_subtwophasestate - 1] = CharGetDatum(opts.twophase ? LOGICALREP_TWOPHASE_STATE_PENDING : @@ -1059,7 +1115,7 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt, if (IsSet(opts.specified_opts, SUBOPT_STREAMING)) { values[Anum_pg_subscription_substream - 1] = - BoolGetDatum(opts.streaming); + CharGetDatum(opts.streaming); replaces[Anum_pg_subscription_substream - 1] = true; } diff --git a/src/backend/postmaster/bgworker.c b/src/backend/postmaster/bgworker.c index 40601aefd9..40ccb8993c 100644 --- a/src/backend/postmaster/bgworker.c +++ b/src/backend/postmaster/bgworker.c @@ -128,6 +128,9 @@ static const struct }, { "ApplyWorkerMain", ApplyWorkerMain + }, + { + "ApplyBgworkerMain", ApplyBgworkerMain } }; diff --git a/src/backend/replication/logical/Makefile b/src/backend/replication/logical/Makefile index c4e2fdeb71..cbfb5d794e 100644 --- a/src/backend/replication/logical/Makefile +++ b/src/backend/replication/logical/Makefile @@ -15,6 +15,7 @@ include $(top_builddir)/src/Makefile.global override CPPFLAGS := -I$(srcdir) $(CPPFLAGS) OBJS = \ + applybgworker.o \ decode.o \ launcher.o \ logical.o \ diff --git a/src/backend/replication/logical/applybgworker.c b/src/backend/replication/logical/applybgworker.c new file mode 100644 index 0000000000..490633f37c --- /dev/null +++ b/src/backend/replication/logical/applybgworker.c @@ -0,0 +1,786 @@ +/*------------------------------------------------------------------------- + * applybgworker.c + * Support routines for applying xact by apply background worker + * + * Copyright (c) 2016-2022, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/replication/logical/applybgworker.c + * + * This file contains routines that are intended to support setting up, using, + * and tearing down a ApplyBgworkerState. + * + * Refer to the comments in file header of logical/worker.c to see more + * information about apply background worker. + * + *------------------------------------------------------------------------- + */ + +#include "postgres.h" + +#include "libpq/pqformat.h" +#include "mb/pg_wchar.h" +#include "pgstat.h" +#include "postmaster/interrupt.h" +#include "replication/logicallauncher.h" +#include "replication/logicalworker.h" +#include "replication/origin.h" +#include "replication/walreceiver.h" +#include "replication/worker_internal.h" +#include "storage/ipc.h" +#include "storage/procarray.h" +#include "tcop/tcopprot.h" +#include "utils/inval.h" +#include "utils/memutils.h" +#include "utils/resowner.h" +#include "utils/syscache.h" + +#define PG_LOGICAL_APPLY_SHM_MAGIC 0x79fb2447 // TODO Consider change + +/* + * DSM keys for apply background worker. Unlike other parallel execution code, + * since we don't need to worry about DSM keys conflicting with plan_node_id we + * can use small integers. + */ +#define APPLY_BGWORKER_KEY_SHARED 1 +#define APPLY_BGWORKER_KEY_MQ 2 + +/* Queue size of DSM, 16 MB for now. */ +#define DSM_QUEUE_SIZE 160000000 + +/* + * There are three fields in message: start_lsn, end_lsn and send_time. Because + * we have updated these statistics in apply worker, we could ignore these + * fields in apply background worker. (see function LogicalRepApplyLoop) + */ +#define IGNORE_SIZE_IN_MESSAGE (3 * sizeof(uint64)) + +/* + * Entry for a hash table we use to map from xid to our apply background worker + * state. + */ +typedef struct ApplyBgworkerEntry +{ + TransactionId xid; + ApplyBgworkerState *wstate; +} ApplyBgworkerEntry; + +/* Apply background workers hash table (initialized on first use) */ +static HTAB *ApplyWorkersHash = NULL; +static List *ApplyWorkersFreeList = NIL; +static List *ApplyWorkersList = NIL; + +/* + * Information shared between main apply worker and apply background worker. + */ +volatile ApplyBgworkerShared *MyParallelState = NULL; + +List *subxactlist = NIL; + +static bool apply_bgworker_can_start(TransactionId xid); +static ApplyBgworkerState *apply_bgworker_setup(void); +static void apply_bgworker_setup_dsm(ApplyBgworkerState *wstate); + +/* + * Check if starting a new apply background worker is allowed. + */ +static bool +apply_bgworker_can_start(TransactionId xid) +{ + if (!TransactionIdIsValid(xid)) + return false; + + /* + * Don't start a new background worker if not in streaming parallel mode. + */ + if (MySubscription->stream != SUBSTREAM_PARALLEL) + return false; + + /* + * Don't start a new background worker if user has set skiplsn as it's + * possible that user want to skip the streaming transaction. For + * streaming transaction, we need to spill the transaction to disk so that + * we can get the last LSN of the transaction to judge whether to skip + * before starting to apply the change. + */ + if (!XLogRecPtrIsInvalid(MySubscription->skiplsn)) + return false; + + /* + * For streaming transactions that are being applied in apply background + * worker, we cannot decide whether to apply the change for a relation + * that is not in the READY state (see should_apply_changes_for_rel) as we + * won't know remote_final_lsn by that time. So, we don't start new apply + * background worker in this case. + */ + if (!AllTablesyncsReady()) + return false; + + return true; +} + +/* + * Try to start an apply background worker and, if successful, cache it in + * ApplyWorkersHash keyed by the specified xid. + */ +ApplyBgworkerState * +apply_bgworker_start(TransactionId xid) +{ + bool found; + ApplyBgworkerState *wstate; + ApplyBgworkerEntry *entry = NULL; + + if (!apply_bgworker_can_start(xid)) + return NULL; + + /* First time through, initialize apply workers hashtable */ + if (ApplyWorkersHash == NULL) + { + HASHCTL ctl; + + MemSet(&ctl, 0, sizeof(ctl)); + ctl.keysize = sizeof(TransactionId); + ctl.entrysize = sizeof(ApplyBgworkerEntry); + ctl.hcxt = ApplyContext; + + ApplyWorkersHash = hash_create("logical apply workers hash", 8, &ctl, + HASH_ELEM | HASH_BLOBS | HASH_CONTEXT); + } + + /* + * Now, we try to get an apply background worker. If there is at least one + * worker in the free list, then take one. Otherwise, we try to start a + * new apply background worker. + */ + if (list_length(ApplyWorkersFreeList) > 0) + { + wstate = (ApplyBgworkerState *) llast(ApplyWorkersFreeList); + ApplyWorkersFreeList = list_delete_last(ApplyWorkersFreeList); + Assert(wstate->pstate->status == APPLY_BGWORKER_FINISHED); + } + else + { + wstate = apply_bgworker_setup(); + + if (wstate == NULL) + return NULL; + } + + /* + * Create entry for requested transaction. + */ + entry = hash_search(ApplyWorkersHash, &xid, HASH_ENTER, &found); + if (found) + elog(ERROR, "hash table corrupted"); + + /* Fill up the hash entry */ + wstate->pstate->status = APPLY_BGWORKER_BUSY; + wstate->pstate->server_version = walrcv_server_version(LogRepWorkerWalRcvConn); + wstate->pstate->stream_xid = xid; + entry->wstate = wstate; + entry->xid = xid; + + return wstate; +} + +/* + * Try to look up worker inside ApplyWorkersHash for requested xid. + */ +ApplyBgworkerState * +apply_bgworker_find(TransactionId xid) +{ + bool found; + ApplyBgworkerEntry *entry = NULL; + + if (!TransactionIdIsValid(xid)) + return NULL; + + if (ApplyWorkersHash == NULL) + return NULL; + + /* + * Find entry for requested transaction. + */ + entry = hash_search(ApplyWorkersHash, &xid, HASH_FIND, &found); + if (found) + { + char status = entry->wstate->pstate->status; + + /* If any workers (or the postmaster) have died, we have failed. */ + if (status == APPLY_BGWORKER_EXIT) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("background worker %u failed to apply transaction %u", + entry->wstate->pstate->n, + entry->wstate->pstate->stream_xid))); + + Assert(status == APPLY_BGWORKER_BUSY); + + return entry->wstate; + } + else + return NULL; +} + +/* + * Add the worker to the free list and remove the entry from the hash table. + */ +void +apply_bgworker_free(ApplyBgworkerState *wstate) +{ + MemoryContext oldctx; + TransactionId xid = wstate->pstate->stream_xid; + + Assert(wstate->pstate->status == APPLY_BGWORKER_FINISHED); + + oldctx = MemoryContextSwitchTo(ApplyContext); + + hash_search(ApplyWorkersHash, &xid, HASH_REMOVE, NULL); + + elog(DEBUG1, "adding finished apply worker #%u for xid %u to the free list", + wstate->pstate->n, wstate->pstate->stream_xid); + + ApplyWorkersFreeList = lappend(ApplyWorkersFreeList, wstate); + + MemoryContextSwitchTo(oldctx); +} + +/* Apply Background Worker main loop */ +static void +LogicalApplyBgwLoop(shm_mq_handle *mqh, volatile ApplyBgworkerShared *pst) +{ + shm_mq_result shmq_res; + PGPROC *registrant; + ErrorContextCallback errcallback; + + registrant = BackendPidGetProc(MyBgworkerEntry->bgw_notify_pid); + SetLatch(®istrant->procLatch); + + /* + * Push apply error context callback. Fields will be filled applying a + * change. + */ + errcallback.callback = apply_error_callback; + errcallback.previous = error_context_stack; + error_context_stack = &errcallback; + + for (;;) + { + void *data; + Size len; + int c; + StringInfoData s; + MemoryContext oldctx; + + CHECK_FOR_INTERRUPTS(); + + /* Ensure we are reading the data into our memory context. */ + oldctx = MemoryContextSwitchTo(ApplyMessageContext); + + shmq_res = shm_mq_receive(mqh, &len, &data, false); + + if (shmq_res != SHM_MQ_SUCCESS) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("lost connection to the main apply worker"))); + + if (len == 0) + break; + + s.cursor = 0; + s.maxlen = -1; + s.data = (char *) data; + s.len = len; + + /* + * We use first byte of message for additional communication between + * main Logical replication worker and apply background workers, so if + * it differs from 'w', then process it first. + */ + c = pq_getmsgbyte(&s); + switch (c) + { + /* End message of streaming chunk */ + case LOGICAL_REP_MSG_STREAM_STOP: + elog(DEBUG1, "[Apply BGW #%u] ended processing streaming chunk," + "waiting on shm_mq_receive", pst->n); + + in_streamed_transaction = false; + pgstat_report_activity(STATE_IDLEINTRANSACTION, NULL); + continue; + case 'w': + break; + default: + elog(ERROR, "[Apply BGW #%u] unexpected message \"%c\"", + pst->n, c); + break; + } + + /* Ignore statistics fields that have been updated. */ + s.cursor += IGNORE_SIZE_IN_MESSAGE; + + apply_dispatch(&s); + + if (ConfigReloadPending) + { + ConfigReloadPending = false; + ProcessConfigFile(PGC_SIGHUP); + } + + MemoryContextSwitchTo(oldctx); + MemoryContextReset(ApplyMessageContext); + } + + MemoryContextSwitchTo(TopMemoryContext); + MemoryContextReset(ApplyContext); + + /* Pop the error context stack */ + error_context_stack = errcallback.previous; + + elog(DEBUG1, "[Apply BGW #%u] exiting", pst->n); + + /* Signal main process that we are done. */ + SetLatch(®istrant->procLatch); +} + +/* + * Set the exit status so that the main apply worker can realize we have + * shutdown. + */ +static void +apply_bgworker_shutdown(int code, Datum arg) +{ + SpinLockAcquire(&MyParallelState->mutex); + MyParallelState->status = APPLY_BGWORKER_EXIT; + SpinLockRelease(&MyParallelState->mutex); + + dsm_detach((dsm_segment *) DatumGetPointer(arg)); +} + +/* + * Apply Background Worker entry point + */ +void +ApplyBgworkerMain(Datum main_arg) +{ + volatile ApplyBgworkerShared *pst; + + dsm_handle handle; + dsm_segment *seg; + shm_toc *toc; + shm_mq *mq; + shm_mq_handle *mqh; + MemoryContext oldcontext; + RepOriginId originid; + int worker_slot = DatumGetInt32(main_arg); + char originname[NAMEDATALEN]; + + MemoryContextSwitchTo(TopMemoryContext); + + /* Init the memory context for the apply background worker to work in. */ + ApplyContext = AllocSetContextCreate(TopMemoryContext, + "ApplyContext", + ALLOCSET_DEFAULT_SIZES); + + /* + * Init the ApplyMessageContext which we clean up after each replication + * protocol message. + */ + ApplyMessageContext = AllocSetContextCreate(ApplyContext, + "ApplyMessageContext", + ALLOCSET_DEFAULT_SIZES); + + /* Setup signal handling */ + pqsignal(SIGHUP, SignalHandlerForConfigReload); + pqsignal(SIGTERM, die); + BackgroundWorkerUnblockSignals(); + + /* + * Connect to the dynamic shared memory segment. + * + * The backend that registered this worker passed us the ID of a shared + * memory segment to which we must attach for further instructions. In + * order to attach to dynamic shared memory, we need a resource owner. + * Once we've mapped the segment in our address space, attach to the table + * of contents so we can locate the various data structures we'll need to + * find within the segment. + */ + CurrentResourceOwner = ResourceOwnerCreate(NULL, "Logical apply worker"); + memcpy(&handle, MyBgworkerEntry->bgw_extra, sizeof(dsm_handle)); + seg = dsm_attach(handle); + if (seg == NULL) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("unable to map dynamic shared memory segment"))); + toc = shm_toc_attach(PG_LOGICAL_APPLY_SHM_MAGIC, dsm_segment_address(seg)); + if (toc == NULL) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("bad magic number in dynamic shared memory segment"))); + + before_shmem_exit(apply_bgworker_shutdown, PointerGetDatum(seg)); + + /* Look up the parallel state. */ + pst = shm_toc_lookup(toc, APPLY_BGWORKER_KEY_SHARED, false); + MyParallelState = pst; + + /* + * Attach to the message queue. + */ + mq = shm_toc_lookup(toc, APPLY_BGWORKER_KEY_MQ, false); + shm_mq_set_receiver(mq, MyProc); + mqh = shm_mq_attach(mq, seg, NULL); + + /* Run as replica session replication role. */ + SetConfigOption("session_replication_role", "replica", + PGC_SUSET, PGC_S_OVERRIDE); + + /* + * Now, we have initialized DSM. Attach to slot. + */ + logicalrep_worker_attach(worker_slot); + + /* Connect to our database. */ + BackgroundWorkerInitializeConnectionByOid(MyLogicalRepWorker->dbid, + MyLogicalRepWorker->userid, + 0); + + /* + * Set always-secure search path, so malicious users can't redirect user + * code (e.g. pg_index.indexprs). + */ + SetConfigOption("search_path", "", PGC_SUSET, PGC_S_OVERRIDE); + + /* + * Set the client encoding to the database encoding, since that is what + * the leader will expect. + */ + SetClientEncoding(GetDatabaseEncoding()); + + stream_xid = pst->stream_xid; + + StartTransactionCommand(); + oldcontext = MemoryContextSwitchTo(ApplyContext); + + MySubscription = GetSubscription(MyLogicalRepWorker->subid, true); + if (!MySubscription) + { + ereport(LOG, + (errmsg("logical replication apply worker for subscription %u will not " + "start because the subscription was removed during startup", + MyLogicalRepWorker->subid))); + proc_exit(0); + } + + MySubscriptionValid = true; + MemoryContextSwitchTo(oldcontext); + + /* Setup synchronous commit according to the user's wishes */ + SetConfigOption("synchronous_commit", MySubscription->synccommit, + PGC_BACKEND, PGC_S_OVERRIDE); + + /* Keep us informed about subscription changes. */ + CacheRegisterSyscacheCallback(SUBSCRIPTIONOID, + subscription_change_cb, + (Datum) 0); + + CommitTransactionCommand(); + + /* Setup replication origin tracking. */ + StartTransactionCommand(); + snprintf(originname, sizeof(originname), "pg_%u", MySubscription->oid); + originid = replorigin_by_name(originname, true); + if (!OidIsValid(originid)) + originid = replorigin_create(originname); + + /* + * The apply background worker doesn't need to monopolize this replication + * origin which was already acquired by its leader process. + */ + replorigin_session_setup(originid, false); + replorigin_session_origin = originid; + CommitTransactionCommand(); + + /* + * Allocate the origin name in long-lived context for error context + * message. + */ + apply_error_callback_arg.origin_name = MemoryContextStrdup(ApplyContext, + originname); + + elog(DEBUG1, "[Apply BGW #%u] started", pst->n); + + LogicalApplyBgwLoop(mqh, pst); + + /* + * We're done. Explicitly detach the shared memory segment so that we + * don't get a resource leak warning at commit time. This will fire any + * on_dsm_detach callbacks we've registered, as well. Once that's done, + * we can go ahead and exit. + */ + dsm_detach(seg); + proc_exit(0); +} + +/* + * Set up a dynamic shared memory segment. + * + * We set up a control region that contains a ApplyBgworkerShared, + * plus one region per message queue. There are as many message queues as + * the number of workers. + */ +static void +apply_bgworker_setup_dsm(ApplyBgworkerState *wstate) +{ + shm_toc_estimator e; + Size segsize; + dsm_segment *seg; + shm_toc *toc; + ApplyBgworkerShared *pst; + shm_mq *mq; + int64 queue_size = DSM_QUEUE_SIZE; + + /* + * Estimate how much shared memory we need. + * + * Because the TOC machinery may choose to insert padding of oddly-sized + * requests, we must estimate each chunk separately. + * + * We need one key to register the location of the header, and we need + * another key to track the location of the message queue. + */ + shm_toc_initialize_estimator(&e); + shm_toc_estimate_chunk(&e, sizeof(ApplyBgworkerShared)); + shm_toc_estimate_chunk(&e, (Size) queue_size); + + shm_toc_estimate_keys(&e, 1 + 1); + segsize = shm_toc_estimate(&e); + + /* Create the shared memory segment and establish a table of contents. */ + seg = dsm_create(shm_toc_estimate(&e), 0); + toc = shm_toc_create(PG_LOGICAL_APPLY_SHM_MAGIC, dsm_segment_address(seg), + segsize); + + /* Set up the header region. */ + pst = shm_toc_allocate(toc, sizeof(ApplyBgworkerShared)); + SpinLockInit(&pst->mutex); + pst->status = APPLY_BGWORKER_BUSY; + pst->server_version = walrcv_server_version(LogRepWorkerWalRcvConn); + pst->stream_xid = stream_xid; + pst->n = list_length(ApplyWorkersList) + 1; + + shm_toc_insert(toc, APPLY_BGWORKER_KEY_SHARED, pst); + + /* Set up message queue for the worker. */ + mq = shm_mq_create(shm_toc_allocate(toc, (Size) queue_size), + (Size) queue_size); + shm_toc_insert(toc, APPLY_BGWORKER_KEY_MQ, mq); + shm_mq_set_sender(mq, MyProc); + + /* Attach the queue. */ + wstate->mq_handle = shm_mq_attach(mq, seg, NULL); + + /* Return results to caller. */ + wstate->dsm_seg = seg; + wstate->pstate = pst; +} + +/* + * Start apply background worker process and allocate shared memory for it. + */ +static ApplyBgworkerState * +apply_bgworker_setup(void) +{ + MemoryContext oldcontext; + bool launched; + ApplyBgworkerState *wstate; + int napplyworkers; + + elog(DEBUG1, "setting up apply worker #%u", list_length(ApplyWorkersList) + 1); + + /* Check if there are free worker slot(s) */ + LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); + napplyworkers = logicalrep_apply_bgworker_count(MyLogicalRepWorker->subid); + LWLockRelease(LogicalRepWorkerLock); + if (napplyworkers >= max_apply_bgworkers_per_subscription) + return NULL; + + oldcontext = MemoryContextSwitchTo(ApplyContext); + + wstate = (ApplyBgworkerState *) palloc0(sizeof(ApplyBgworkerState)); + + /* Setup shared memory */ + apply_bgworker_setup_dsm(wstate); + + launched = logicalrep_worker_launch(MyLogicalRepWorker->dbid, + MySubscription->oid, + MySubscription->name, + MyLogicalRepWorker->userid, + InvalidOid, + dsm_segment_handle(wstate->dsm_seg)); + + if (launched) + ApplyWorkersList = lappend(ApplyWorkersList, wstate); + else + { + dsm_detach(wstate->dsm_seg); + wstate->dsm_seg = NULL; + + pfree(wstate); + wstate = NULL; + } + + MemoryContextSwitchTo(oldcontext); + + return wstate; +} + +/* + * Send the data to the specified apply background worker via shared-memory queue. + */ +void +apply_bgworker_send_data(ApplyBgworkerState *wstate, Size nbytes, const void *data) +{ + shm_mq_result result; + + result = shm_mq_send(wstate->mq_handle, nbytes, data, false, true); + + if (result != SHM_MQ_SUCCESS) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("could not send tuples to shared-memory queue"))); +} + +/* + * Wait until the status of apply background worker reaches the + * 'wait_for_status' + */ +void +apply_bgworker_wait_for(ApplyBgworkerState *wstate, + ApplyBgworkerStatus wait_for_status) +{ + for (;;) + { + char status; + + SpinLockAcquire(&wstate->pstate->mutex); + status = wstate->pstate->status; + SpinLockRelease(&wstate->pstate->mutex); + + /* Done if already in correct status. */ + if (status == wait_for_status) + break; + + /* If any workers (or the postmaster) have died, we have failed. */ + if (status == APPLY_BGWORKER_EXIT) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("background worker %u failed to apply transaction %u", + wstate->pstate->n, wstate->pstate->stream_xid))); + + /* Wait to be signalled. */ + WaitLatch(MyLatch, WL_LATCH_SET | WL_EXIT_ON_PM_DEATH, 0, + WAIT_EVENT_LOGICAL_APPLY_WORKER_STATE_CHANGE); + + /* Reset the latch so we don't spin. */ + ResetLatch(MyLatch); + + /* An interrupt may have occurred while we were waiting. */ + CHECK_FOR_INTERRUPTS(); + } +} + +/* + * Check the status of workers and report an error if any apply background + * worker has exited unexpectedly. + */ +void +apply_bgworker_check_status(void) +{ + ListCell *lc; + + if (am_apply_bgworker() || MySubscription->stream != SUBSTREAM_PARALLEL) + return; + + foreach(lc, ApplyWorkersList) + { + ApplyBgworkerState *wstate = (ApplyBgworkerState *) lfirst(lc); + + /* + * We don't lock here as in the worst case we will just detect the + * failure of worker a bit later. + */ + if (wstate->pstate->status == APPLY_BGWORKER_EXIT) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("background worker %u exited unexpectedly", + wstate->pstate->n))); + } + + /* + * Exit if any relation is not in the READY state and if any worker is + * handling the streaming transaction at the same time. Because for + * streaming transactions that is being applied in apply background + * worker, we cannot decide whether to apply the change for a relation + * that is not in the READY state (see should_apply_changes_for_rel) as we + * won't know remote_final_lsn by that time. + */ + if (list_length(ApplyWorkersFreeList) != list_length(ApplyWorkersList) && + !AllTablesyncsReady()) + { + ereport(LOG, + (errmsg("logical replication apply workers for subscription \"%s\" will restart", + MySubscription->name), + errdetail("Cannot handle streamed replication transaction by apply " + "background workers until all tables are synchronized"))); + + proc_exit(0); + } +} + +/* Set the apply background worker status */ +void +apply_bgworker_set_status(ApplyBgworkerStatus status) +{ + if (!am_apply_bgworker()) + return; + + elog(DEBUG1, "[Apply BGW #%u] set status to %d", MyParallelState->n, status); + + SpinLockAcquire(&MyParallelState->mutex); + MyParallelState->status = status; + SpinLockRelease(&MyParallelState->mutex); +} + +/* + * Define a savepoint for a subxact in apply background worker if needed. + * + * Inside apply background worker we can figure out that new subtransaction was + * started if new change arrived with different xid. In that case we can define + * named savepoint, so that we were able to commit/rollback it separately + * later. + * Special case is if the first change comes from subtransaction, then + * we check that current_xid differs from stream_xid. + */ +void +apply_bgworker_subxact_info_add(TransactionId current_xid) +{ + if (current_xid != stream_xid && + !list_member_int(subxactlist, (int) current_xid)) + { + MemoryContext oldctx; + char spname[MAXPGPATH]; + + snprintf(spname, MAXPGPATH, "savepoint_for_xid_%u", current_xid); + + elog(DEBUG1, "[Apply BGW #%u] defining savepoint %s", + MyParallelState->n, spname); + + DefineSavepoint(spname); + CommitTransactionCommand(); + + oldctx = MemoryContextSwitchTo(ApplyContext); + subxactlist = lappend_int(subxactlist, (int) current_xid); + MemoryContextSwitchTo(oldctx); + } +} diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c index c5c6a2ba68..d4d5093a0b 100644 --- a/src/backend/replication/logical/decode.c +++ b/src/backend/replication/logical/decode.c @@ -651,9 +651,10 @@ DecodeCommit(LogicalDecodingContext *ctx, XLogRecordBuffer *buf, { for (i = 0; i < parsed->nsubxacts; i++) { - ReorderBufferForget(ctx->reorder, parsed->subxacts[i], buf->origptr); + ReorderBufferForget(ctx->reorder, parsed->subxacts[i], buf->origptr, + commit_time); } - ReorderBufferForget(ctx->reorder, xid, buf->origptr); + ReorderBufferForget(ctx->reorder, xid, buf->origptr, commit_time); return; } @@ -821,10 +822,11 @@ DecodeAbort(LogicalDecodingContext *ctx, XLogRecordBuffer *buf, for (i = 0; i < parsed->nsubxacts; i++) { ReorderBufferAbort(ctx->reorder, parsed->subxacts[i], - buf->record->EndRecPtr); + buf->record->EndRecPtr, abort_time); } - ReorderBufferAbort(ctx->reorder, xid, buf->record->EndRecPtr); + ReorderBufferAbort(ctx->reorder, xid, buf->record->EndRecPtr, + abort_time); } /* update the decoding stats */ diff --git a/src/backend/replication/logical/launcher.c b/src/backend/replication/logical/launcher.c index 2bdab53e19..d11d2361c6 100644 --- a/src/backend/replication/logical/launcher.c +++ b/src/backend/replication/logical/launcher.c @@ -54,6 +54,7 @@ int max_logical_replication_workers = 4; int max_sync_workers_per_subscription = 2; +int max_apply_bgworkers_per_subscription = 2; LogicalRepWorker *MyLogicalRepWorker = NULL; @@ -73,6 +74,7 @@ static void logicalrep_launcher_onexit(int code, Datum arg); static void logicalrep_worker_onexit(int code, Datum arg); static void logicalrep_worker_detach(void); static void logicalrep_worker_cleanup(LogicalRepWorker *worker); +static void logicalrep_worker_stop_internal(LogicalRepWorker *worker); static bool on_commit_launcher_wakeup = false; @@ -151,8 +153,10 @@ get_subscription_list(void) * * This is only needed for cleaning up the shared memory in case the worker * fails to attach. + * + * Return false if the attach fails. Otherwise return true. */ -static void +static bool WaitForReplicationWorkerAttach(LogicalRepWorker *worker, uint16 generation, BackgroundWorkerHandle *handle) @@ -168,11 +172,11 @@ WaitForReplicationWorkerAttach(LogicalRepWorker *worker, LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); - /* Worker either died or has started; no need to do anything. */ + /* Worker either died or has started. Return false if died. */ if (!worker->in_use || worker->proc) { LWLockRelease(LogicalRepWorkerLock); - return; + return worker->in_use; } LWLockRelease(LogicalRepWorkerLock); @@ -187,7 +191,7 @@ WaitForReplicationWorkerAttach(LogicalRepWorker *worker, if (generation == worker->generation) logicalrep_worker_cleanup(worker); LWLockRelease(LogicalRepWorkerLock); - return; + return false; } /* @@ -223,6 +227,13 @@ logicalrep_worker_find(Oid subid, Oid relid, bool only_running) { LogicalRepWorker *w = &LogicalRepCtx->workers[i]; + /* + * We are only interested in the main apply worker or table sync worker + * here. + */ + if (w->subworker) + continue; + if (w->in_use && w->subid == subid && w->relid == relid && (!only_running || w->proc)) { @@ -259,11 +270,11 @@ logicalrep_workers_find(Oid subid, bool only_running) } /* - * Start new apply background worker, if possible. + * Start new background worker, if possible. */ -void +bool logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, Oid userid, - Oid relid) + Oid relid, dsm_handle subworker_dsm) { BackgroundWorker bgw; BackgroundWorkerHandle *bgw_handle; @@ -273,6 +284,10 @@ logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, Oid userid, LogicalRepWorker *worker = NULL; int nsyncworkers; TimestampTz now; + bool is_subworker = (subworker_dsm != DSM_HANDLE_INVALID); + + /* Sanity check : we don't support table sync in subworker. */ + Assert(!(is_subworker && OidIsValid(relid))); ereport(DEBUG1, (errmsg_internal("starting logical replication worker for subscription \"%s\"", @@ -350,7 +365,7 @@ retry: if (OidIsValid(relid) && nsyncworkers >= max_sync_workers_per_subscription) { LWLockRelease(LogicalRepWorkerLock); - return; + return false; } /* @@ -364,7 +379,7 @@ retry: (errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED), errmsg("out of logical replication worker slots"), errhint("You might need to increase max_logical_replication_workers."))); - return; + return false; } /* Prepare the worker slot. */ @@ -379,6 +394,7 @@ retry: worker->relstate = SUBREL_STATE_UNKNOWN; worker->relstate_lsn = InvalidXLogRecPtr; worker->stream_fileset = NULL; + worker->subworker = is_subworker; worker->last_lsn = InvalidXLogRecPtr; TIMESTAMP_NOBEGIN(worker->last_send_time); TIMESTAMP_NOBEGIN(worker->last_recv_time); @@ -396,19 +412,31 @@ retry: BGWORKER_BACKEND_DATABASE_CONNECTION; bgw.bgw_start_time = BgWorkerStart_RecoveryFinished; snprintf(bgw.bgw_library_name, BGW_MAXLEN, "postgres"); - snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyWorkerMain"); + + if (is_subworker) + snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyBgworkerMain"); + else + snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyWorkerMain"); + if (OidIsValid(relid)) snprintf(bgw.bgw_name, BGW_MAXLEN, "logical replication worker for subscription %u sync %u", subid, relid); + else if (is_subworker) + snprintf(bgw.bgw_name, BGW_MAXLEN, + "logical replication apply background worker for subscription %u", subid); else snprintf(bgw.bgw_name, BGW_MAXLEN, "logical replication worker for subscription %u", subid); + snprintf(bgw.bgw_type, BGW_MAXLEN, "logical replication worker"); bgw.bgw_restart_time = BGW_NEVER_RESTART; bgw.bgw_notify_pid = MyProcPid; bgw.bgw_main_arg = Int32GetDatum(slot); + if (is_subworker) + memcpy(bgw.bgw_extra, &subworker_dsm, sizeof(dsm_handle)); + if (!RegisterDynamicBackgroundWorker(&bgw, &bgw_handle)) { /* Failed to start worker, so clean up the worker slot. */ @@ -421,11 +449,11 @@ retry: (errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED), errmsg("out of background worker slots"), errhint("You might need to increase max_worker_processes."))); - return; + return false; } /* Now wait until it attaches. */ - WaitForReplicationWorkerAttach(worker, generation, bgw_handle); + return WaitForReplicationWorkerAttach(worker, generation, bgw_handle); } /* @@ -436,18 +464,27 @@ void logicalrep_worker_stop(Oid subid, Oid relid) { LogicalRepWorker *worker; - uint16 generation; LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); worker = logicalrep_worker_find(subid, relid, false); - /* No worker, nothing to do. */ - if (!worker) - { - LWLockRelease(LogicalRepWorkerLock); - return; - } + if (worker) + logicalrep_worker_stop_internal(worker); + + LWLockRelease(LogicalRepWorkerLock); +} + +/* + * Workhorse for logicalrep_worker_stop() and logicalrep_worker_detach(). Stop + * the worker and wait for it to die. + */ +static void +logicalrep_worker_stop_internal(LogicalRepWorker *worker) +{ + uint16 generation; + + Assert(LWLockHeldByMe(LogicalRepWorkerLock)); /* * Remember which generation was our worker so we can check if what we see @@ -485,10 +522,7 @@ logicalrep_worker_stop(Oid subid, Oid relid) * different, meaning that a different worker has taken the slot. */ if (!worker->in_use || worker->generation != generation) - { - LWLockRelease(LogicalRepWorkerLock); return; - } /* Worker has assigned proc, so it has started. */ if (worker->proc) @@ -522,8 +556,6 @@ logicalrep_worker_stop(Oid subid, Oid relid) LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); } - - LWLockRelease(LogicalRepWorkerLock); } /* @@ -599,6 +631,29 @@ logicalrep_worker_attach(int slot) static void logicalrep_worker_detach(void) { + /* + * If we are the main apply worker, stop all the apply background workers + * we started before. + */ + if (!MyLogicalRepWorker->subworker) + { + List *workers; + ListCell *lc; + + LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); + + workers = logicalrep_workers_find(MyLogicalRepWorker->subid, true); + foreach(lc, workers) + { + LogicalRepWorker *w = (LogicalRepWorker *) lfirst(lc); + + if (w->subworker) + logicalrep_worker_stop_internal(w); + } + + LWLockRelease(LogicalRepWorkerLock); + } + /* Block concurrent access. */ LWLockAcquire(LogicalRepWorkerLock, LW_EXCLUSIVE); @@ -621,6 +676,7 @@ logicalrep_worker_cleanup(LogicalRepWorker *worker) worker->userid = InvalidOid; worker->subid = InvalidOid; worker->relid = InvalidOid; + worker->subworker = false; } /* @@ -679,6 +735,30 @@ logicalrep_sync_worker_count(Oid subid) return res; } +/* + * Count the number of registered (not necessarily running) apply background + * workers for a subscription. + */ +int +logicalrep_apply_bgworker_count(Oid subid) +{ + int i; + int res = 0; + + Assert(LWLockHeldByMe(LogicalRepWorkerLock)); + + /* Search for attached worker for a given subscription id. */ + for (i = 0; i < max_logical_replication_workers; i++) + { + LogicalRepWorker *w = &LogicalRepCtx->workers[i]; + + if (w->subid == subid && w->subworker) + res++; + } + + return res; +} + /* * ApplyLauncherShmemSize * Compute space needed for replication launcher shared memory @@ -868,7 +948,7 @@ ApplyLauncherMain(Datum main_arg) wait_time = wal_retrieve_retry_interval; logicalrep_worker_launch(sub->dbid, sub->oid, sub->name, - sub->owner, InvalidOid); + sub->owner, InvalidOid, DSM_HANDLE_INVALID); } } diff --git a/src/backend/replication/logical/origin.c b/src/backend/replication/logical/origin.c index 21937ab2d3..50c567fb6e 100644 --- a/src/backend/replication/logical/origin.c +++ b/src/backend/replication/logical/origin.c @@ -1063,12 +1063,21 @@ ReplicationOriginExitCleanup(int code, Datum arg) * array doesn't have to be searched when calling * replorigin_session_advance(). * - * Obviously only one such cached origin can exist per process and the current + * Normally only one such cached origin can exist per process and the current * cached value can only be set again after the previous value is torn down * with replorigin_session_reset(). + * + * However, if the function parameter 'must_acquire' is false, we allow the + * process to use the same slot already acquired by another process. It's safe + * because 1) The only caller (apply background workers) will maintain the + * commit order by allowing only one process to commit at a time, so no two + * workers will be operating on the same origin at the same time (see comments + * in logical/worker.c). 2) Even though we try to advance the session's origin + * concurrently, it's safe to do so as we change/advance the session_origin + * LSNs under replicate_state LWLock. */ void -replorigin_session_setup(RepOriginId node) +replorigin_session_setup(RepOriginId node, bool must_acquire) { static bool registered_cleanup; int i; @@ -1110,7 +1119,7 @@ replorigin_session_setup(RepOriginId node) if (curstate->roident != node) continue; - else if (curstate->acquired_by != 0) + else if (curstate->acquired_by != 0 && must_acquire) { ereport(ERROR, (errcode(ERRCODE_OBJECT_IN_USE), @@ -1141,7 +1150,14 @@ replorigin_session_setup(RepOriginId node) Assert(session_replication_state->roident != InvalidRepOriginId); - session_replication_state->acquired_by = MyProcPid; + if (must_acquire) + session_replication_state->acquired_by = MyProcPid; + else if (session_replication_state->acquired_by == 0) + ereport(ERROR, + (errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED), + errmsg("apply background worker could not find replication state slot for replication origin with OID %u", + node), + errdetail("There is no replication state slot set by its main apply worker."))); LWLockRelease(ReplicationOriginLock); @@ -1321,7 +1337,7 @@ pg_replication_origin_session_setup(PG_FUNCTION_ARGS) name = text_to_cstring((text *) DatumGetPointer(PG_GETARG_DATUM(0))); origin = replorigin_by_name(name, false); - replorigin_session_setup(origin); + replorigin_session_setup(origin, true); replorigin_session_origin = origin; diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c index ff8513e2d2..affd08cfa4 100644 --- a/src/backend/replication/logical/proto.c +++ b/src/backend/replication/logical/proto.c @@ -1163,31 +1163,56 @@ logicalrep_read_stream_commit(StringInfo in, LogicalRepCommitData *commit_data) /* * Write STREAM ABORT to the output stream. Note that xid and subxid will be * same for the top-level transaction abort. + * + * If write_abort_lsn is true, send the abort_lsn and abort_time fields. + * Otherwise not. */ void logicalrep_write_stream_abort(StringInfo out, TransactionId xid, - TransactionId subxid) + ReorderBufferTXN *txn, XLogRecPtr abort_lsn, + bool write_abort_lsn) { pq_sendbyte(out, LOGICAL_REP_MSG_STREAM_ABORT); - Assert(TransactionIdIsValid(xid) && TransactionIdIsValid(subxid)); + Assert(TransactionIdIsValid(xid) && TransactionIdIsValid(txn->xid)); /* transaction ID */ pq_sendint32(out, xid); - pq_sendint32(out, subxid); + pq_sendint32(out, txn->xid); + + if (write_abort_lsn) + { + pq_sendint64(out, abort_lsn); + pq_sendint64(out, txn->xact_time.abort_time); + } } /* * Read STREAM ABORT from the output stream. + * + * If read_abort_lsn is true, try to read the abort_lsn and abort_time fields. + * Otherwise not. */ void -logicalrep_read_stream_abort(StringInfo in, TransactionId *xid, - TransactionId *subxid) +logicalrep_read_stream_abort(StringInfo in, + LogicalRepStreamAbortData *abort_data, + bool read_abort_lsn) { - Assert(xid && subxid); + Assert(abort_data); - *xid = pq_getmsgint(in, 4); - *subxid = pq_getmsgint(in, 4); + abort_data->xid = pq_getmsgint(in, 4); + abort_data->subxid = pq_getmsgint(in, 4); + + if (read_abort_lsn) + { + abort_data->abort_lsn = pq_getmsgint64(in); + abort_data->abort_time = pq_getmsgint64(in); + } + else + { + abort_data->abort_lsn = InvalidXLogRecPtr; + abort_data->abort_time = 0; + } } /* diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c index 88a37fde72..8989328046 100644 --- a/src/backend/replication/logical/reorderbuffer.c +++ b/src/backend/replication/logical/reorderbuffer.c @@ -2826,7 +2826,8 @@ ReorderBufferFinishPrepared(ReorderBuffer *rb, TransactionId xid, * disk. */ void -ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn) +ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn, + TimestampTz abort_time) { ReorderBufferTXN *txn; @@ -2837,6 +2838,8 @@ ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn) if (txn == NULL) return; + txn->xact_time.abort_time = abort_time; + /* For streamed transactions notify the remote node about the abort. */ if (rbtxn_is_streamed(txn)) { @@ -2911,7 +2914,8 @@ ReorderBufferAbortOld(ReorderBuffer *rb, TransactionId oldestRunningXid) * to this xid might re-create the transaction incompletely. */ void -ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn) +ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn, + TimestampTz abort_time) { ReorderBufferTXN *txn; @@ -2922,6 +2926,8 @@ ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn) if (txn == NULL) return; + txn->xact_time.abort_time = abort_time; + /* For streamed transactions notify the remote node about the abort. */ if (rbtxn_is_streamed(txn)) rb->stream_abort(rb, txn, lsn); diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c index 670c6fcada..8ffba7e2e5 100644 --- a/src/backend/replication/logical/tablesync.c +++ b/src/backend/replication/logical/tablesync.c @@ -568,7 +568,8 @@ process_syncing_tables_for_apply(XLogRecPtr current_lsn) MySubscription->oid, MySubscription->name, MyLogicalRepWorker->userid, - rstate->relid); + rstate->relid, + DSM_HANDLE_INVALID); hentry->last_start_time = now; } } @@ -589,6 +590,9 @@ process_syncing_tables_for_apply(XLogRecPtr current_lsn) void process_syncing_tables(XLogRecPtr current_lsn) { + if (MyLogicalRepWorker->subworker) + return; + if (am_tablesync_worker()) process_syncing_tables_for_sync(current_lsn); else @@ -1273,7 +1277,7 @@ LogicalRepSyncTableStart(XLogRecPtr *origin_startpos) * time this tablesync was launched. */ originid = replorigin_by_name(originname, false); - replorigin_session_setup(originid); + replorigin_session_setup(originid, true); replorigin_session_origin = originid; *origin_startpos = replorigin_session_get_progress(false); @@ -1384,7 +1388,7 @@ LogicalRepSyncTableStart(XLogRecPtr *origin_startpos) true /* go backward */ , true /* WAL log */ ); UnlockRelationOid(ReplicationOriginRelationId, RowExclusiveLock); - replorigin_session_setup(originid); + replorigin_session_setup(originid, true); replorigin_session_origin = originid; } else diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c index 38e3b1c1b3..bbd0304a8d 100644 --- a/src/backend/replication/logical/worker.c +++ b/src/backend/replication/logical/worker.c @@ -22,8 +22,28 @@ * STREAMED TRANSACTIONS * --------------------- * Streamed transactions (large transactions exceeding a memory limit on the - * upstream) are not applied immediately, but instead, the data is written - * to temporary files and then applied at once when the final commit arrives. + * upstream) are applied using one of two approaches. + * + * 1) Separate background workers + * + * If streaming = parallel, We assign a new apply background worker (if + * available) as soon as the xact's first stream is received. The main apply + * worker will send changes to this new worker via shared memory. We keep this + * worker assigned till the transaction commit is received and also wait for + * the worker to finish at commit. This preserves commit ordering and avoids + * file I/O in most cases. We still need to spill to a file if there is no + * worker available. It is important to maintain commit order to avoid failures + * due to (a) transaction dependencies, say if we insert a row in the first + * transaction and update it in the second transaction then allowing to apply + * both in parallel can lead to failure in the update. (b) deadlocks, allowing + * transactions that update the same set of rows/tables in opposite order to be + * applied in parallel can lead to deadlocks. + * + * 2) Write to temporary files and apply when the final commit arrives + * + * If no worker is available to handle streamed transaction, the data is + * written to temporary files and then applied at once when the final commit + * arrives. * * Unlike the regular (non-streamed) case, handling streamed transactions has * to handle aborts of both the toplevel transaction and subtransactions. This @@ -219,20 +239,8 @@ typedef struct ApplyExecutionData PartitionTupleRouting *proute; /* partition routing info */ } ApplyExecutionData; -/* Struct for saving and restoring apply errcontext information */ -typedef struct ApplyErrorCallbackArg -{ - LogicalRepMsgType command; /* 0 if invalid */ - LogicalRepRelMapEntry *rel; - - /* Remote node information */ - int remote_attnum; /* -1 if invalid */ - TransactionId remote_xid; - XLogRecPtr finish_lsn; - char *origin_name; -} ApplyErrorCallbackArg; - -static ApplyErrorCallbackArg apply_error_callback_arg = +/* errcontext tracker */ +ApplyErrorCallbackArg apply_error_callback_arg = { .command = 0, .rel = NULL, @@ -242,7 +250,7 @@ static ApplyErrorCallbackArg apply_error_callback_arg = .origin_name = NULL, }; -static MemoryContext ApplyMessageContext = NULL; +MemoryContext ApplyMessageContext = NULL; MemoryContext ApplyContext = NULL; /* per stream context for streaming transactions */ @@ -251,27 +259,38 @@ static MemoryContext LogicalStreamingContext = NULL; WalReceiverConn *LogRepWorkerWalRcvConn = NULL; Subscription *MySubscription = NULL; -static bool MySubscriptionValid = false; +bool MySubscriptionValid = false; bool in_remote_transaction = false; static XLogRecPtr remote_final_lsn = InvalidXLogRecPtr; /* fields valid only when processing streamed transaction */ -static bool in_streamed_transaction = false; +bool in_streamed_transaction = false; + +TransactionId stream_xid = InvalidTransactionId; +static ApplyBgworkerState *stream_apply_worker = NULL; -static TransactionId stream_xid = InvalidTransactionId; +/* Check if we are applying the transaction in an apply background worker */ +#define apply_bgworker_active() (in_streamed_transaction && stream_apply_worker != NULL) + +/* + * The number of changes during one streaming block (only for apply background + * workers) + */ +static uint32 nchanges = 0; /* * We enable skipping all data modification changes (INSERT, UPDATE, etc.) for * the subscription if the remote transaction's finish LSN matches the subskiplsn. * Once we start skipping changes, we don't stop it until we skip all changes of * the transaction even if pg_subscription is updated and MySubscription->skiplsn - * gets changed or reset during that. Also, in streaming transaction cases, we - * don't skip receiving and spooling the changes since we decide whether or not + * gets changed or reset during that. Also, in streaming transaction cases (streaming = on), + * we don't skip receiving and spooling the changes since we decide whether or not * to skip applying the changes when starting to apply changes. The subskiplsn is * cleared after successfully skipping the transaction or applying non-empty * transaction. The latter prevents the mistakenly specified subskiplsn from - * being left. + * being left. Note that we cannot skip the streaming transaction in parallel + * mode, because we cannot get the finish LSN before applying the changes. */ static XLogRecPtr skip_xact_finish_lsn = InvalidXLogRecPtr; #define is_skipping_changes() (unlikely(!XLogRecPtrIsInvalid(skip_xact_finish_lsn))) @@ -324,9 +343,6 @@ static void maybe_reread_subscription(void); static void DisableSubscriptionAndExit(void); -/* prototype needed because of stream_commit */ -static void apply_dispatch(StringInfo s); - static void apply_handle_commit_internal(LogicalRepCommitData *commit_data); static void apply_handle_insert_internal(ApplyExecutionData *edata, ResultRelInfo *relinfo, @@ -359,7 +375,6 @@ static void stop_skipping_changes(void); static void clear_subscription_skip_lsn(XLogRecPtr finish_lsn); /* Functions for apply error callback */ -static void apply_error_callback(void *arg); static inline void set_apply_error_context_xact(TransactionId xid, XLogRecPtr lsn); static inline void reset_apply_error_context_info(void); @@ -426,40 +441,85 @@ end_replication_step(void) } /* - * Handle streamed transactions. + * Handle streamed transactions for both the main apply worker and the apply + * background workers. + * + * In streaming case (receiving a block of streamed transaction), for + * SUBSTREAM_ON mode, simply redirect it to a file for the proper toplevel + * transaction, and for SUBSTREAM_PARALLEL mode, send the changes to apply + * background workers (LOGICAL_REP_MSG_RELATION or LOGICAL_REP_MSG_TYPE changes + * will also be applied in main apply worker). * - * If in streaming mode (receiving a block of streamed transaction), we - * simply redirect it to a file for the proper toplevel transaction. + * For non-streamed transactions, returns false; + * For streamed transactions, returns true if in main apply worker, false + * otherwise. * - * Returns true for streamed transactions, false otherwise (regular mode). + * Exception: When parallel mode is applying streamed transaction in the main + * apply worker, (e.g. when addressing LOGICAL_REP_MSG_RELATION or + * LOGICAL_REP_MSG_TYPE changes), then return false. */ static bool handle_streamed_transaction(LogicalRepMsgType action, StringInfo s) { - TransactionId xid; + TransactionId current_xid = InvalidTransactionId; - /* not in streaming mode */ - if (!in_streamed_transaction) + /* Not in streaming mode */ + if (!(in_streamed_transaction || am_apply_bgworker())) return false; - Assert(stream_fd != NULL); Assert(TransactionIdIsValid(stream_xid)); /* * We should have received XID of the subxact as the first part of the * message, so extract it. */ - xid = pq_getmsgint(s, 4); + current_xid = pq_getmsgint(s, 4); - if (!TransactionIdIsValid(xid)) + if (!TransactionIdIsValid(current_xid)) ereport(ERROR, (errcode(ERRCODE_PROTOCOL_VIOLATION), errmsg_internal("invalid transaction ID in streamed replication transaction"))); - /* Add the new subxact to the array (unless already there). */ - subxact_info_add(xid); + if (am_apply_bgworker()) + { + /* Define a savepoint for a subxact if needed. */ + apply_bgworker_subxact_info_add(current_xid); + + return false; + } + + if (apply_bgworker_active()) + { + /* + * This is the main apply worker, but there is an apply background + * worker, so apply the changes of this transaction in that background + * worker. Pass the data to the worker. + */ + apply_bgworker_send_data(stream_apply_worker, s->len, s->data); + nchanges += 1; + + /* + * XXX The publisher side doesn't always send relation/type update + * messages after the streaming transaction, so also update the + * relation/type in main apply worker here. See function + * cleanup_rel_sync_cache. + */ + if (action == LOGICAL_REP_MSG_RELATION || + action == LOGICAL_REP_MSG_TYPE) + return false; + + return true; + } + + /* + * This is the main apply worker, but there is no apply background worker, + * so write to temporary files and apply when the final commit arrives. + * + * Add the new subxact to the array (unless already there). + */ + subxact_info_add(current_xid); - /* write the change to the current file */ + /* Write the change to the current file */ stream_write_change(action, s); return true; @@ -844,6 +904,9 @@ apply_handle_commit(StringInfo s) apply_handle_commit_internal(&commit_data); + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); + /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(commit_data.end_lsn); @@ -898,7 +961,9 @@ apply_handle_prepare_internal(LogicalRepPreparedTxnData *prepare_data) * BeginTransactionBlock is necessary to balance the EndTransactionBlock * called within the PrepareTransactionBlock below. */ - BeginTransactionBlock(); + if (!IsTransactionBlock()) + BeginTransactionBlock(); + CommitTransactionCommand(); /* Completes the preceding Begin command. */ /* @@ -950,6 +1015,9 @@ apply_handle_prepare(StringInfo s) in_remote_transaction = false; + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); + /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(prepare_data.end_lsn); @@ -1064,10 +1132,6 @@ apply_handle_rollback_prepared(StringInfo s) /* * Handle STREAM PREPARE. - * - * Logic is in two parts: - * 1. Replay all the spooled operations - * 2. Mark the transaction as prepared */ static void apply_handle_stream_prepare(StringInfo s) @@ -1088,24 +1152,76 @@ apply_handle_stream_prepare(StringInfo s) logicalrep_read_stream_prepare(s, &prepare_data); set_apply_error_context_xact(prepare_data.xid, prepare_data.prepare_lsn); - elog(DEBUG1, "received prepare for streamed transaction %u", prepare_data.xid); + if (am_apply_bgworker()) + { + /* Mark the transaction as prepared. */ + apply_handle_prepare_internal(&prepare_data); - /* Replay all the spooled operations. */ - apply_spooled_messages(prepare_data.xid, prepare_data.prepare_lsn); + CommitTransactionCommand(); - /* Mark the transaction as prepared. */ - apply_handle_prepare_internal(&prepare_data); + pgstat_report_stat(false); - CommitTransactionCommand(); + list_free(subxactlist); + subxactlist = NIL; - pgstat_report_stat(false); + apply_bgworker_set_status(APPLY_BGWORKER_FINISHED); + } + else + { + /* This is the main apply worker. */ + ApplyBgworkerState *wstate = apply_bgworker_find(prepare_data.xid); - store_flush_position(prepare_data.end_lsn); + elog(DEBUG1, "received prepare for streamed transaction %u", + prepare_data.xid); + + /* + * Check if we are processing this transaction in an apply background + * worker and if so, send the changes to that worker. + */ + if (wstate) + { + /* + * After sending the data to the apply background worker, wait for + * that worker to finish. This is necessary to maintain commit + * order which avoids failures due to transaction dependencies and + * deadlocks. + */ + apply_bgworker_send_data(wstate, s->len, s->data); + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_FINISHED); + apply_bgworker_free(wstate); + + pgstat_report_stat(false); + store_flush_position(prepare_data.end_lsn); + } + else + { + /* + * The transaction has been serialized to file, so replay all the + * spooled operations. + */ + apply_spooled_messages(prepare_data.xid, prepare_data.prepare_lsn); + + /* Mark the transaction as prepared. */ + apply_handle_prepare_internal(&prepare_data); + + CommitTransactionCommand(); + + pgstat_report_stat(false); + + store_flush_position(prepare_data.end_lsn); + + in_remote_transaction = false; + + /* Unlink the files with serialized changes and subxact info. */ + stream_cleanup_files(MyLogicalRepWorker->subid, prepare_data.xid); + } + } in_remote_transaction = false; + stream_apply_worker = NULL; - /* unlink the files with serialized changes and subxact info. */ - stream_cleanup_files(MyLogicalRepWorker->subid, prepare_data.xid); + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(prepare_data.end_lsn); @@ -1155,15 +1271,6 @@ apply_handle_stream_start(StringInfo s) (errcode(ERRCODE_PROTOCOL_VIOLATION), errmsg_internal("duplicate STREAM START message"))); - /* - * Start a transaction on stream start, this transaction will be committed - * on the stream stop unless it is a tablesync worker in which case it - * will be committed after processing all the messages. We need the - * transaction for handling the buffile, used for serializing the - * streaming data and subxact info. - */ - begin_replication_step(); - /* notify handle methods we're processing a remote transaction */ in_streamed_transaction = true; @@ -1177,36 +1284,93 @@ apply_handle_stream_start(StringInfo s) set_apply_error_context_xact(stream_xid, InvalidXLogRecPtr); - /* - * Initialize the worker's stream_fileset if we haven't yet. This will be - * used for the entire duration of the worker so create it in a permanent - * context. We create this on the very first streaming message from any - * transaction and then use it for this and other streaming transactions. - * Now, we could create a fileset at the start of the worker as well but - * then we won't be sure that it will ever be used. - */ - if (MyLogicalRepWorker->stream_fileset == NULL) + if (am_apply_bgworker()) { - MemoryContext oldctx; - - oldctx = MemoryContextSwitchTo(ApplyContext); + /* + * Make sure the handle apply_dispatch methods are aware we're in a + * remote transaction. + */ + in_remote_transaction = true; - MyLogicalRepWorker->stream_fileset = palloc(sizeof(FileSet)); - FileSetInit(MyLogicalRepWorker->stream_fileset); + /* Begin the transaction. */ + AcceptInvalidationMessages(); + maybe_reread_subscription(); - MemoryContextSwitchTo(oldctx); + StartTransactionCommand(); + BeginTransactionBlock(); + CommitTransactionCommand(); } + else + { + /* + * This is the main apply worker. Check if there is any free apply + * background worker we can use to process this transaction. + */ + if (first_segment) + stream_apply_worker = apply_bgworker_start(stream_xid); + else + stream_apply_worker = apply_bgworker_find(stream_xid); - /* open the spool file for this transaction */ - stream_open_file(MyLogicalRepWorker->subid, stream_xid, first_segment); + if (stream_apply_worker) + { + /* + * If we have found a free worker or if we are already applying this + * transaction in an apply background worker, then we pass the data to + * that worker. + */ + if (first_segment) + apply_bgworker_send_data(stream_apply_worker, s->len, s->data); - /* if this is not the first segment, open existing subxact file */ - if (!first_segment) - subxact_info_read(MyLogicalRepWorker->subid, stream_xid); + nchanges = 0; + elog(DEBUG1, "starting streaming of xid %u", stream_xid); + } + else + { + /* + * Since no apply background worker is available for the first + * stream start, serialize all the changes of the transaction. + * + * Start a transaction on stream start, this transaction will be + * committed on the stream stop unless it is a tablesync worker in + * which case it will be committed after processing all the + * messages. We need the transaction for handling the buffile, + * used for serializing the streaming data and subxact info. + */ + begin_replication_step(); - pgstat_report_activity(STATE_RUNNING, NULL); + /* + * Initialize the worker's stream_fileset if we haven't yet. This will + * be used for the entire duration of the worker so create it in a + * permanent context. We create this on the very first streaming + * message from any transaction and then use it for this and other + * streaming transactions. Now, we could create a fileset at the start + * of the worker as well but then we won't be sure that it will ever + * be used. + */ + if (MyLogicalRepWorker->stream_fileset == NULL) + { + MemoryContext oldctx; - end_replication_step(); + oldctx = MemoryContextSwitchTo(ApplyContext); + + MyLogicalRepWorker->stream_fileset = palloc(sizeof(FileSet)); + FileSetInit(MyLogicalRepWorker->stream_fileset); + + MemoryContextSwitchTo(oldctx); + } + + /* Open the spool file for this transaction. */ + stream_open_file(MyLogicalRepWorker->subid, stream_xid, first_segment); + + /* If this is not the first segment, open existing subxact file. */ + if (!first_segment) + subxact_info_read(MyLogicalRepWorker->subid, stream_xid); + + end_replication_step(); + } + } + + pgstat_report_activity(STATE_RUNNING, NULL); } /* @@ -1220,53 +1384,52 @@ apply_handle_stream_stop(StringInfo s) (errcode(ERRCODE_PROTOCOL_VIOLATION), errmsg_internal("STREAM STOP message without STREAM START"))); - /* - * Close the file with serialized changes, and serialize information about - * subxacts for the toplevel transaction. - */ - subxact_info_write(MyLogicalRepWorker->subid, stream_xid); - stream_close_file(); + if (apply_bgworker_active()) + { + char action = LOGICAL_REP_MSG_STREAM_STOP; - /* We must be in a valid transaction state */ - Assert(IsTransactionState()); + apply_bgworker_send_data(stream_apply_worker, 1, &action); - /* Commit the per-stream transaction */ - CommitTransactionCommand(); + elog(DEBUG1, "stopped streaming of xid %u, %u changes streamed", stream_xid, nchanges); + } + else + { + /* + * Close the file with serialized changes, and serialize information + * about subxacts for the toplevel transaction. + */ + subxact_info_write(MyLogicalRepWorker->subid, stream_xid); + stream_close_file(); - in_streamed_transaction = false; + /* We must be in a valid transaction state */ + Assert(IsTransactionState()); - /* Reset per-stream context */ - MemoryContextReset(LogicalStreamingContext); + /* Commit the per-stream transaction */ + CommitTransactionCommand(); + + /* Reset per-stream context */ + MemoryContextReset(LogicalStreamingContext); + } + + in_streamed_transaction = false; + stream_apply_worker = NULL; pgstat_report_activity(STATE_IDLE, NULL); reset_apply_error_context_info(); } /* - * Handle STREAM abort message. + * Handle STREAM ABORT message when the transaction was spilled to disk. */ static void -apply_handle_stream_abort(StringInfo s) +serialize_stream_abort(TransactionId xid, TransactionId subxid) { - TransactionId xid; - TransactionId subxid; - - if (in_streamed_transaction) - ereport(ERROR, - (errcode(ERRCODE_PROTOCOL_VIOLATION), - errmsg_internal("STREAM ABORT message without STREAM STOP"))); - - logicalrep_read_stream_abort(s, &xid, &subxid); - /* * If the two XIDs are the same, it's in fact abort of toplevel xact, so * just delete the files with serialized info. */ if (xid == subxid) - { - set_apply_error_context_xact(xid, InvalidXLogRecPtr); stream_cleanup_files(MyLogicalRepWorker->subid, xid); - } else { /* @@ -1290,8 +1453,6 @@ apply_handle_stream_abort(StringInfo s) bool found = false; char path[MAXPGPATH]; - set_apply_error_context_xact(subxid, InvalidXLogRecPtr); - subidx = -1; begin_replication_step(); subxact_info_read(MyLogicalRepWorker->subid, xid); @@ -1316,7 +1477,6 @@ apply_handle_stream_abort(StringInfo s) cleanup_subxact_info(); end_replication_step(); CommitTransactionCommand(); - reset_apply_error_context_info(); return; } @@ -1339,6 +1499,142 @@ apply_handle_stream_abort(StringInfo s) end_replication_step(); CommitTransactionCommand(); } +} + +/* + * Handle STREAM ABORT message. + */ +static void +apply_handle_stream_abort(StringInfo s) +{ + TransactionId xid; + TransactionId subxid; + LogicalRepStreamAbortData abort_data; + bool read_abort_lsn = false; + + if (in_streamed_transaction) + ereport(ERROR, + (errcode(ERRCODE_PROTOCOL_VIOLATION), + errmsg_internal("STREAM ABORT message without STREAM STOP"))); + + /* Check whether the publisher sends abort_lsn and abort_time. */ + if (am_apply_bgworker()) + read_abort_lsn = MyParallelState->server_version >= 160000; + + logicalrep_read_stream_abort(s, &abort_data, read_abort_lsn); + + xid = abort_data.xid; + subxid = abort_data.subxid; + + set_apply_error_context_xact(subxid, abort_data.abort_lsn); + + if (am_apply_bgworker()) + { + elog(DEBUG1, "[Apply BGW #%u] aborting current transaction xid=%u, subxid=%u", + MyParallelState->n, GetCurrentTransactionIdIfAny(), + GetCurrentSubTransactionId()); + + /* + * Update origin state so we can restart streaming from correct + * position in case of crash. + */ + if (read_abort_lsn) + { + replorigin_session_origin_lsn = abort_data.abort_lsn; + replorigin_session_origin_timestamp = abort_data.abort_time; + } + + /* + * If the two XIDs are the same, it's in fact abort of toplevel xact, + * so just free the subxactlist. + */ + if (subxid == xid) + { + AbortCurrentTransaction(); + + EndTransactionBlock(false); + CommitTransactionCommand(); + + in_remote_transaction = false; + pgstat_report_activity(STATE_IDLE, NULL); + + list_free(subxactlist); + subxactlist = NIL; + + apply_bgworker_set_status(APPLY_BGWORKER_FINISHED); + } + else + { + /* + * OK, so it's a subxact. Rollback to the savepoint. + * + * We also need to read the subxactlist, determine the offset + * tracked for the subxact, and truncate the list. + */ + int i; + bool found = false; + char spname[MAXPGPATH]; + + snprintf(spname, MAXPGPATH, "savepoint_for_xid_%u", subxid); + + elog(DEBUG1, "[Apply BGW #%u] rolling back to savepoint %s", + MyParallelState->n, spname); + + for (i = list_length(subxactlist) - 1; i >= 0; i--) + { + xid = (TransactionId) list_nth_int(subxactlist, i); + if (xid == subxid) + { + found = true; + break; + } + } + + if (found) + { + RollbackToSavepoint(spname); + CommitTransactionCommand(); + subxactlist = list_truncate(subxactlist, i + 1); + } + + pgstat_report_activity(STATE_IDLEINTRANSACTION, NULL); + } + } + else + { + /* This is the main apply worker. */ + ApplyBgworkerState *wstate = apply_bgworker_find(xid); + + /* + * Check if we are processing this transaction in an apply background + * worker and if so, send the changes to that worker. + */ + if (wstate) + { + /* Send STREAM ABORT message to the apply background worker. */ + apply_bgworker_send_data(wstate, s->len, s->data); + + /* + * After sending the data to the apply background worker, wait for + * that worker to finish. This is necessary to maintain commit + * order which avoids failures due to transaction dependencies and + * deadlocks. + */ + if (subxid == xid) + { + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_FINISHED); + apply_bgworker_free(wstate); + } + } + else + { + /* + * We are in main apply worker and the transaction has been + * serialized to file. + */ + serialize_stream_abort(xid, subxid); + } + } reset_apply_error_context_info(); } @@ -1468,8 +1764,8 @@ apply_spooled_messages(TransactionId xid, XLogRecPtr lsn) static void apply_handle_stream_commit(StringInfo s) { - TransactionId xid; LogicalRepCommitData commit_data; + TransactionId xid; if (in_streamed_transaction) ereport(ERROR, @@ -1479,14 +1775,81 @@ apply_handle_stream_commit(StringInfo s) xid = logicalrep_read_stream_commit(s, &commit_data); set_apply_error_context_xact(xid, commit_data.commit_lsn); - elog(DEBUG1, "received commit for streamed transaction %u", xid); + if (am_apply_bgworker()) + { + /* + * Update origin state so we can restart streaming from correct + * position in case of crash. + */ + replorigin_session_origin_lsn = commit_data.end_lsn; + replorigin_session_origin_timestamp = commit_data.committime; + + CommitTransactionCommand(); + EndTransactionBlock(false); + CommitTransactionCommand(); - apply_spooled_messages(xid, commit_data.commit_lsn); + in_remote_transaction = false; - apply_handle_commit_internal(&commit_data); + pgstat_report_stat(false); + + list_free(subxactlist); + subxactlist = NIL; + + apply_bgworker_set_status(APPLY_BGWORKER_FINISHED); + } + else + { + /* This is the main apply worker. */ + ApplyBgworkerState *wstate = apply_bgworker_find(xid); + + elog(DEBUG1, "received commit for streamed transaction %u", xid); + + /* + * Check if we are processing this transaction in an apply background + * worker and if so, send the changes to that worker. + */ + if (wstate) + { + /* Send STREAM COMMIT message to the apply background worker. */ + apply_bgworker_send_data(wstate, s->len, s->data); + + /* + * After sending the data to the apply background worker, wait for + * that worker to finish. This is necessary to maintain commit + * order which avoids failures due to transaction dependencies and + * deadlocks. + */ + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_FINISHED); - /* unlink the files with serialized changes and subxact info */ - stream_cleanup_files(MyLogicalRepWorker->subid, xid); + pgstat_report_stat(false); + store_flush_position(commit_data.end_lsn); + stop_skipping_changes(); + + apply_bgworker_free(wstate); + + /* + * The transaction is either non-empty or skipped, so we clear the + * subskiplsn. + */ + clear_subscription_skip_lsn(commit_data.commit_lsn); + } + else + { + /* + * The transaction has been serialized to file, so replay all the + * spooled operations. + */ + apply_spooled_messages(xid, commit_data.commit_lsn); + + apply_handle_commit_internal(&commit_data); + + /* Unlink the files with serialized changes and subxact info. */ + stream_cleanup_files(MyLogicalRepWorker->subid, xid); + } + } + + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(commit_data.end_lsn); @@ -2467,7 +2830,7 @@ apply_handle_truncate(StringInfo s) /* * Logical replication protocol message dispatcher. */ -static void +void apply_dispatch(StringInfo s) { LogicalRepMsgType action = pq_getmsgbyte(s); @@ -2636,6 +2999,10 @@ store_flush_position(XLogRecPtr remote_lsn) { FlushPosition *flushpos; + /* We only need to collect the LSN in main apply worker */ + if (am_apply_bgworker()) + return; + /* Need to do this in permanent context */ MemoryContextSwitchTo(ApplyContext); @@ -2650,7 +3017,7 @@ store_flush_position(XLogRecPtr remote_lsn) /* Update statistics of the worker. */ -static void +void UpdateWorkerStats(XLogRecPtr last_lsn, TimestampTz send_time, bool reply) { MyLogicalRepWorker->last_lsn = last_lsn; @@ -2812,6 +3179,9 @@ LogicalRepApplyLoop(XLogRecPtr last_received) AcceptInvalidationMessages(); maybe_reread_subscription(); + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); + /* Process any table synchronization changes. */ process_syncing_tables(last_received); } @@ -3113,7 +3483,7 @@ maybe_reread_subscription(void) /* * Callback from subscription syscache invalidation. */ -static void +void subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue) { MySubscriptionValid = false; @@ -3709,7 +4079,7 @@ ApplyWorkerMain(Datum main_arg) originid = replorigin_by_name(originname, true); if (!OidIsValid(originid)) originid = replorigin_create(originname); - replorigin_session_setup(originid); + replorigin_session_setup(originid, true); replorigin_session_origin = originid; origin_startpos = replorigin_session_get_progress(false); CommitTransactionCommand(); @@ -3750,13 +4120,14 @@ ApplyWorkerMain(Datum main_arg) server_version = walrcv_server_version(LogRepWorkerWalRcvConn); options.proto.logical.proto_version = + server_version >= 160000 ? LOGICALREP_PROTO_STREAM_PARALLEL_VERSION_NUM : server_version >= 150000 ? LOGICALREP_PROTO_TWOPHASE_VERSION_NUM : server_version >= 140000 ? LOGICALREP_PROTO_STREAM_VERSION_NUM : LOGICALREP_PROTO_VERSION_NUM; options.proto.logical.publication_names = MySubscription->publications; options.proto.logical.binary = MySubscription->binary; - options.proto.logical.streaming = MySubscription->stream; + options.proto.logical.streaming = (MySubscription->stream != SUBSTREAM_OFF); options.proto.logical.twophase = false; if (!am_tablesync_worker()) @@ -3914,7 +4285,8 @@ clear_subscription_skip_lsn(XLogRecPtr finish_lsn) XLogRecPtr myskiplsn = MySubscription->skiplsn; bool started_tx = false; - if (likely(XLogRecPtrIsInvalid(myskiplsn))) + if (likely(XLogRecPtrIsInvalid(myskiplsn)) || + am_apply_bgworker()) return; if (!IsTransactionState()) @@ -3986,7 +4358,7 @@ clear_subscription_skip_lsn(XLogRecPtr finish_lsn) } /* Error callback to give more context info about the change being applied */ -static void +void apply_error_callback(void *arg) { ApplyErrorCallbackArg *errarg = &apply_error_callback_arg; @@ -4014,23 +4386,47 @@ apply_error_callback(void *arg) errarg->remote_xid, LSN_FORMAT_ARGS(errarg->finish_lsn)); } - else if (errarg->remote_attnum < 0) - errcontext("processing remote data for replication origin \"%s\" during \"%s\" for replication target relation \"%s.%s\" in transaction %u finished at %X/%X", - errarg->origin_name, - logicalrep_message_type(errarg->command), - errarg->rel->remoterel.nspname, - errarg->rel->remoterel.relname, - errarg->remote_xid, - LSN_FORMAT_ARGS(errarg->finish_lsn)); else - errcontext("processing remote data for replication origin \"%s\" during \"%s\" for replication target relation \"%s.%s\" column \"%s\" in transaction %u finished at %X/%X", - errarg->origin_name, - logicalrep_message_type(errarg->command), - errarg->rel->remoterel.nspname, - errarg->rel->remoterel.relname, - errarg->rel->remoterel.attnames[errarg->remote_attnum], - errarg->remote_xid, - LSN_FORMAT_ARGS(errarg->finish_lsn)); + { + if (errarg->remote_attnum < 0) + { + if (XLogRecPtrIsInvalid(errarg->finish_lsn)) + errcontext("processing remote data for replication origin \"%s\" during \"%s\" for replication target relation \"%s.%s\" in transaction %u", + errarg->origin_name, + logicalrep_message_type(errarg->command), + errarg->rel->remoterel.nspname, + errarg->rel->remoterel.relname, + errarg->remote_xid); + else + errcontext("processing remote data for replication origin \"%s\" during \"%s\" for replication target relation \"%s.%s\" in transaction %u finished at %X/%X", + errarg->origin_name, + logicalrep_message_type(errarg->command), + errarg->rel->remoterel.nspname, + errarg->rel->remoterel.relname, + errarg->remote_xid, + LSN_FORMAT_ARGS(errarg->finish_lsn)); + } + else + { + if (XLogRecPtrIsInvalid(errarg->finish_lsn)) + errcontext("processing remote data for replication origin \"%s\" during \"%s\" for replication target relation \"%s.%s\" column \"%s\" in transaction %u", + errarg->origin_name, + logicalrep_message_type(errarg->command), + errarg->rel->remoterel.nspname, + errarg->rel->remoterel.relname, + errarg->rel->remoterel.attnames[errarg->remote_attnum], + errarg->remote_xid); + else + errcontext("processing remote data for replication origin \"%s\" during \"%s\" for replication target relation \"%s.%s\" column \"%s\" in transaction %u finished at %X/%X", + errarg->origin_name, + logicalrep_message_type(errarg->command), + errarg->rel->remoterel.nspname, + errarg->rel->remoterel.relname, + errarg->rel->remoterel.attnames[errarg->remote_attnum], + errarg->remote_xid, + LSN_FORMAT_ARGS(errarg->finish_lsn)); + } + } } /* Set transaction information of apply error callback */ diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c index 2cbca4a087..1aaca04982 100644 --- a/src/backend/replication/pgoutput/pgoutput.c +++ b/src/backend/replication/pgoutput/pgoutput.c @@ -1820,6 +1820,8 @@ pgoutput_stream_abort(struct LogicalDecodingContext *ctx, XLogRecPtr abort_lsn) { ReorderBufferTXN *toptxn; + bool write_abort_lsn = false; + PGOutputData *data = (PGOutputData *) ctx->output_plugin_private; /* * The abort should happen outside streaming block, even for streamed @@ -1832,8 +1834,13 @@ pgoutput_stream_abort(struct LogicalDecodingContext *ctx, Assert(rbtxn_is_streamed(toptxn)); + /* We only send abort_lsn and abort_time if the subscriber needs them. */ + if (data->protocol_version >= LOGICALREP_PROTO_STREAM_PARALLEL_VERSION_NUM) + write_abort_lsn = true; + OutputPluginPrepareWrite(ctx, true); - logicalrep_write_stream_abort(ctx->out, toptxn->xid, txn->xid); + logicalrep_write_stream_abort(ctx->out, toptxn->xid, txn, abort_lsn, + write_abort_lsn); OutputPluginWrite(ctx, true); cleanup_rel_sync_cache(toptxn->xid, false); diff --git a/src/backend/utils/activity/wait_event.c b/src/backend/utils/activity/wait_event.c index 87c15b9c6f..ba781e6f08 100644 --- a/src/backend/utils/activity/wait_event.c +++ b/src/backend/utils/activity/wait_event.c @@ -388,6 +388,9 @@ pgstat_get_wait_ipc(WaitEventIPC w) case WAIT_EVENT_HASH_GROW_BUCKETS_REINSERT: event_name = "HashGrowBucketsReinsert"; break; + case WAIT_EVENT_LOGICAL_APPLY_WORKER_STATE_CHANGE: + event_name = "LogicalApplyWorkerStateChange"; + break; case WAIT_EVENT_LOGICAL_SYNC_DATA: event_name = "LogicalSyncData"; break; diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c index 0328029d43..4284bcbcd1 100644 --- a/src/backend/utils/misc/guc.c +++ b/src/backend/utils/misc/guc.c @@ -3220,6 +3220,18 @@ static struct config_int ConfigureNamesInt[] = NULL, NULL, NULL }, + { + {"max_apply_bgworkers_per_subscription", + PGC_SIGHUP, + REPLICATION_SUBSCRIBERS, + gettext_noop("Maximum number of apply background workers per subscription."), + NULL, + }, + &max_apply_bgworkers_per_subscription, + 2, 0, MAX_BACKENDS, + NULL, NULL, NULL + }, + { {"log_rotation_age", PGC_SIGHUP, LOGGING_WHERE, gettext_noop("Sets the amount of time to wait before forcing " diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample index b4bc06e5f5..ad18710af4 100644 --- a/src/backend/utils/misc/postgresql.conf.sample +++ b/src/backend/utils/misc/postgresql.conf.sample @@ -360,6 +360,7 @@ #max_logical_replication_workers = 4 # taken from max_worker_processes # (change requires restart) #max_sync_workers_per_subscription = 2 # taken from max_logical_replication_workers +#max_apply_bgworkers_per_subscription = 2 # taken from max_logical_replication_workers #------------------------------------------------------------------------------ diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index e4fdb6b75b..9099fe5f74 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -4456,7 +4456,7 @@ getSubscriptions(Archive *fout) if (fout->remoteVersion >= 140000) appendPQExpBufferStr(query, " s.substream,\n"); else - appendPQExpBufferStr(query, " false AS substream,\n"); + appendPQExpBufferStr(query, " 'f' AS substream,\n"); if (fout->remoteVersion >= 150000) appendPQExpBufferStr(query, @@ -4586,8 +4586,10 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo) if (strcmp(subinfo->subbinary, "t") == 0) appendPQExpBufferStr(query, ", binary = true"); - if (strcmp(subinfo->substream, "f") != 0) + if (strcmp(subinfo->substream, "t") == 0) appendPQExpBufferStr(query, ", streaming = on"); + else if (strcmp(subinfo->substream, "p") == 0) + appendPQExpBufferStr(query, ", streaming = parallel"); if (strcmp(subinfo->subtwophasestate, two_phase_disabled) != 0) appendPQExpBufferStr(query, ", two_phase = on"); diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h index d1260f590c..d54540f5f5 100644 --- a/src/include/catalog/pg_subscription.h +++ b/src/include/catalog/pg_subscription.h @@ -68,7 +68,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW bool subbinary; /* True if the subscription wants the * publisher to send data in binary */ - bool substream; /* Stream in-progress transactions. */ + char substream; /* Stream in-progress transactions. + * See SUBSTREAM_xxx constants. */ char subtwophasestate; /* Stream two-phase transactions */ @@ -109,7 +110,8 @@ typedef struct Subscription bool enabled; /* Indicates if the subscription is enabled */ bool binary; /* Indicates if the subscription wants data in * binary format */ - bool stream; /* Allow streaming in-progress transactions. */ + char stream; /* Allow streaming in-progress transactions. + * See SUBSTREAM_xxx constants. */ char twophasestate; /* Allow streaming two-phase transactions */ bool disableonerr; /* Indicates if the subscription should be * automatically disabled if a worker error @@ -120,6 +122,21 @@ typedef struct Subscription List *publications; /* List of publication names to subscribe to */ } Subscription; +/* Disallow streaming in-progress transactions */ +#define SUBSTREAM_OFF 'f' + +/* + * Streaming in-progress transactions are written to a temporary file and + * applied only after the transaction is committed on upstream. + */ +#define SUBSTREAM_ON 't' + +/* + * Streaming in-progress transactions are applied immediately via a background + * worker + */ +#define SUBSTREAM_PARALLEL 'p' + extern Subscription *GetSubscription(Oid subid, bool missing_ok); extern void FreeSubscription(Subscription *sub); extern void DisableSubscription(Oid subid); diff --git a/src/include/replication/logicallauncher.h b/src/include/replication/logicallauncher.h index f1e2821e25..ac8ef94381 100644 --- a/src/include/replication/logicallauncher.h +++ b/src/include/replication/logicallauncher.h @@ -14,6 +14,7 @@ extern PGDLLIMPORT int max_logical_replication_workers; extern PGDLLIMPORT int max_sync_workers_per_subscription; +extern PGDLLIMPORT int max_apply_bgworkers_per_subscription; extern void ApplyLauncherRegister(void); extern void ApplyLauncherMain(Datum main_arg); diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h index a771ab8ff3..0f74a3392b 100644 --- a/src/include/replication/logicalproto.h +++ b/src/include/replication/logicalproto.h @@ -32,12 +32,17 @@ * * LOGICALREP_PROTO_TWOPHASE_VERSION_NUM is the minimum protocol version with * support for two-phase commit decoding (at prepare time). Introduced in PG15. + * + * LOGICALREP_PROTO_STREAM_PARALLEL_VERSION_NUM is the minimum protocol version + * with support for streaming large transactions in apply background worker. + * Introduced in PG16. */ #define LOGICALREP_PROTO_MIN_VERSION_NUM 1 #define LOGICALREP_PROTO_VERSION_NUM 1 #define LOGICALREP_PROTO_STREAM_VERSION_NUM 2 #define LOGICALREP_PROTO_TWOPHASE_VERSION_NUM 3 -#define LOGICALREP_PROTO_MAX_VERSION_NUM LOGICALREP_PROTO_TWOPHASE_VERSION_NUM +#define LOGICALREP_PROTO_STREAM_PARALLEL_VERSION_NUM 4 +#define LOGICALREP_PROTO_MAX_VERSION_NUM LOGICALREP_PROTO_STREAM_PARALLEL_VERSION_NUM /* * Logical message types @@ -175,6 +180,17 @@ typedef struct LogicalRepRollbackPreparedTxnData char gid[GIDSIZE]; } LogicalRepRollbackPreparedTxnData; +/* + * Transaction protocol information for stream abort. + */ +typedef struct LogicalRepStreamAbortData +{ + TransactionId xid; + TransactionId subxid; + XLogRecPtr abort_lsn; + TimestampTz abort_time; +} LogicalRepStreamAbortData; + extern void logicalrep_write_begin(StringInfo out, ReorderBufferTXN *txn); extern void logicalrep_read_begin(StringInfo in, LogicalRepBeginData *begin_data); @@ -246,9 +262,12 @@ extern void logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn extern TransactionId logicalrep_read_stream_commit(StringInfo out, LogicalRepCommitData *commit_data); extern void logicalrep_write_stream_abort(StringInfo out, TransactionId xid, - TransactionId subxid); -extern void logicalrep_read_stream_abort(StringInfo in, TransactionId *xid, - TransactionId *subxid); + ReorderBufferTXN *txn, + XLogRecPtr abort_lsn, + bool write_abort_lsn); +extern void logicalrep_read_stream_abort(StringInfo in, + LogicalRepStreamAbortData *abort_data, + bool include_abort_lsn); extern char *logicalrep_message_type(LogicalRepMsgType action); #endif /* LOGICAL_PROTO_H */ diff --git a/src/include/replication/logicalworker.h b/src/include/replication/logicalworker.h index cd1b6e8afc..6a1af7f13c 100644 --- a/src/include/replication/logicalworker.h +++ b/src/include/replication/logicalworker.h @@ -13,6 +13,7 @@ #define LOGICALWORKER_H extern void ApplyWorkerMain(Datum main_arg); +extern void ApplyBgworkerMain(Datum main_arg); extern bool IsLogicalWorker(void); diff --git a/src/include/replication/origin.h b/src/include/replication/origin.h index 2d1b5e5c28..c7389b40a7 100644 --- a/src/include/replication/origin.h +++ b/src/include/replication/origin.h @@ -53,7 +53,7 @@ extern XLogRecPtr replorigin_get_progress(RepOriginId node, bool flush); extern void replorigin_session_advance(XLogRecPtr remote_commit, XLogRecPtr local_commit); -extern void replorigin_session_setup(RepOriginId node); +extern void replorigin_session_setup(RepOriginId node, bool must_acquire); extern void replorigin_session_reset(void); extern XLogRecPtr replorigin_session_get_progress(bool flush); diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h index d109d0baed..d2a80d79e5 100644 --- a/src/include/replication/reorderbuffer.h +++ b/src/include/replication/reorderbuffer.h @@ -301,6 +301,7 @@ typedef struct ReorderBufferTXN { TimestampTz commit_time; TimestampTz prepare_time; + TimestampTz abort_time; } xact_time; /* @@ -647,9 +648,11 @@ extern void ReorderBufferFinishPrepared(ReorderBuffer *rb, TransactionId xid, extern void ReorderBufferAssignChild(ReorderBuffer *, TransactionId, TransactionId, XLogRecPtr commit_lsn); extern void ReorderBufferCommitChild(ReorderBuffer *, TransactionId, TransactionId, XLogRecPtr commit_lsn, XLogRecPtr end_lsn); -extern void ReorderBufferAbort(ReorderBuffer *, TransactionId, XLogRecPtr lsn); +extern void ReorderBufferAbort(ReorderBuffer *, TransactionId, XLogRecPtr lsn, + TimestampTz abort_time); extern void ReorderBufferAbortOld(ReorderBuffer *, TransactionId xid); -extern void ReorderBufferForget(ReorderBuffer *, TransactionId, XLogRecPtr lsn); +extern void ReorderBufferForget(ReorderBuffer *, TransactionId, XLogRecPtr lsn, + TimestampTz abort_time); extern void ReorderBufferInvalidate(ReorderBuffer *, TransactionId, XLogRecPtr lsn); extern void ReorderBufferSetBaseSnapshot(ReorderBuffer *, TransactionId, XLogRecPtr lsn, struct SnapshotData *snap); diff --git a/src/include/replication/worker_internal.h b/src/include/replication/worker_internal.h index 901845abc2..5be8f5755e 100644 --- a/src/include/replication/worker_internal.h +++ b/src/include/replication/worker_internal.h @@ -17,8 +17,11 @@ #include "access/xlogdefs.h" #include "catalog/pg_subscription.h" #include "datatype/timestamp.h" +#include "replication/logicalrelation.h" #include "storage/fileset.h" #include "storage/lock.h" +#include "storage/shm_mq.h" +#include "storage/shm_toc.h" #include "storage/spin.h" @@ -60,6 +63,9 @@ typedef struct LogicalRepWorker */ FileSet *stream_fileset; + /* Indicates if this slot is used for an apply background worker. */ + bool subworker; + /* Stats. */ XLogRecPtr last_lsn; TimestampTz last_send_time; @@ -68,8 +74,68 @@ typedef struct LogicalRepWorker TimestampTz reply_time; } LogicalRepWorker; +/* Struct for saving and restoring apply errcontext information */ +typedef struct ApplyErrorCallbackArg +{ + LogicalRepMsgType command; /* 0 if invalid */ + LogicalRepRelMapEntry *rel; + + /* Remote node information */ + int remote_attnum; /* -1 if invalid */ + TransactionId remote_xid; + XLogRecPtr finish_lsn; + char *origin_name; +} ApplyErrorCallbackArg; + +/* + * Status of apply background worker. + */ +typedef enum ApplyBgworkerStatus +{ + APPLY_BGWORKER_BUSY = 0, /* assigned to a transaction */ + APPLY_BGWORKER_FINISHED, /* transaction is completed */ + APPLY_BGWORKER_EXIT /* exit */ +} ApplyBgworkerStatus; + +/* + * Struct for sharing information between apply main and apply background + * workers. + */ +typedef struct ApplyBgworkerShared +{ + slock_t mutex; + + /* Status of apply background worker. */ + ApplyBgworkerStatus status; + + /* server version of publisher. */ + int server_version; + + TransactionId stream_xid; + uint32 n; /* id of apply background worker */ +} ApplyBgworkerShared; + +/* + * Struct for maintaining an apply background worker. + */ +typedef struct ApplyBgworkerState +{ + shm_mq_handle *mq_handle; + dsm_segment *dsm_seg; + ApplyBgworkerShared volatile *pstate; +} ApplyBgworkerState; + /* Main memory context for apply worker. Permanent during worker lifetime. */ extern PGDLLIMPORT MemoryContext ApplyContext; +extern PGDLLIMPORT MemoryContext ApplyMessageContext; + +extern PGDLLIMPORT ApplyErrorCallbackArg apply_error_callback_arg; + +extern PGDLLIMPORT bool MySubscriptionValid; + +extern PGDLLIMPORT volatile ApplyBgworkerShared *MyParallelState; + +extern PGDLLIMPORT List *subxactlist; /* libpqreceiver connection */ extern PGDLLIMPORT struct WalReceiverConn *LogRepWorkerWalRcvConn; @@ -79,18 +145,22 @@ extern PGDLLIMPORT Subscription *MySubscription; extern PGDLLIMPORT LogicalRepWorker *MyLogicalRepWorker; extern PGDLLIMPORT bool in_remote_transaction; +extern PGDLLIMPORT bool in_streamed_transaction; +extern PGDLLIMPORT TransactionId stream_xid; extern void logicalrep_worker_attach(int slot); extern LogicalRepWorker *logicalrep_worker_find(Oid subid, Oid relid, bool only_running); extern List *logicalrep_workers_find(Oid subid, bool only_running); -extern void logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, - Oid userid, Oid relid); +extern bool logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, + Oid userid, Oid relid, + dsm_handle subworker_dsm); extern void logicalrep_worker_stop(Oid subid, Oid relid); extern void logicalrep_worker_wakeup(Oid subid, Oid relid); extern void logicalrep_worker_wakeup_ptr(LogicalRepWorker *worker); extern int logicalrep_sync_worker_count(Oid subid); +extern int logicalrep_apply_bgworker_count(Oid subid); extern void ReplicationOriginNameForTablesync(Oid suboid, Oid relid, char *originname, int szorgname); @@ -103,10 +173,38 @@ extern void process_syncing_tables(XLogRecPtr current_lsn); extern void invalidate_syncing_table_states(Datum arg, int cacheid, uint32 hashvalue); +extern void UpdateWorkerStats(XLogRecPtr last_lsn, TimestampTz send_time, + bool reply); + +extern void apply_dispatch(StringInfo s); + +/* Function for apply error callback */ +extern void apply_error_callback(void *arg); + +extern void subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue); + +/* Apply background worker setup and interactions */ +extern ApplyBgworkerState *apply_bgworker_start(TransactionId xid); +extern ApplyBgworkerState *apply_bgworker_find(TransactionId xid); +extern void apply_bgworker_wait_for(ApplyBgworkerState *wstate, + ApplyBgworkerStatus wait_for_status); +extern void apply_bgworker_send_data(ApplyBgworkerState *wstate, Size nbytes, + const void *data); +extern void apply_bgworker_free(ApplyBgworkerState *wstate); +extern void apply_bgworker_check_status(void); +extern void apply_bgworker_set_status(ApplyBgworkerStatus status); +extern void apply_bgworker_subxact_info_add(TransactionId current_xid); + static inline bool am_tablesync_worker(void) { return OidIsValid(MyLogicalRepWorker->relid); } +static inline bool +am_apply_bgworker(void) +{ + return MyLogicalRepWorker->subworker; +} + #endif /* WORKER_INTERNAL_H */ diff --git a/src/include/utils/wait_event.h b/src/include/utils/wait_event.h index b578e2ec75..c2d2a114d7 100644 --- a/src/include/utils/wait_event.h +++ b/src/include/utils/wait_event.h @@ -105,6 +105,7 @@ typedef enum WAIT_EVENT_HASH_GROW_BUCKETS_ALLOCATE, WAIT_EVENT_HASH_GROW_BUCKETS_ELECT, WAIT_EVENT_HASH_GROW_BUCKETS_REINSERT, + WAIT_EVENT_LOGICAL_APPLY_WORKER_STATE_CHANGE, WAIT_EVENT_LOGICAL_SYNC_DATA, WAIT_EVENT_LOGICAL_SYNC_STATE_CHANGE, WAIT_EVENT_MQ_INTERNAL, diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out index 5db7146e06..919266ae06 100644 --- a/src/test/regress/expected/subscription.out +++ b/src/test/regress/expected/subscription.out @@ -197,7 +197,7 @@ ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); DROP SUBSCRIPTION regress_testsub; -- fail - streaming must be boolean CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = foo); -ERROR: streaming requires a Boolean value +ERROR: streaming requires a Boolean value or "parallel" -- now it works CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = true); WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 34a76ceb60..4137dc77b4 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -116,6 +116,10 @@ Append AppendPath AppendRelInfo AppendState +ApplyBgworkerEntry +ApplyBgworkerShared +ApplyBgworkerState +ApplyBgworkerStatus ApplyErrorCallbackArg ApplyExecutionData ApplySubXactData @@ -1485,6 +1489,7 @@ LogicalRepRelId LogicalRepRelMapEntry LogicalRepRelation LogicalRepRollbackPreparedTxnData +LogicalRepStreamAbortData LogicalRepTupleData LogicalRepTyp LogicalRepWorker -- 2.23.0.windows.1 [application/octet-stream] v17-0002-Test-streaming-parallel-option-in-tap-test.patch (69.1K, ../../OSZPR01MB6278023241CEDDE233C8386F9E899@OSZPR01MB6278.jpnprd01.prod.outlook.com/3-v17-0002-Test-streaming-parallel-option-in-tap-test.patch) download | inline diff: From 3d863e38583a97a27afcf8a24751dfd2f6eb3d3c Mon Sep 17 00:00:00 2001 From: "shiy.fnst" <[email protected]> Date: Fri, 13 May 2022 14:50:30 +0800 Subject: [PATCH v17 2/4] Test streaming parallel option in tap test Change all TAP tests using the SUBSCRIPTION "streaming" parameter, so they now test both 'on' and 'parallel' values. --- src/test/subscription/t/015_stream.pl | 199 ++++--- src/test/subscription/t/016_stream_subxact.pl | 119 +++-- src/test/subscription/t/017_stream_ddl.pl | 188 ++++--- .../t/018_stream_subxact_abort.pl | 195 ++++--- .../t/019_stream_subxact_ddl_abort.pl | 110 +++- .../subscription/t/022_twophase_cascade.pl | 363 +++++++------ .../subscription/t/023_twophase_stream.pl | 498 ++++++++++-------- 7 files changed, 1035 insertions(+), 637 deletions(-) diff --git a/src/test/subscription/t/015_stream.pl b/src/test/subscription/t/015_stream.pl index 6561b189de..0bdd234935 100644 --- a/src/test/subscription/t/015_stream.pl +++ b/src/test/subscription/t/015_stream.pl @@ -8,6 +8,116 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" +# parameter so the same code can be run both for the streaming=on and +# streaming=parallel cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_parallel) = @_; + + # Interleave a pair of transactions, each exceeding the 64kB limit. + my $in = ''; + my $out = ''; + + my $timer = IPC::Run::timeout($PostgreSQL::Test::Utils::timeout_default); + + my $h = $node_publisher->background_psql('postgres', \$in, \$out, $timer, + on_error_stop => 0); + + # Check that a background worker starts if "streaming" parameter is + # specified as "parallel". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_parallel) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + $in .= q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + }; + $h->pump_nb; + + if ($is_parallel) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001, 9999) s(i); + DELETE FROM test_tab WHERE a > 5000; + COMMIT; + }); + + $in .= q{ + COMMIT; + \q + }; + $h->finish; # errors make the next test fail, so ignore them here + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), + 'check extra columns contain local defaults'); + + # Test the streaming in binary mode + $node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET (binary = on)"); + + # Insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001, 10000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + COMMIT; + }); + + $node_publisher->wait_for_catchup($appname); + + $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(6667|6667|6667), + 'check extra columns contain local defaults'); + + # Change the local values of the extra columns on the subscriber, + # update publisher, and check that subscriber retains the expected + # values. This is to ensure that non-streaming transactions behave + # properly after a streaming transaction. + $node_subscriber->safe_psql('postgres', + "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'" + ); + $node_publisher->safe_psql('postgres', + "UPDATE test_tab SET b = md5(a::text)"); + + $node_publisher->wait_for_catchup($appname); + + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab" + ); + is($result, qq(6667|6667|6667), + 'check extra columns contain locally changed data'); + + # Cleanup the test data + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE (a > 2)"); + $node_publisher->wait_for_catchup($appname); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -37,6 +147,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -54,82 +168,25 @@ my $result = "SELECT count(*), count(c), count(d = 999) FROM test_tab"); is($result, qq(2|2|2), 'check initial data was copied to subscriber'); -# Interleave a pair of transactions, each exceeding the 64kB limit. -my $in = ''; -my $out = ''; - -my $timer = IPC::Run::timeout($PostgreSQL::Test::Utils::timeout_default); - -my $h = $node_publisher->background_psql('postgres', \$in, \$out, $timer, - on_error_stop => 0); - -$in .= q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -}; -$h->pump_nb; - -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001, 9999) s(i); -DELETE FROM test_tab WHERE a > 5000; -COMMIT; -}); - -$in .= q{ -COMMIT; -\q -}; -$h->finish; # errors make the next test fail, so ignore them here - -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), 'check extra columns contain local defaults'); - -# Test the streaming in binary mode -$node_subscriber->safe_psql('postgres', - "ALTER SUBSCRIPTION tap_sub SET (binary = on)"); - -# Insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001, 10000) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -COMMIT; -}); - -$node_publisher->wait_for_catchup($appname); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(6667|6667|6667), 'check extra columns contain local defaults'); +###################################### +# Test using streaming mode 'parallel' +###################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" +); -# Change the local values of the extra columns on the subscriber, -# update publisher, and check that subscriber retains the expected -# values. This is to ensure that non-streaming transactions behave -# properly after a streaming transaction. $node_subscriber->safe_psql('postgres', - "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'" -); -$node_publisher->safe_psql('postgres', - "UPDATE test_tab SET b = md5(a::text)"); + "ALTER SUBSCRIPTION tap_sub SET(streaming = parallel, binary = off)"); -$node_publisher->wait_for_catchup($appname); +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab" -); -is($result, qq(6667|6667|6667), - 'check extra columns contain locally changed data'); +test_streaming($node_publisher, $node_subscriber, $appname, 1); $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/016_stream_subxact.pl b/src/test/subscription/t/016_stream_subxact.pl index f27f1694f2..45429dddba 100644 --- a/src/test/subscription/t/016_stream_subxact.pl +++ b/src/test/subscription/t/016_stream_subxact.pl @@ -8,6 +8,72 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" +# parameter so the same code can be run both for the streaming=on and +# streaming=parallel cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_parallel) = @_; + + # Check that a background worker starts if "streaming" parameter is + # specified as "parallel". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_parallel) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + # Insert, update and delete enough rows to exceed 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series( 3, 500) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501, 1000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001, 1500) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + SAVEPOINT s3; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501, 2000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + SAVEPOINT s4; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001, 2500) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + COMMIT; + }); + + if ($is_parallel) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(1667|1667|1667), + 'check data was copied to subscriber in streaming mode and extra columns contain local defaults' + ); + + # Cleanup the test data + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE (a > 2)"); + $node_publisher->wait_for_catchup($appname); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -37,6 +103,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -54,41 +124,26 @@ my $result = "SELECT count(*), count(c), count(d = 999) FROM test_tab"); is($result, qq(2|2|2), 'check initial data was copied to subscriber'); -# Insert, update and delete enough rows to exceed 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series( 3, 500) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501, 1000) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001, 1500) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -SAVEPOINT s3; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501, 2000) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -SAVEPOINT s4; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001, 2500) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -COMMIT; -}); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(1667|1667|1667), - 'check data was copied to subscriber in streaming mode and extra columns contain local defaults' +###################################### +# Test using streaming mode 'parallel' +###################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" ); +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = parallel)"); + +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; + +test_streaming($node_publisher, $node_subscriber, $appname, 1); + $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/017_stream_ddl.pl b/src/test/subscription/t/017_stream_ddl.pl index 0bce63b716..52dfef4780 100644 --- a/src/test/subscription/t/017_stream_ddl.pl +++ b/src/test/subscription/t/017_stream_ddl.pl @@ -8,6 +8,111 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" +# parameter so the same code can be run both for the streaming=on and +# streaming=parallel cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_parallel) = @_; + + # a small (non-streamed) transaction with DDL and DML + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab VALUES (3, md5(3::text)); + ALTER TABLE test_tab ADD COLUMN c INT; + SAVEPOINT s1; + INSERT INTO test_tab VALUES (4, md5(4::text), -4); + COMMIT; + }); + + # Check that a background worker starts if "streaming" parameter is + # specified as "parallel". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_parallel) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + # large (streamed) transaction with DDL and DML + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(5, 1000) s(i); + ALTER TABLE test_tab ADD COLUMN d INT; + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001, 2000) s(i); + COMMIT; + }); + + if ($is_parallel) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + # a small (non-streamed) transaction with DDL and DML + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab VALUES (2001, md5(2001::text), -2001, 2*2001); + ALTER TABLE test_tab ADD COLUMN e INT; + SAVEPOINT s1; + INSERT INTO test_tab VALUES (2002, md5(2002::text), -2002, 2*2002, -3*2002); + COMMIT; + }); + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d), count(e) FROM test_tab"); + is($result, qq(2002|1999|1002|1), + 'check data was copied to subscriber in streaming mode and extra columns contain local defaults' + ); + + # A large (streamed) transaction with DDL and DML. One of the DDL is performed + # after DML to ensure that we invalidate the schema sent for test_tab so that + # the next transaction has to send the schema again. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(2003,5000) s(i); + ALTER TABLE test_tab ADD COLUMN f INT; + COMMIT; + }); + + # A small transaction that won't get streamed. This is just to ensure that we + # send the schema again to reflect the last column added in the previous test. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i, 4*i FROM generate_series(5001,5005) s(i); + COMMIT; + }); + + $node_publisher->wait_for_catchup($appname); + + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d), count(e), count(f) FROM test_tab" + ); + is($result, qq(5005|5002|4005|3004|5), + 'check data was copied to subscriber for both streaming and non-streaming transactions' + ); + + # Cleanup the test data + $node_publisher->safe_psql( + 'postgres', q{ + DELETE FROM test_tab WHERE (a > 2); + ALTER TABLE test_tab DROP COLUMN c, DROP COLUMN d, DROP COLUMN e, DROP COLUMN f; + }); + $node_publisher->wait_for_catchup($appname); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -37,6 +142,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -54,76 +163,25 @@ my $result = "SELECT count(*), count(c), count(d = 999) FROM test_tab"); is($result, qq(2|0|0), 'check initial data was copied to subscriber'); -# a small (non-streamed) transaction with DDL and DML -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab VALUES (3, md5(3::text)); -ALTER TABLE test_tab ADD COLUMN c INT; -SAVEPOINT s1; -INSERT INTO test_tab VALUES (4, md5(4::text), -4); -COMMIT; -}); - -# large (streamed) transaction with DDL and DML -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(5, 1000) s(i); -ALTER TABLE test_tab ADD COLUMN d INT; -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001, 2000) s(i); -COMMIT; -}); - -# a small (non-streamed) transaction with DDL and DML -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab VALUES (2001, md5(2001::text), -2001, 2*2001); -ALTER TABLE test_tab ADD COLUMN e INT; -SAVEPOINT s1; -INSERT INTO test_tab VALUES (2002, md5(2002::text), -2002, 2*2002, -3*2002); -COMMIT; -}); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d), count(e) FROM test_tab"); -is($result, qq(2002|1999|1002|1), - 'check data was copied to subscriber in streaming mode and extra columns contain local defaults' +###################################### +# Test using streaming mode 'parallel' +###################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" ); -# A large (streamed) transaction with DDL and DML. One of the DDL is performed -# after DML to ensure that we invalidate the schema sent for test_tab so that -# the next transaction has to send the schema again. -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(2003,5000) s(i); -ALTER TABLE test_tab ADD COLUMN f INT; -COMMIT; -}); - -# A small transaction that won't get streamed. This is just to ensure that we -# send the schema again to reflect the last column added in the previous test. -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i, 4*i FROM generate_series(5001,5005) s(i); -COMMIT; -}); +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = parallel)"); -$node_publisher->wait_for_catchup($appname); +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d), count(e), count(f) FROM test_tab"); -is($result, qq(5005|5002|4005|3004|5), - 'check data was copied to subscriber for both streaming and non-streaming transactions' -); +test_streaming($node_publisher, $node_subscriber, $appname, 1); $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/018_stream_subxact_abort.pl b/src/test/subscription/t/018_stream_subxact_abort.pl index 7155442e76..68f0e4b0d1 100644 --- a/src/test/subscription/t/018_stream_subxact_abort.pl +++ b/src/test/subscription/t/018_stream_subxact_abort.pl @@ -8,6 +8,113 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" +# parameter so the same code can be run both for the streaming=on and +# streaming=parallel cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_parallel) = @_; + + # Check that a background worker starts if "streaming" parameter is + # specified as "parallel". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_parallel) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + # large (streamed) transaction with DDL, DML and ROLLBACKs + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i); + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501,1000) s(i); + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001,1500) s(i); + SAVEPOINT s3; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501,2000) s(i); + ROLLBACK TO s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001,2500) s(i); + ROLLBACK TO s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2501,3000) s(i); + SAVEPOINT s4; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3001,3500) s(i); + SAVEPOINT s5; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3501,4000) s(i); + COMMIT; + }); + + if ($is_parallel) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c) FROM test_tab"); + is($result, qq(2000|0), + 'check rollback to savepoint was reflected on subscriber and extra columns contain local defaults' + ); + + # large (streamed) transaction with subscriber receiving out of order + # subtransaction ROLLBACKs + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(4001,4500) s(i); + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001,5500) s(i); + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(6001,6500) s(i); + SAVEPOINT s3; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(7001,7500) s(i); + RELEASE s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(8001,8500) s(i); + ROLLBACK TO s1; + COMMIT; + }); + + $node_publisher->wait_for_catchup($appname); + + $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c) FROM test_tab"); + is($result, qq(2500|0), + 'check rollback to savepoint was reflected on subscriber'); + + # large (streamed) transaction with subscriber receiving rollback + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(8501,9000) s(i); + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(9001,9500) s(i); + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(9501,10000) s(i); + ROLLBACK; + }); + + $node_publisher->wait_for_catchup($appname); + + $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c) FROM test_tab"); + is($result, qq(2500|0), 'check rollback was reflected on subscriber'); + + # Cleanup the test data + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE (a > 2)"); + $node_publisher->wait_for_catchup($appname); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -36,6 +143,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -53,81 +164,25 @@ my $result = "SELECT count(*), count(c) FROM test_tab"); is($result, qq(2|0), 'check initial data was copied to subscriber'); -# large (streamed) transaction with DDL, DML and ROLLBACKs -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i); -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501,1000) s(i); -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001,1500) s(i); -SAVEPOINT s3; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501,2000) s(i); -ROLLBACK TO s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001,2500) s(i); -ROLLBACK TO s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2501,3000) s(i); -SAVEPOINT s4; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3001,3500) s(i); -SAVEPOINT s5; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3501,4000) s(i); -COMMIT; -}); - -$node_publisher->wait_for_catchup($appname); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c) FROM test_tab"); -is($result, qq(2000|0), - 'check rollback to savepoint was reflected on subscriber and extra columns contain local defaults' +###################################### +# Test using streaming mode 'parallel' +###################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" ); -# large (streamed) transaction with subscriber receiving out of order -# subtransaction ROLLBACKs -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(4001,4500) s(i); -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001,5500) s(i); -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(6001,6500) s(i); -SAVEPOINT s3; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(7001,7500) s(i); -RELEASE s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(8001,8500) s(i); -ROLLBACK TO s1; -COMMIT; -}); - -$node_publisher->wait_for_catchup($appname); +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = parallel)"); -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c) FROM test_tab"); -is($result, qq(2500|0), - 'check rollback to savepoint was reflected on subscriber'); - -# large (streamed) transaction with subscriber receiving rollback -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(8501,9000) s(i); -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(9001,9500) s(i); -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(9501,10000) s(i); -ROLLBACK; -}); +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c) FROM test_tab"); -is($result, qq(2500|0), 'check rollback was reflected on subscriber'); +test_streaming($node_publisher, $node_subscriber, $appname, 1); $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/019_stream_subxact_ddl_abort.pl b/src/test/subscription/t/019_stream_subxact_ddl_abort.pl index dbd0fca4d1..b276063721 100644 --- a/src/test/subscription/t/019_stream_subxact_ddl_abort.pl +++ b/src/test/subscription/t/019_stream_subxact_ddl_abort.pl @@ -9,6 +9,69 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" +# parameter so the same code can be run both for the streaming=on and +# streaming=parallel cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_parallel) = @_; + + # Check that a background worker starts if "streaming" parameter is + # specified as "parallel". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_parallel) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + # large (streamed) transaction with DDL, DML and ROLLBACKs + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i); + ALTER TABLE test_tab ADD COLUMN c INT; + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(501,1000) s(i); + ALTER TABLE test_tab ADD COLUMN d INT; + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001,1500) s(i); + ALTER TABLE test_tab ADD COLUMN e INT; + SAVEPOINT s3; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(1501,2000) s(i); + ALTER TABLE test_tab DROP COLUMN c; + ROLLBACK TO s1; + INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(501,1000) s(i); + COMMIT; + }); + + if ($is_parallel) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c) FROM test_tab"); + is($result, qq(1000|500), + 'check rollback to savepoint was reflected on subscriber and extra columns contain local defaults' + ); + + # Cleanup the test data + $node_publisher->safe_psql( + 'postgres', q{ + DELETE FROM test_tab WHERE (a > 2); + ALTER TABLE test_tab DROP COLUMN c; + }); + $node_publisher->wait_for_catchup($appname); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -37,6 +100,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -54,35 +121,26 @@ my $result = "SELECT count(*), count(c) FROM test_tab"); is($result, qq(2|0), 'check initial data was copied to subscriber'); -# large (streamed) transaction with DDL, DML and ROLLBACKs -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i); -ALTER TABLE test_tab ADD COLUMN c INT; -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(501,1000) s(i); -ALTER TABLE test_tab ADD COLUMN d INT; -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001,1500) s(i); -ALTER TABLE test_tab ADD COLUMN e INT; -SAVEPOINT s3; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(1501,2000) s(i); -ALTER TABLE test_tab DROP COLUMN c; -ROLLBACK TO s1; -INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(501,1000) s(i); -COMMIT; -}); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c) FROM test_tab"); -is($result, qq(1000|500), - 'check rollback to savepoint was reflected on subscriber and extra columns contain local defaults' +###################################### +# Test using streaming mode 'parallel' +###################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" ); +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = parallel)"); + +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; + +test_streaming($node_publisher, $node_subscriber, $appname, 1); + $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/022_twophase_cascade.pl b/src/test/subscription/t/022_twophase_cascade.pl index 7a797f37ba..0a4152d3be 100644 --- a/src/test/subscription/t/022_twophase_cascade.pl +++ b/src/test/subscription/t/022_twophase_cascade.pl @@ -11,6 +11,208 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" parameter +# so the same code can be run both for the streaming=on and streaming=parallel +# cases. +sub test_streaming +{ + my ($node_A, $node_B, $node_C, $appname_B, $appname_C, $streaming_mode) = + @_; + + my $oldpid_B = $node_A->safe_psql( + 'postgres', " + SELECT pid FROM pg_stat_replication + WHERE application_name = '$appname_B' AND state = 'streaming';"); + my $oldpid_C = $node_B->safe_psql( + 'postgres', " + SELECT pid FROM pg_stat_replication + WHERE application_name = '$appname_C' AND state = 'streaming';"); + + # Setup logical replication streaming mode + + $node_B->safe_psql( + 'postgres', " + ALTER SUBSCRIPTION tap_sub_B + SET (streaming = $streaming_mode);"); + $node_C->safe_psql( + 'postgres', " + ALTER SUBSCRIPTION tap_sub_C + SET (streaming = $streaming_mode)"); + + # Wait for subscribers to finish initialization + + $node_A->poll_query_until( + 'postgres', " + SELECT pid != $oldpid_B FROM pg_stat_replication + WHERE application_name = '$appname_B' AND state = 'streaming';" + ) or die "Timed out while waiting for apply to restart"; + $node_B->poll_query_until( + 'postgres', " + SELECT pid != $oldpid_C FROM pg_stat_replication + WHERE application_name = '$appname_C' AND state = 'streaming';" + ) or die "Timed out while waiting for apply to restart"; + + ############################### + # Test 2PC PREPARE / COMMIT PREPARED. + # 1. Data is streamed as a 2PC transaction. + # 2. Then do commit prepared. + # + # Expect all data is replicated on subscriber(s) after the commit. + ############################### + + # Check that a background worker starts if "streaming" optparameterion is + # specified as "parallel". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($streaming_mode eq 'parallel') + { + $node_B->append_conf('postgresql.conf', "log_min_messages = debug1"); + $node_B->reload; + + $node_C->append_conf('postgresql.conf', "log_min_messages = debug1"); + $node_C->reload; + } + + # Insert, update and delete enough rows to exceed the 64kB limit. + # Then 2PC PREPARE + $node_A->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + if ($streaming_mode eq 'parallel') + { + $node_B->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_B->append_conf('postgresql.conf', "log_min_messages = warning"); + $node_B->reload; + + $node_C->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_C->append_conf('postgresql.conf', "log_min_messages = warning"); + $node_C->reload; + } + + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); + + # check the transaction state is prepared on subscriber(s) + my $result = + $node_B->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber B'); + $result = + $node_C->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber C'); + + # 2PC COMMIT + $node_A->safe_psql('postgres', "COMMIT PREPARED 'test_prepared_tab';"); + + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); + + # check that transaction was committed on subscriber(s) + $result = $node_B->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), + 'Rows inserted by 2PC have committed on subscriber B, and extra columns have local defaults' + ); + $result = $node_C->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), + 'Rows inserted by 2PC have committed on subscriber C, and extra columns have local defaults' + ); + + # check the transaction state is ended on subscriber(s) + $result = + $node_B->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is committed on subscriber B'); + $result = + $node_C->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is committed on subscriber C'); + + ############################### + # Test 2PC PREPARE with a nested ROLLBACK TO SAVEPOINT. + # 0. Cleanup from previous test leaving only 2 rows. + # 1. Insert one more row. + # 2. Record a SAVEPOINT. + # 3. Data is streamed using 2PC. + # 4. Do rollback to SAVEPOINT prior to the streamed inserts. + # 5. Then COMMIT PREPARED. + # + # Expect data after the SAVEPOINT is aborted leaving only 3 rows (= 2 original + 1 from step 1). + ############################### + + # First, delete the data except for 2 rows (delete will be replicated) + $node_A->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); + + # 2PC PREPARE with a nested ROLLBACK TO SAVEPOINT + $node_A->safe_psql( + 'postgres', " + BEGIN; + INSERT INTO test_tab VALUES (9999, 'foobar'); + SAVEPOINT sp_inner; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + ROLLBACK TO SAVEPOINT sp_inner; + PREPARE TRANSACTION 'outer'; + "); + + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); + + # check the transaction state prepared on subscriber(s) + $result = + $node_B->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber B'); + $result = + $node_C->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber C'); + + # 2PC COMMIT + $node_A->safe_psql('postgres', "COMMIT PREPARED 'outer';"); + + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); + + # check the transaction state is ended on subscriber + $result = + $node_B->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is ended on subscriber B'); + $result = + $node_C->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is ended on subscriber C'); + + # check inserts are visible at subscriber(s). + # All the streamed data (prior to the SAVEPOINT) should be rolled back. + # (9999, 'foobar') should be committed. + $result = $node_B->safe_psql('postgres', + "SELECT count(*) FROM test_tab where b = 'foobar';"); + is($result, qq(1), 'Rows committed are present on subscriber B'); + $result = + $node_B->safe_psql('postgres', "SELECT count(*) FROM test_tab;"); + is($result, qq(3), 'Rows committed are present on subscriber B'); + $result = $node_C->safe_psql('postgres', + "SELECT count(*) FROM test_tab where b = 'foobar';"); + is($result, qq(1), 'Rows committed are present on subscriber C'); + $result = + $node_C->safe_psql('postgres', "SELECT count(*) FROM test_tab;"); + is($result, qq(3), 'Rows committed are present on subscriber C'); + + # Cleanup the test data + $node_A->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); +} + ############################### # Setup a cascade of pub/sub nodes. # node_A -> node_B -> node_C @@ -260,160 +462,15 @@ is($result, qq(21), 'Rows committed are present on subscriber C'); # 2PC + STREAMING TESTS # --------------------- -my $oldpid_B = $node_A->safe_psql( - 'postgres', " - SELECT pid FROM pg_stat_replication - WHERE application_name = '$appname_B' AND state = 'streaming';"); -my $oldpid_C = $node_B->safe_psql( - 'postgres', " - SELECT pid FROM pg_stat_replication - WHERE application_name = '$appname_C' AND state = 'streaming';"); - -# Setup logical replication (streaming = on) - -$node_B->safe_psql( - 'postgres', " - ALTER SUBSCRIPTION tap_sub_B - SET (streaming = on);"); -$node_C->safe_psql( - 'postgres', " - ALTER SUBSCRIPTION tap_sub_C - SET (streaming = on)"); - -# Wait for subscribers to finish initialization - -$node_A->poll_query_until( - 'postgres', " - SELECT pid != $oldpid_B FROM pg_stat_replication - WHERE application_name = '$appname_B' AND state = 'streaming';" -) or die "Timed out while waiting for apply to restart"; -$node_B->poll_query_until( - 'postgres', " - SELECT pid != $oldpid_C FROM pg_stat_replication - WHERE application_name = '$appname_C' AND state = 'streaming';" -) or die "Timed out while waiting for apply to restart"; - -############################### -# Test 2PC PREPARE / COMMIT PREPARED. -# 1. Data is streamed as a 2PC transaction. -# 2. Then do commit prepared. -# -# Expect all data is replicated on subscriber(s) after the commit. -############################### - -# Insert, update and delete enough rows to exceed the 64kB limit. -# Then 2PC PREPARE -$node_A->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_A->wait_for_catchup($appname_B); -$node_B->wait_for_catchup($appname_C); +################################ +# Test using streaming mode 'on' +################################ +test_streaming($node_A, $node_B, $node_C, $appname_B, $appname_C, 'on'); -# check the transaction state is prepared on subscriber(s) -$result = - $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber B'); -$result = - $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber C'); - -# 2PC COMMIT -$node_A->safe_psql('postgres', "COMMIT PREPARED 'test_prepared_tab';"); - -$node_A->wait_for_catchup($appname_B); -$node_B->wait_for_catchup($appname_C); - -# check that transaction was committed on subscriber(s) -$result = $node_B->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), - 'Rows inserted by 2PC have committed on subscriber B, and extra columns have local defaults' -); -$result = $node_C->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), - 'Rows inserted by 2PC have committed on subscriber C, and extra columns have local defaults' -); - -# check the transaction state is ended on subscriber(s) -$result = - $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is committed on subscriber B'); -$result = - $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is committed on subscriber C'); - -############################### -# Test 2PC PREPARE with a nested ROLLBACK TO SAVEPOINT. -# 0. Cleanup from previous test leaving only 2 rows. -# 1. Insert one more row. -# 2. Record a SAVEPOINT. -# 3. Data is streamed using 2PC. -# 4. Do rollback to SAVEPOINT prior to the streamed inserts. -# 5. Then COMMIT PREPARED. -# -# Expect data after the SAVEPOINT is aborted leaving only 3 rows (= 2 original + 1 from step 1). -############################### - -# First, delete the data except for 2 rows (delete will be replicated) -$node_A->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); - -# 2PC PREPARE with a nested ROLLBACK TO SAVEPOINT -$node_A->safe_psql( - 'postgres', " - BEGIN; - INSERT INTO test_tab VALUES (9999, 'foobar'); - SAVEPOINT sp_inner; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - ROLLBACK TO SAVEPOINT sp_inner; - PREPARE TRANSACTION 'outer'; - "); - -$node_A->wait_for_catchup($appname_B); -$node_B->wait_for_catchup($appname_C); - -# check the transaction state prepared on subscriber(s) -$result = - $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber B'); -$result = - $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber C'); - -# 2PC COMMIT -$node_A->safe_psql('postgres', "COMMIT PREPARED 'outer';"); - -$node_A->wait_for_catchup($appname_B); -$node_B->wait_for_catchup($appname_C); - -# check the transaction state is ended on subscriber -$result = - $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is ended on subscriber B'); -$result = - $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is ended on subscriber C'); - -# check inserts are visible at subscriber(s). -# All the streamed data (prior to the SAVEPOINT) should be rolled back. -# (9999, 'foobar') should be committed. -$result = $node_B->safe_psql('postgres', - "SELECT count(*) FROM test_tab where b = 'foobar';"); -is($result, qq(1), 'Rows committed are present on subscriber B'); -$result = $node_B->safe_psql('postgres', "SELECT count(*) FROM test_tab;"); -is($result, qq(3), 'Rows committed are present on subscriber B'); -$result = $node_C->safe_psql('postgres', - "SELECT count(*) FROM test_tab where b = 'foobar';"); -is($result, qq(1), 'Rows committed are present on subscriber C'); -$result = $node_C->safe_psql('postgres', "SELECT count(*) FROM test_tab;"); -is($result, qq(3), 'Rows committed are present on subscriber C'); +###################################### +# Test using streaming mode 'parallel' +###################################### +test_streaming($node_A, $node_B, $node_C, $appname_B, $appname_C, 'parallel'); ############################### # check all the cleanup diff --git a/src/test/subscription/t/023_twophase_stream.pl b/src/test/subscription/t/023_twophase_stream.pl index d8475d25a4..b89414ab74 100644 --- a/src/test/subscription/t/023_twophase_stream.pl +++ b/src/test/subscription/t/023_twophase_stream.pl @@ -8,6 +8,266 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" +# parameter so the same code can be run both for the streaming=on and +# streaming=parallel cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_parallel) = @_; + + ############################### + # Test 2PC PREPARE / COMMIT PREPARED. + # 1. Data is streamed as a 2PC transaction. + # 2. Then do commit prepared. + # + # Expect all data is replicated on subscriber side after the commit. + ############################### + + # Check that a background worker starts if "streaming" parameter is + # specified as "parallel". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_parallel) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + # check that 2PC gets replicated to subscriber + # Insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + if ($is_parallel) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is in prepared state on subscriber + my $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber'); + + # 2PC transaction gets committed + $node_publisher->safe_psql('postgres', + "COMMIT PREPARED 'test_prepared_tab';"); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is committed on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), + 'Rows inserted by 2PC have committed on subscriber, and extra columns contain local defaults' + ); + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is committed on subscriber'); + + ############################### + # Test 2PC PREPARE / ROLLBACK PREPARED. + # 1. Table is deleted back to 2 rows which are replicated on subscriber. + # 2. Data is streamed using 2PC. + # 3. Do rollback prepared. + # + # Expect data rolls back leaving only the original 2 rows. + ############################### + + # First, delete the data except for 2 rows (will be replicated) + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE a > 2;"); + + # Then insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is in prepared state on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber'); + + # 2PC transaction gets aborted + $node_publisher->safe_psql('postgres', + "ROLLBACK PREPARED 'test_prepared_tab';"); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is aborted on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(2|2|2), + 'Rows inserted by 2PC are rolled back, leaving only the original 2 rows' + ); + + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is aborted on subscriber'); + + ############################### + # Check that 2PC COMMIT PREPARED is decoded properly on crash restart. + # 1. insert, update and delete enough rows to exceed the 64kB limit. + # 2. Then server crashes before the 2PC transaction is committed. + # 3. After servers are restarted the pending transaction is committed. + # + # Expect all data is replicated on subscriber side after the commit. + # Note: both publisher and subscriber do crash/restart. + ############################### + + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + $node_subscriber->stop('immediate'); + $node_publisher->stop('immediate'); + + $node_publisher->start; + $node_subscriber->start; + + # commit post the restart + $node_publisher->safe_psql('postgres', + "COMMIT PREPARED 'test_prepared_tab';"); + $node_publisher->wait_for_catchup($appname); + + # check inserts are visible + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), + 'Rows inserted by 2PC have committed on subscriber, and extra columns contain local defaults' + ); + + ############################### + # Do INSERT after the PREPARE but before ROLLBACK PREPARED. + # 1. Table is deleted back to 2 rows which are replicated on subscriber. + # 2. Data is streamed using 2PC. + # 3. A single row INSERT is done which is after the PREPARE. + # 4. Then do a ROLLBACK PREPARED. + # + # Expect the 2PC data rolls back leaving only 3 rows on the subscriber + # (the original 2 + inserted 1). + ############################### + + # First, delete the data except for 2 rows (will be replicated) + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE a > 2;"); + + # Then insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is in prepared state on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber'); + + # Insert a different record (now we are outside of the 2PC transaction) + # Note: the 2PC transaction still holds row locks so make sure this insert is for a separate primary key + $node_publisher->safe_psql('postgres', + "INSERT INTO test_tab VALUES (99999, 'foobar')"); + + # 2PC transaction gets aborted + $node_publisher->safe_psql('postgres', + "ROLLBACK PREPARED 'test_prepared_tab';"); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is aborted on subscriber, + # but the extra INSERT outside of the 2PC still was replicated + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3|3|3), + 'check the outside insert was copied to subscriber'); + + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is aborted on subscriber'); + + ############################### + # Do INSERT after the PREPARE but before COMMIT PREPARED. + # 1. Table is deleted back to 2 rows which are replicated on subscriber. + # 2. Data is streamed using 2PC. + # 3. A single row INSERT is done which is after the PREPARE. + # 4. Then do a COMMIT PREPARED. + # + # Expect 2PC data + the extra row are on the subscriber + # (the 3334 + inserted 1 = 3335). + ############################### + + # First, delete the data except for 2 rows (will be replicated) + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE a > 2;"); + + # Then insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is in prepared state on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber'); + + # Insert a different record (now we are outside of the 2PC transaction) + # Note: the 2PC transaction still holds row locks so make sure this insert is for a separare primary key + $node_publisher->safe_psql('postgres', + "INSERT INTO test_tab VALUES (99999, 'foobar')"); + + # 2PC transaction gets committed + $node_publisher->safe_psql('postgres', + "COMMIT PREPARED 'test_prepared_tab';"); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is committed on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3335|3335|3335), + 'Rows inserted by 2PC (as well as outside insert) have committed on subscriber, and extra columns contain local defaults' + ); + + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is committed on subscriber'); + + # Cleanup the test data + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE a > 2;"); + $node_publisher->wait_for_catchup($appname); +} + ############################### # Setup ############################### @@ -48,6 +308,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql( 'postgres', " CREATE SUBSCRIPTION tap_sub @@ -70,236 +334,30 @@ my $twophase_query = $node_subscriber->poll_query_until('postgres', $twophase_query) or die "Timed out while waiting for subscriber to enable twophase"; -############################### # Check initial data was copied to subscriber -############################### my $result = $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab"); is($result, qq(2|2|2), 'check initial data was copied to subscriber'); -############################### -# Test 2PC PREPARE / COMMIT PREPARED. -# 1. Data is streamed as a 2PC transaction. -# 2. Then do commit prepared. -# -# Expect all data is replicated on subscriber side after the commit. -############################### - -# check that 2PC gets replicated to subscriber -# Insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is in prepared state on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber'); - -# 2PC transaction gets committed -$node_publisher->safe_psql('postgres', - "COMMIT PREPARED 'test_prepared_tab';"); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is committed on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), - 'Rows inserted by 2PC have committed on subscriber, and extra columns contain local defaults' -); -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is committed on subscriber'); - -############################### -# Test 2PC PREPARE / ROLLBACK PREPARED. -# 1. Table is deleted back to 2 rows which are replicated on subscriber. -# 2. Data is streamed using 2PC. -# 3. Do rollback prepared. -# -# Expect data rolls back leaving only the original 2 rows. -############################### - -# First, delete the data except for 2 rows (will be replicated) -$node_publisher->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); - -# Then insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is in prepared state on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber'); - -# 2PC transaction gets aborted -$node_publisher->safe_psql('postgres', - "ROLLBACK PREPARED 'test_prepared_tab';"); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is aborted on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(2|2|2), - 'Rows inserted by 2PC are rolled back, leaving only the original 2 rows'); - -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is aborted on subscriber'); - -############################### -# Check that 2PC COMMIT PREPARED is decoded properly on crash restart. -# 1. insert, update and delete enough rows to exceed the 64kB limit. -# 2. Then server crashes before the 2PC transaction is committed. -# 3. After servers are restarted the pending transaction is committed. -# -# Expect all data is replicated on subscriber side after the commit. -# Note: both publisher and subscriber do crash/restart. -############################### - -$node_publisher->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_subscriber->stop('immediate'); -$node_publisher->stop('immediate'); - -$node_publisher->start; -$node_subscriber->start; - -# commit post the restart -$node_publisher->safe_psql('postgres', - "COMMIT PREPARED 'test_prepared_tab';"); -$node_publisher->wait_for_catchup($appname); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -# check inserts are visible -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), - 'Rows inserted by 2PC have committed on subscriber, and extra columns contain local defaults' +###################################### +# Test using streaming mode 'parallel' +###################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" ); -############################### -# Do INSERT after the PREPARE but before ROLLBACK PREPARED. -# 1. Table is deleted back to 2 rows which are replicated on subscriber. -# 2. Data is streamed using 2PC. -# 3. A single row INSERT is done which is after the PREPARE. -# 4. Then do a ROLLBACK PREPARED. -# -# Expect the 2PC data rolls back leaving only 3 rows on the subscriber -# (the original 2 + inserted 1). -############################### - -# First, delete the data except for 2 rows (will be replicated) -$node_publisher->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); - -# Then insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is in prepared state on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber'); - -# Insert a different record (now we are outside of the 2PC transaction) -# Note: the 2PC transaction still holds row locks so make sure this insert is for a separate primary key -$node_publisher->safe_psql('postgres', - "INSERT INTO test_tab VALUES (99999, 'foobar')"); - -# 2PC transaction gets aborted -$node_publisher->safe_psql('postgres', - "ROLLBACK PREPARED 'test_prepared_tab';"); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is aborted on subscriber, -# but the extra INSERT outside of the 2PC still was replicated -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3|3|3), 'check the outside insert was copied to subscriber'); - -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is aborted on subscriber'); - -############################### -# Do INSERT after the PREPARE but before COMMIT PREPARED. -# 1. Table is deleted back to 2 rows which are replicated on subscriber. -# 2. Data is streamed using 2PC. -# 3. A single row INSERT is done which is after the PREPARE. -# 4. Then do a COMMIT PREPARED. -# -# Expect 2PC data + the extra row are on the subscriber -# (the 3334 + inserted 1 = 3335). -############################### - -# First, delete the data except for 2 rows (will be replicated) -$node_publisher->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); - -# Then insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is in prepared state on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber'); - -# Insert a different record (now we are outside of the 2PC transaction) -# Note: the 2PC transaction still holds row locks so make sure this insert is for a separare primary key -$node_publisher->safe_psql('postgres', - "INSERT INTO test_tab VALUES (99999, 'foobar')"); - -# 2PC transaction gets committed -$node_publisher->safe_psql('postgres', - "COMMIT PREPARED 'test_prepared_tab';"); - -$node_publisher->wait_for_catchup($appname); +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = parallel)"); -# check that transaction is committed on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3335|3335|3335), - 'Rows inserted by 2PC (as well as outside insert) have committed on subscriber, and extra columns contain local defaults' -); +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is committed on subscriber'); +test_streaming($node_publisher, $node_subscriber, $appname, 1); ############################### # check all the cleanup -- 2.23.0.windows.1 [application/octet-stream] v17-0003-Add-some-checks-before-using-apply-background-wo.patch (36.8K, ../../OSZPR01MB6278023241CEDDE233C8386F9E899@OSZPR01MB6278.jpnprd01.prod.outlook.com/4-v17-0003-Add-some-checks-before-using-apply-background-wo.patch) download | inline diff: From be8230c0a792e28c12f04c413b62e8fcaff91dd2 Mon Sep 17 00:00:00 2001 From: wangw <[email protected]> Date: Tue, 14 Jun 2022 11:23:52 +0800 Subject: [PATCH v17 3/4] Add some checks before using apply background worker to apply changes. streaming=parallel mode has two requirements: 1) The unique column in the relation on the subscriber-side should also be the unique column on the publisher-side; 2) There cannot be any non-immutable functions in the subscriber-side replicated table. Look for functions in the following places: * a. Trigger functions * b. Column default value expressions and domain constraints * c. Constraint expressions * d. Foreign keys --- doc/src/sgml/ref/create_subscription.sgml | 4 + .../replication/logical/applybgworker.c | 42 ++ src/backend/replication/logical/proto.c | 67 ++- src/backend/replication/logical/relation.c | 201 +++++++++ src/backend/replication/logical/tablesync.c | 1 + src/backend/replication/logical/worker.c | 23 +- src/backend/utils/cache/typcache.c | 17 + src/include/replication/logicalproto.h | 1 + src/include/replication/logicalrelation.h | 15 + src/include/replication/worker_internal.h | 1 + src/include/utils/typcache.h | 2 + .../subscription/t/022_twophase_cascade.pl | 6 + .../subscription/t/032_streaming_apply.pl | 391 ++++++++++++++++++ src/tools/pgindent/typedefs.list | 1 + 14 files changed, 762 insertions(+), 10 deletions(-) create mode 100644 src/test/subscription/t/032_streaming_apply.pl diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml index 71dd4aca81..270e3d382e 100644 --- a/doc/src/sgml/ref/create_subscription.sgml +++ b/doc/src/sgml/ref/create_subscription.sgml @@ -240,6 +240,10 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl transaction is committed. Note that if an error happens when applying changes in a background worker, the finish LSN of the remote transaction might not be reported in the server log. + Parallel mode has two requirements: 1) the unique column in the + relation on the subscriber-side should also be the unique column on + the publisher-side; 2) there cannot be any non-immutable functions + in the subscriber-side replicated table. </para> </listitem> </varlistentry> diff --git a/src/backend/replication/logical/applybgworker.c b/src/backend/replication/logical/applybgworker.c index 490633f37c..396dc39605 100644 --- a/src/backend/replication/logical/applybgworker.c +++ b/src/backend/replication/logical/applybgworker.c @@ -784,3 +784,45 @@ apply_bgworker_subxact_info_add(TransactionId current_xid) MemoryContextSwitchTo(oldctx); } } + +/* + * Check if changes on this relation can be applied by an apply background + * worker. + * + * Although the commit order is maintained only allowing one process to commit + * at a time, the access order to the relation has changed. This could cause + * unexpected problems if the unique column on the replicated table is + * inconsistent with the publisher-side or contains non-immutable functions + * when applying transactions in the apply background worker. + */ +void +apply_bgworker_relation_check(LogicalRepRelMapEntry *rel) +{ + /* Skip check if not an apply background worker. */ + if (!am_apply_bgworker()) + return; + + /* + * Partition table checks are done later in function + * apply_handle_tuple_routing. + */ + if (rel->localrel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + return; + + /* + * Return if changes on this relation can be applied by an apply background + * worker. + */ + if (rel->parallel == PARALLEL_APPLY_SAFE) + return; + + /* We are in error mode and should give user correct error. */ + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot replicate target relation \"%s.%s\" in parallel " + "mode", rel->remoterel.nspname, rel->remoterel.relname), + errdetail("The unique column on subscriber is not the unique " + "column on publisher or there is at least one " + "non-immutable function."), + errhint("Please change the streaming option to 'on' instead of 'parallel'."))); +} diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c index affd08cfa4..5a47cd0018 100644 --- a/src/backend/replication/logical/proto.c +++ b/src/backend/replication/logical/proto.c @@ -23,7 +23,8 @@ /* * Protocol message flags. */ -#define LOGICALREP_IS_REPLICA_IDENTITY 1 +#define ATTR_IS_REPLICA_IDENTITY (1 << 0) +#define ATTR_IS_UNIQUE (1 << 1) #define MESSAGE_TRANSACTIONAL (1<<0) #define TRUNCATE_CASCADE (1<<0) @@ -933,11 +934,55 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns) TupleDesc desc; int i; uint16 nliveatts = 0; - Bitmapset *idattrs = NULL; + Bitmapset *idattrs = NULL, + *attunique = NULL; bool replidentfull; desc = RelationGetDescr(rel); + if (rel->rd_rel->relhasindex) + { + List *indexoidlist = RelationGetIndexList(rel); + ListCell *indexoidscan; + + foreach(indexoidscan, indexoidlist) + { + Oid indexoid = lfirst_oid(indexoidscan); + Relation indexRel; + + /* Look up the description for index */ + indexRel = RelationIdGetRelation(indexoid); + + if (!RelationIsValid(indexRel)) + elog(ERROR, "could not open relation with OID %u", indexoid); + + if (indexRel->rd_index->indisunique) + { + int i; + + /* Add referenced attributes to idindexattrs */ + for (i = 0; i < indexRel->rd_index->indnatts; i++) + { + int attrnum = indexRel->rd_index->indkey.values[i]; + + /* + * We don't include non-key columns into idindexattrs + * bitmaps. See RelationGetIndexAttrBitmap. + */ + if (attrnum != 0) + { + if (i < indexRel->rd_index->indnkeyatts && + !bms_is_member(attrnum - FirstLowInvalidHeapAttributeNumber, attunique)) + attunique = bms_add_member(attunique, + attrnum - FirstLowInvalidHeapAttributeNumber); + } + } + } + RelationClose(indexRel); + } + list_free(indexoidlist); + } + /* send number of live attributes */ for (i = 0; i < desc->natts; i++) { @@ -974,7 +1019,11 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns) if (replidentfull || bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber, idattrs)) - flags |= LOGICALREP_IS_REPLICA_IDENTITY; + flags |= ATTR_IS_REPLICA_IDENTITY; + + if (bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber, + attunique)) + flags |= ATTR_IS_UNIQUE; pq_sendbyte(out, flags); @@ -989,6 +1038,7 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns) } bms_free(idattrs); + bms_free(attunique); } /* @@ -1001,7 +1051,8 @@ logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel) int natts; char **attnames; Oid *atttyps; - Bitmapset *attkeys = NULL; + Bitmapset *attkeys = NULL, + *attunique = NULL; natts = pq_getmsgint(in, 2); attnames = palloc(natts * sizeof(char *)); @@ -1012,11 +1063,14 @@ logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel) { uint8 flags; - /* Check for replica identity column */ + /* Check for replica identity and unique column */ flags = pq_getmsgbyte(in); - if (flags & LOGICALREP_IS_REPLICA_IDENTITY) + if (flags & ATTR_IS_REPLICA_IDENTITY) attkeys = bms_add_member(attkeys, i); + if (flags & ATTR_IS_UNIQUE) + attunique = bms_add_member(attunique, i); + /* attribute name */ attnames[i] = pstrdup(pq_getmsgstring(in)); @@ -1030,6 +1084,7 @@ logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel) rel->attnames = attnames; rel->atttyps = atttyps; rel->attkeys = attkeys; + rel->attunique = attunique; rel->natts = natts; } diff --git a/src/backend/replication/logical/relation.c b/src/backend/replication/logical/relation.c index e989047681..48511b3610 100644 --- a/src/backend/replication/logical/relation.c +++ b/src/backend/replication/logical/relation.c @@ -19,12 +19,19 @@ #include "access/table.h" #include "catalog/namespace.h" +#include "catalog/pg_proc.h" #include "catalog/pg_subscription_rel.h" +#include "commands/trigger.h" #include "executor/executor.h" #include "nodes/makefuncs.h" +#include "optimizer/optimizer.h" #include "replication/logicalrelation.h" #include "replication/worker_internal.h" +#include "rewrite/rewriteHandler.h" #include "utils/inval.h" +#include "utils/lsyscache.h" +#include "utils/syscache.h" +#include "utils/typcache.h" static MemoryContext LogicalRepRelMapContext = NULL; @@ -91,6 +98,26 @@ logicalrep_relmap_invalidate_cb(Datum arg, Oid reloid) } } +/* + * Relcache invalidation callback to reset parallel flag. + */ +static void +logicalrep_relmap_reset_parallel_cb(Datum arg, int cacheid, uint32 hashvalue) +{ + HASH_SEQ_STATUS hash_seq; + LogicalRepRelMapEntry *entry; + + if (LogicalRepRelMap == NULL) + return; + + hash_seq_init(&hash_seq, LogicalRepRelMap); + while ((entry = hash_seq_search(&hash_seq)) != NULL) + { + entry->parallel = PARALLEL_APPLY_UNKNOWN; + entry->localrelvalid = false; + } +} + /* * Initialize the relation map cache. */ @@ -116,6 +143,9 @@ logicalrep_relmap_init(void) /* Watch for invalidation events. */ CacheRegisterRelcacheCallback(logicalrep_relmap_invalidate_cb, (Datum) 0); + CacheRegisterSyscacheCallback(PROCOID, + logicalrep_relmap_reset_parallel_cb, + (Datum) 0); } /* @@ -142,6 +172,7 @@ logicalrep_relmap_free_entry(LogicalRepRelMapEntry *entry) pfree(remoterel->atttyps); } bms_free(remoterel->attkeys); + bms_free(remoterel->attunique); if (entry->attrmap) free_attrmap(entry->attrmap); @@ -190,6 +221,7 @@ logicalrep_relmap_update(LogicalRepRelation *remoterel) } entry->remoterel.replident = remoterel->replident; entry->remoterel.attkeys = bms_copy(remoterel->attkeys); + entry->remoterel.attunique = bms_copy(remoterel->attunique); MemoryContextSwitchTo(oldctx); } @@ -310,6 +342,168 @@ logicalrep_rel_mark_updatable(LogicalRepRelMapEntry *entry) } } +/* + * Check if changes on one relation can be applied by an apply background + * worker and assign the 'parallel' flag. + * + * There are two requirements for applying changes in an apply background + * worker: 1) The unique column in the relation on the subscriber-side should + * also be the unique column on the publisher-side; 2) There cannot be any + * non-immutable functions in the subscriber-side. + * + * We just mark the relation entry as 'PARALLEL_APPLY_UNSAFE' here if changes + * on one relation can not be applied by an apply background worker and leave + * it to apply_bgworker_relation_check() to throw the actual error if needed. + */ +static void +logicalrep_rel_mark_parallel(LogicalRepRelMapEntry *entry) +{ + Bitmapset *ukey; + int i; + TupleDesc tupdesc; + int attnum; + List *fkeys = NIL; + + /* Fast path if we marked 'parallel' flag. */ + if (entry->parallel != PARALLEL_APPLY_UNKNOWN) + return; + + /* Initialize the flag. */ + entry->parallel = PARALLEL_APPLY_SAFE; + + /* + * First, we check if the unique column in the relation on the + * subscriber-side is also the unique column on the publisher-side. + */ + ukey = RelationGetIndexAttrBitmap(entry->localrel, + INDEX_ATTR_BITMAP_KEY); + + if (ukey) + { + i = -1; + while ((i = bms_next_member(ukey, i)) >= 0) + { + attnum = AttrNumberGetAttrOffset(i + FirstLowInvalidHeapAttributeNumber); + + if (entry->attrmap->attnums[attnum] < 0 || + !bms_is_member(entry->attrmap->attnums[attnum], entry->remoterel.attunique)) + { + entry->parallel = PARALLEL_APPLY_UNSAFE; + return; + } + } + + bms_free(ukey); + } + + /* + * Then, We check if there is any non-immutable function in the local + * table. Look for functions in the following places: + * a. trigger functions; + * b. Column default value expressions and domain constraints; + * c. Constraint expressions; + * d. Foreign keys. + */ + /* Check the trigger functions. */ + if (entry->localrel->trigdesc != NULL) + { + for (i = 0; i < entry->localrel->trigdesc->numtriggers; i++) + { + Trigger *trig = entry->localrel->trigdesc->triggers + i; + + if (trig->tgenabled != TRIGGER_FIRES_ALWAYS && + trig->tgenabled != TRIGGER_FIRES_ON_REPLICA) + continue; + + if (func_volatile(trig->tgfoid) != PROVOLATILE_IMMUTABLE) + { + entry->parallel = PARALLEL_APPLY_UNSAFE; + return; + } + } + } + + /* Check the columns. */ + tupdesc = RelationGetDescr(entry->localrel); + for (attnum = 0; attnum < tupdesc->natts; attnum++) + { + Form_pg_attribute att = TupleDescAttr(tupdesc, attnum); + + /* We don't need info for dropped or generated attributes */ + if (att->attisdropped || att->attgenerated) + continue; + + /* + * We don't need to check columns that only exist on the + * subscriber + */ + if (entry->attrmap->attnums[attnum] < 0) + continue; + + if (att->atthasdef) + { + Node *defaultexpr; + + defaultexpr = build_column_default(entry->localrel, attnum + 1); + if (contain_mutable_functions(defaultexpr)) + { + entry->parallel = PARALLEL_APPLY_UNSAFE; + return; + } + } + + /* + * If the column is of a DOMAIN type, determine whether + * that domain has any CHECK expressions that are not + * immutable. + */ + if (get_typtype(att->atttypid) == TYPTYPE_DOMAIN) + { + List *domain_constraints; + ListCell *lc; + + domain_constraints = GetDomainConstraints(att->atttypid); + + foreach(lc, domain_constraints) + { + DomainConstraintState *con = (DomainConstraintState *) lfirst(lc); + + if (con->check_expr && contain_mutable_functions((Node *) con->check_expr)) + { + entry->parallel = PARALLEL_APPLY_UNSAFE; + return; + } + } + } + } + + /* Check the constraints. */ + if (tupdesc->constr) + { + ConstrCheck *check = tupdesc->constr->check; + + /* + * Determine if there are any CHECK constraints which + * contains non-immutable function. + */ + for (i = 0; i < tupdesc->constr->num_check; i++) + { + Expr *check_expr = stringToNode(check[i].ccbin); + + if (contain_mutable_functions((Node *) check_expr)) + { + entry->parallel = PARALLEL_APPLY_UNSAFE; + return; + } + } + } + + /* Check the foreign keys. */ + fkeys = RelationGetFKeyList(entry->localrel); + if (fkeys) + entry->parallel = PARALLEL_APPLY_UNSAFE; +} + /* * Open the local relation associated with the remote one. * @@ -438,6 +632,9 @@ logicalrep_rel_open(LogicalRepRelId remoteid, LOCKMODE lockmode) */ logicalrep_rel_mark_updatable(entry); + /* Set if changes could be applied in the apply background worker. */ + logicalrep_rel_mark_parallel(entry); + entry->localrelvalid = true; } @@ -653,6 +850,7 @@ logicalrep_partition_open(LogicalRepRelMapEntry *root, } entry->remoterel.replident = remoterel->replident; entry->remoterel.attkeys = bms_copy(remoterel->attkeys); + entry->remoterel.attunique = bms_copy(remoterel->attunique); } entry->localrel = partrel; @@ -696,6 +894,9 @@ logicalrep_partition_open(LogicalRepRelMapEntry *root, /* Set if the table's replica identity is enough to apply update/delete. */ logicalrep_rel_mark_updatable(entry); + /* Set if changes could be applied in the apply background worker. */ + logicalrep_rel_mark_parallel(entry); + entry->localrelvalid = true; /* state and statelsn are left set to 0. */ diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c index 8ffba7e2e5..3cdbf8b457 100644 --- a/src/backend/replication/logical/tablesync.c +++ b/src/backend/replication/logical/tablesync.c @@ -884,6 +884,7 @@ fetch_remote_table_info(char *nspname, char *relname, lrel->attnames = palloc0(MaxTupleAttributeNumber * sizeof(char *)); lrel->atttyps = palloc0(MaxTupleAttributeNumber * sizeof(Oid)); lrel->attkeys = NULL; + lrel->attunique = NULL; /* * Store the columns as a list of names. Ignore those that are not diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c index bbd0304a8d..a46eb7dfab 100644 --- a/src/backend/replication/logical/worker.c +++ b/src/backend/replication/logical/worker.c @@ -1388,6 +1388,14 @@ apply_handle_stream_stop(StringInfo s) { char action = LOGICAL_REP_MSG_STREAM_STOP; + /* + * Unlike stream_commit, we don't need to wait here for stream_stop to + * finish. Allowing the other transaction to be applied before + * stream_stop is finished can lead to failures if the unique + * index/constraint is different between publisher and subscriber. But + * for such cases, we don't allow streamed transactions to be applied + * in parallel. See apply_bgworker_relation_check. + */ apply_bgworker_send_data(stream_apply_worker, 1, &action); elog(DEBUG1, "stopped streaming of xid %u, %u changes streamed", stream_xid, nchanges); @@ -2040,6 +2048,8 @@ apply_handle_insert(StringInfo s) /* Set relation for error callback */ apply_error_callback_arg.rel = rel; + apply_bgworker_relation_check(rel); + /* Initialize the executor state. */ edata = create_edata_for_relation(rel); estate = edata->estate; @@ -2183,6 +2193,8 @@ apply_handle_update(StringInfo s) /* Check if we can do the update. */ check_relation_updatable(rel); + apply_bgworker_relation_check(rel); + /* Initialize the executor state. */ edata = create_edata_for_relation(rel); estate = edata->estate; @@ -2351,6 +2363,8 @@ apply_handle_delete(StringInfo s) /* Check if we can do the delete. */ check_relation_updatable(rel); + apply_bgworker_relation_check(rel); + /* Initialize the executor state. */ edata = create_edata_for_relation(rel); estate = edata->estate; @@ -2536,13 +2550,14 @@ apply_handle_tuple_routing(ApplyExecutionData *edata, } MemoryContextSwitchTo(oldctx); + part_entry = logicalrep_partition_open(relmapentry, partrel, + attrmap); + /* Check if we can do the update or delete on the leaf partition. */ if (operation == CMD_UPDATE || operation == CMD_DELETE) - { - part_entry = logicalrep_partition_open(relmapentry, partrel, - attrmap); check_relation_updatable(part_entry); - } + + apply_bgworker_relation_check(part_entry); switch (operation) { diff --git a/src/backend/utils/cache/typcache.c b/src/backend/utils/cache/typcache.c index 808f9ebd0d..b248899d82 100644 --- a/src/backend/utils/cache/typcache.c +++ b/src/backend/utils/cache/typcache.c @@ -2540,6 +2540,23 @@ compare_values_of_enum(TypeCacheEntry *tcache, Oid arg1, Oid arg2) return 0; } +/* + * GetDomainConstraints --- get DomainConstraintState list of specified domain type + */ +List * +GetDomainConstraints(Oid type_id) +{ + TypeCacheEntry *typentry; + List *constraints = NIL; + + typentry = lookup_type_cache(type_id, TYPECACHE_DOMAIN_CONSTR_INFO); + + if(typentry->domainData != NULL) + constraints = typentry->domainData->constraints; + + return constraints; +} + /* * Load (or re-load) the enumData member of the typcache entry. */ diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h index 0f74a3392b..130c4eb82f 100644 --- a/src/include/replication/logicalproto.h +++ b/src/include/replication/logicalproto.h @@ -113,6 +113,7 @@ typedef struct LogicalRepRelation char replident; /* replica identity */ char relkind; /* remote relation kind */ Bitmapset *attkeys; /* Bitmap of key columns */ + Bitmapset *attunique; /* Bitmap of unique columns */ } LogicalRepRelation; /* Type mapping info */ diff --git a/src/include/replication/logicalrelation.h b/src/include/replication/logicalrelation.h index 78cd7e77f5..7a1e9f762a 100644 --- a/src/include/replication/logicalrelation.h +++ b/src/include/replication/logicalrelation.h @@ -15,6 +15,19 @@ #include "access/attmap.h" #include "replication/logicalproto.h" +/* + * States to determine if changes on one relation can be applied by an apply + * background worker. + */ +typedef enum RelParallel +{ + PARALLEL_APPLY_UNKNOWN = 0, /* unknown */ + PARALLEL_APPLY_SAFE, /* Can apply changes in an apply background + worker */ + PARALLEL_APPLY_UNSAFE /* Can not apply changes in an apply background + worker */ +} RelParallel; + typedef struct LogicalRepRelMapEntry { LogicalRepRelation remoterel; /* key is remoterel.remoteid */ @@ -31,6 +44,8 @@ typedef struct LogicalRepRelMapEntry Relation localrel; /* relcache entry (NULL when closed) */ AttrMap *attrmap; /* map of local attributes to remote ones */ bool updatable; /* Can apply updates/deletes? */ + RelParallel parallel; /* Can apply changes in an apply + background worker? */ /* Sync state. */ char state; diff --git a/src/include/replication/worker_internal.h b/src/include/replication/worker_internal.h index 5be8f5755e..b28f4ab977 100644 --- a/src/include/replication/worker_internal.h +++ b/src/include/replication/worker_internal.h @@ -194,6 +194,7 @@ extern void apply_bgworker_free(ApplyBgworkerState *wstate); extern void apply_bgworker_check_status(void); extern void apply_bgworker_set_status(ApplyBgworkerStatus status); extern void apply_bgworker_subxact_info_add(TransactionId current_xid); +extern void apply_bgworker_relation_check(LogicalRepRelMapEntry *rel); static inline bool am_tablesync_worker(void) diff --git a/src/include/utils/typcache.h b/src/include/utils/typcache.h index 431ad7f1b3..ed7c2e7f48 100644 --- a/src/include/utils/typcache.h +++ b/src/include/utils/typcache.h @@ -199,6 +199,8 @@ extern uint64 assign_record_type_identifier(Oid type_id, int32 typmod); extern int compare_values_of_enum(TypeCacheEntry *tcache, Oid arg1, Oid arg2); +extern List *GetDomainConstraints(Oid type_id); + extern size_t SharedRecordTypmodRegistryEstimate(void); extern void SharedRecordTypmodRegistryInit(SharedRecordTypmodRegistry *, diff --git a/src/test/subscription/t/022_twophase_cascade.pl b/src/test/subscription/t/022_twophase_cascade.pl index 0a4152d3be..30a01f7305 100644 --- a/src/test/subscription/t/022_twophase_cascade.pl +++ b/src/test/subscription/t/022_twophase_cascade.pl @@ -39,6 +39,12 @@ sub test_streaming ALTER SUBSCRIPTION tap_sub_C SET (streaming = $streaming_mode)"); + if ($streaming_mode eq 'parallel') + { + $node_C->safe_psql( + 'postgres', "ALTER TABLE test_tab ALTER c DROP DEFAULT"); + } + # Wait for subscribers to finish initialization $node_A->poll_query_until( diff --git a/src/test/subscription/t/032_streaming_apply.pl b/src/test/subscription/t/032_streaming_apply.pl new file mode 100644 index 0000000000..eca4328676 --- /dev/null +++ b/src/test/subscription/t/032_streaming_apply.pl @@ -0,0 +1,391 @@ +# Copyright (c) 2022, PostgreSQL Global Development Group + +# Test the restrictions of streaming mode "parallel" in logical replication + +use strict; +use warnings; +use PostgreSQL::Test::Cluster; +use PostgreSQL::Test::Utils; +use Test::More; + +my $offset = 0; + +# Create publisher node +my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); +$node_publisher->init(allows_streaming => 'logical'); +$node_publisher->append_conf('postgresql.conf', + 'logical_decoding_work_mem = 64kB'); +$node_publisher->start; + +# Create subscriber node +my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber'); +$node_subscriber->init; +$node_subscriber->start; + +# Setup structure on publisher +$node_publisher->safe_psql('postgres', + "CREATE TABLE test_tab (a int primary key, b varchar)"); +$node_publisher->safe_psql('postgres', + "CREATE TABLE test_tab_partitioned (a int primary key, b varchar)"); + +# Setup structure on subscriber +# We need to test normal table and partition table. +$node_subscriber->safe_psql('postgres', + "CREATE TABLE test_tab (a int primary key, b varchar)"); +$node_subscriber->safe_psql('postgres', + "CREATE TABLE test_tab_partitioned (a int primary key, b varchar) PARTITION BY RANGE(a)"); +$node_subscriber->safe_psql('postgres', + "CREATE TABLE test_tab_partition (LIKE test_tab_partitioned)"); +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partitioned ATTACH PARTITION test_tab_partition DEFAULT" +); + +# Setup logical replication +my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres'; +$node_publisher->safe_psql('postgres', + "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); +$node_publisher->safe_psql('postgres', + "CREATE PUBLICATION tap_pub_partitioned FOR TABLE test_tab_partitioned"); + +my $appname = 'tap_sub'; +$node_subscriber->safe_psql( + 'postgres', " + CREATE SUBSCRIPTION tap_sub + CONNECTION '$publisher_connstr application_name=$appname' + PUBLICATION tap_pub, tap_pub_partitioned + WITH (streaming = parallel, copy_data = false)"); + +$node_publisher->wait_for_catchup($appname); + +# It is not allowed that the unique index on the publisher and the subscriber +# is different. Check the error reported by background worker in this case. +# First we check the unique index on normal table. +$node_subscriber->safe_psql('postgres', + "CREATE UNIQUE INDEX test_tab_b_idx ON test_tab (b)"); + +# Check that a background worker starts if "streaming" option is specified as +# "parallel". We have to look for the DEBUG1 log messages about that, so +# temporarily bump up the log verbosity. +$node_subscriber->append_conf('postgresql.conf', "log_min_messages = debug1"); +$node_subscriber->reload; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); +$node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); +$node_subscriber->reload; + +$node_subscriber->wait_for_log( + qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab" in parallel mode/, + $offset); + +# Drop the unique index on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', "DROP INDEX test_tab_b_idx"); + +$node_publisher->wait_for_catchup($appname); + +my $result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); +is($result, qq(5000), 'data replicated to subscriber after dropping index'); + +# Then we check the unique index on partition table. +$node_subscriber->safe_psql('postgres', + "CREATE UNIQUE INDEX test_tab_b_partition_idx ON test_tab_partition (b)"); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab_partitioned SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log( + qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab_partitioned" in parallel mode/, + $offset); + +# Drop the unique index on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', "DROP INDEX test_tab_b_partition_idx"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); +is($result, qq(5000), 'data replicated to subscriber after dropping index'); + +# Triggers which execute non-immutable function are not allowed on the +# subscriber side. Check the error reported by background worker in this case. +# First we check the trigger function on normal table. +$node_subscriber->safe_psql( + 'postgres', qq{ +CREATE FUNCTION trigger_func() RETURNS TRIGGER AS \$\$ + BEGIN + RETURN NULL; + END +\$\$ language plpgsql; +CREATE TRIGGER insert_trig +BEFORE INSERT ON test_tab +FOR EACH ROW EXECUTE PROCEDURE trigger_func(); +ALTER TABLE test_tab ENABLE REPLICA TRIGGER insert_trig; +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', "DELETE FROM test_tab"); + +$node_subscriber->wait_for_log( + qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab" in parallel mode/, + $offset); + +# Drop the trigger on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "DROP TRIGGER insert_trig ON test_tab"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); +is($result, qq(0), 'data replicated to subscriber after dropping trigger'); + +# Then we check the trigger function on partition table. +$node_subscriber->safe_psql( + 'postgres', qq{ +CREATE TRIGGER insert_trig +BEFORE INSERT ON test_tab_partition +FOR EACH ROW EXECUTE PROCEDURE trigger_func(); +ALTER TABLE test_tab_partition ENABLE REPLICA TRIGGER insert_trig; +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', "DELETE FROM test_tab_partitioned"); + +$node_subscriber->wait_for_log( + qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab_partitioned" in parallel mode/, + $offset); + +# Drop the trigger on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "DROP TRIGGER insert_trig ON test_tab_partition"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); +is($result, qq(0), 'data replicated to subscriber after dropping trigger'); + +# It is not allowed that column default value expression contains a +# non-immutable function on the subscriber side. Check the error reported by +# background worker in this case. +# First we check the column default value expression on normal table. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab ALTER COLUMN b SET DEFAULT random()"); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log( + qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab" in parallel mode/, + $offset); + +# Drop default value on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab ALTER COLUMN b DROP DEFAULT"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); +is($result, qq(5000), + 'data replicated to subscriber after dropping default value expression'); + +# Then we check the column default value expression on partition table. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partition ALTER COLUMN b SET DEFAULT random()"); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab_partitioned SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log( + qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab_partitioned" in parallel mode/, + $offset); + +# Drop default value on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partition ALTER COLUMN b DROP DEFAULT"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); +is($result, qq(5000), + 'data replicated to subscriber after dropping default value expression'); + +# It is not allowed that domain constraint expression contains a non-immutable +# function on the subscriber side. Check the error reported by background +# worker in this case. +# Because the column type of the partition table must be the same as its parent +# table, only test normal table here. +$node_subscriber->safe_psql( + 'postgres', qq{ +CREATE DOMAIN test_domain AS int CHECK(VALUE > random()); +ALTER TABLE test_tab ALTER COLUMN a TYPE test_domain; +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', "DELETE FROM test_tab"); + +$node_subscriber->wait_for_log( + qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab" in parallel mode/, + $offset); + +# Drop domain constraint expression on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab ALTER COLUMN a TYPE int"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); +is($result, qq(0), + 'data replicated to subscriber after dropping domain constraint expression' +); + +# It is not allowed that constraint expression contains a non-immutable function +# on the subscriber side. Check the error reported by background worker in this +# case. +# First we check the constraint expression on normal table. +$node_subscriber->safe_psql( + 'postgres', qq{ +ALTER TABLE test_tab ADD CONSTRAINT test_tab_con check (a > random()); +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log( + qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab" in parallel mode/, + $offset); + +# Drop constraint on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab DROP CONSTRAINT test_tab_con"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); +is($result, qq(5000), + 'data replicated to subscriber after dropping constraint expression'); + +# Then we check the constraint expression on partition table. +$node_subscriber->safe_psql( + 'postgres', qq{ +ALTER TABLE test_tab_partition ADD CONSTRAINT test_tab_con check (a > random()); +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', "DELETE FROM test_tab_partitioned"); + +$node_subscriber->wait_for_log( + qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab_partitioned" in parallel mode/, + $offset); + +# Drop constraint on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partition DROP CONSTRAINT test_tab_con"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); +is($result, qq(0), + 'data replicated to subscriber after dropping constraint expression'); + +# It is not allowed that foreign key on the subscriber side. Check the error +# reported by background worker in this case. +# First we check the foreign key on normal table. +$node_publisher->safe_psql('postgres', "DELETE FROM test_tab"); +$node_publisher->wait_for_catchup($appname); +$node_subscriber->safe_psql( + 'postgres', qq{ +CREATE TABLE test_tab_f (a int primary key); +ALTER TABLE test_tab ADD CONSTRAINT test_tabfk FOREIGN KEY(a) REFERENCES test_tab_f(a); +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log( + qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab" in parallel mode/, + $offset); + +# Drop the foreign key constraint on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab DROP CONSTRAINT test_tabfk"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); +is($result, qq(5000), + 'data replicated to subscriber after dropping the foreign key'); + +# Then we check the foreign key on partition table. +$node_publisher->wait_for_catchup($appname); +$node_subscriber->safe_psql( + 'postgres', qq{ +CREATE TABLE test_tab_partition_f (a int primary key); +ALTER TABLE test_tab_partition ADD CONSTRAINT test_tab_patition_fk FOREIGN KEY(a) REFERENCES test_tab_partition_f(a); +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab_partitioned SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log( + qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab_partitioned" in parallel mode/, + $offset); + +# Drop the foreign key constraint on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partition DROP CONSTRAINT test_tab_patition_fk"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); +is($result, qq(5000), + 'data replicated to subscriber after dropping the foreign key'); + +$node_subscriber->stop; +$node_publisher->stop; + +done_testing(); diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 4137dc77b4..ae7cd99159 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -2271,6 +2271,7 @@ RelMapFile RelMapping RelOptInfo RelOptKind +RelParallel RelToCheck RelToCluster RelabelType -- 2.23.0.windows.1 [application/octet-stream] v17-0004-Retry-to-apply-streaming-xact-only-in-apply-work.patch (28.9K, ../../OSZPR01MB6278023241CEDDE233C8386F9E899@OSZPR01MB6278.jpnprd01.prod.outlook.com/5-v17-0004-Retry-to-apply-streaming-xact-only-in-apply-work.patch) download | inline diff: From 5d1fd26d3359fe3379e39382e23b5b1f42f32b5b Mon Sep 17 00:00:00 2001 From: wangw <[email protected]> Date: Wed, 15 Jun 2022 11:02:08 +0800 Subject: [PATCH v17 4/4] Retry to apply streaming xact only in apply worker If the user sets the subscription_parameter "streaming" to "parallel", when applying a streaming transaction, we will try to apply this transaction in apply background worker. However, when the changes in this transaction cannot be applied in apply background worker, the background worker will exit with an error. In this case, we can retry applying this streaming transaction in "on" mode. In this way, we may avoid blocking logical replication here. So we introduce field "subretry" in catalog "pg_subscription". When the subscriber exit with an error, we will try to set this flag to true, and when the transaction is applied successfully, we will try to set this flag to false. Then when we try to apply a streaming transaction in apply background worker, we can see if this transaction has failed before based on the "subretry" field. --- doc/src/sgml/catalogs.sgml | 9 + doc/src/sgml/ref/create_subscription.sgml | 5 + src/backend/catalog/pg_subscription.c | 1 + src/backend/catalog/system_views.sql | 4 +- src/backend/commands/subscriptioncmds.c | 1 + .../replication/logical/applybgworker.c | 15 +- src/backend/replication/logical/worker.c | 165 +++++++++++++----- src/bin/pg_dump/pg_dump.c | 5 +- src/include/catalog/pg_subscription.h | 4 + .../subscription/t/032_streaming_apply.pl | 154 +++++++++------- 10 files changed, 254 insertions(+), 109 deletions(-) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index 815cae6082..28f3d121d9 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -7907,6 +7907,15 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>subretry</structfield> <type>bool</type> + </para> + <para> + True if the previous apply change failed and a retry was required. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>subconninfo</structfield> <type>text</type> diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml index 270e3d382e..bd5361991e 100644 --- a/doc/src/sgml/ref/create_subscription.sgml +++ b/doc/src/sgml/ref/create_subscription.sgml @@ -244,6 +244,11 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl relation on the subscriber-side should also be the unique column on the publisher-side; 2) there cannot be any non-immutable functions in the subscriber-side replicated table. + When applying a streaming transaction, if either requirement is not + met, the background worker will exit with an error. + <literal>parallel</literal> mode is disregarded when retrying; + instead the transaction will be applied using <literal>on</literal> + mode. </para> </listitem> </varlistentry> diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c index 8856ce3b50..9b7f09653d 100644 --- a/src/backend/catalog/pg_subscription.c +++ b/src/backend/catalog/pg_subscription.c @@ -71,6 +71,7 @@ GetSubscription(Oid subid, bool missing_ok) sub->stream = subform->substream; sub->twophasestate = subform->subtwophasestate; sub->disableonerr = subform->subdisableonerr; + sub->retry = subform->subretry; /* Get conninfo */ datum = SysCacheGetAttr(SUBSCRIPTIONOID, diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql index fedaed533b..10f4dd6785 100644 --- a/src/backend/catalog/system_views.sql +++ b/src/backend/catalog/system_views.sql @@ -1298,8 +1298,8 @@ REVOKE ALL ON pg_replication_origin_status FROM public; -- All columns of pg_subscription except subconninfo are publicly readable. REVOKE ALL ON pg_subscription FROM public; GRANT SELECT (oid, subdbid, subskiplsn, subname, subowner, subenabled, - subbinary, substream, subtwophasestate, subdisableonerr, subslotname, - subsynccommit, subpublications) + subbinary, substream, subtwophasestate, subdisableonerr, + subretry, subslotname, subsynccommit, subpublications) ON pg_subscription TO public; CREATE VIEW pg_stat_subscription_stats AS diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c index d4b2616ee6..612a83393a 100644 --- a/src/backend/commands/subscriptioncmds.c +++ b/src/backend/commands/subscriptioncmds.c @@ -662,6 +662,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt, LOGICALREP_TWOPHASE_STATE_PENDING : LOGICALREP_TWOPHASE_STATE_DISABLED); values[Anum_pg_subscription_subdisableonerr - 1] = BoolGetDatum(opts.disableonerr); + values[Anum_pg_subscription_subretry - 1] = BoolGetDatum(false); values[Anum_pg_subscription_subconninfo - 1] = CStringGetTextDatum(conninfo); if (opts.slot_name) diff --git a/src/backend/replication/logical/applybgworker.c b/src/backend/replication/logical/applybgworker.c index 396dc39605..7711b58be2 100644 --- a/src/backend/replication/logical/applybgworker.c +++ b/src/backend/replication/logical/applybgworker.c @@ -106,6 +106,18 @@ apply_bgworker_can_start(TransactionId xid) if (!XLogRecPtrIsInvalid(MySubscription->skiplsn)) return false; + /* + * Don't use apply background workers for retries, because it is possible + * that the last time we tried to apply a transaction using an apply + * background worker the checks failed (see function + * apply_bgworker_relation_check). + */ + if (MySubscription->retry) + { + elog(DEBUG1, "apply background workers are not used for retries"); + return false; + } + /* * For streaming transactions that are being applied in apply background * worker, we cannot decide whether to apply the change for a relation @@ -823,6 +835,5 @@ apply_bgworker_relation_check(LogicalRepRelMapEntry *rel) "mode", rel->remoterel.nspname, rel->remoterel.relname), errdetail("The unique column on subscriber is not the unique " "column on publisher or there is at least one " - "non-immutable function."), - errhint("Please change the streaming option to 'on' instead of 'parallel'."))); + "non-immutable function."))); } diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c index a46eb7dfab..25b31c277b 100644 --- a/src/backend/replication/logical/worker.c +++ b/src/backend/replication/logical/worker.c @@ -378,6 +378,8 @@ static void clear_subscription_skip_lsn(XLogRecPtr finish_lsn); static inline void set_apply_error_context_xact(TransactionId xid, XLogRecPtr lsn); static inline void reset_apply_error_context_info(void); +static void set_subscription_retry(bool retry); + /* * Should this worker apply changes for given relation. * @@ -904,6 +906,9 @@ apply_handle_commit(StringInfo s) apply_handle_commit_internal(&commit_data); + /* Reset the retry flag. */ + set_subscription_retry(false); + /* Check the status of apply background worker if any. */ apply_bgworker_check_status(); @@ -1015,6 +1020,9 @@ apply_handle_prepare(StringInfo s) in_remote_transaction = false; + /* Reset the retry flag. */ + set_subscription_retry(false); + /* Check the status of apply background worker if any. */ apply_bgworker_check_status(); @@ -1068,6 +1076,9 @@ apply_handle_commit_prepared(StringInfo s) store_flush_position(prepare_data.end_lsn); in_remote_transaction = false; + /* Reset the retry flag. */ + set_subscription_retry(false); + /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(prepare_data.end_lsn); @@ -1123,6 +1134,9 @@ apply_handle_rollback_prepared(StringInfo s) store_flush_position(rollback_data.rollback_end_lsn); in_remote_transaction = false; + /* Reset the retry flag. */ + set_subscription_retry(false); + /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(rollback_data.rollback_end_lsn); @@ -1215,6 +1229,9 @@ apply_handle_stream_prepare(StringInfo s) /* Unlink the files with serialized changes and subxact info. */ stream_cleanup_files(MyLogicalRepWorker->subid, prepare_data.xid); } + + /* Reset the retry flag. */ + set_subscription_retry(false); } in_remote_transaction = false; @@ -1642,6 +1659,9 @@ apply_handle_stream_abort(StringInfo s) */ serialize_stream_abort(xid, subxid); } + + /* Reset the retry flag. */ + set_subscription_retry(false); } reset_apply_error_context_info(); @@ -1854,6 +1874,9 @@ apply_handle_stream_commit(StringInfo s) /* Unlink the files with serialized changes and subxact info. */ stream_cleanup_files(MyLogicalRepWorker->subid, xid); } + + /* Reset the retry flag. */ + set_subscription_retry(false); } /* Check the status of apply background worker if any. */ @@ -3897,20 +3920,28 @@ start_table_sync(XLogRecPtr *origin_startpos, char **myslotname) } PG_CATCH(); { + /* + * Emit the error message, and recover from the error state to an idle + * state + */ + HOLD_INTERRUPTS(); + + EmitErrorReport(); + AbortOutOfAnyTransaction(); + FlushErrorState(); + + RESUME_INTERRUPTS(); + + /* Report the worker failed during table synchronization */ + pgstat_report_subscription_error(MySubscription->oid, false); + + /* Set the retry flag. */ + set_subscription_retry(true); + if (MySubscription->disableonerr) DisableSubscriptionAndExit(); - else - { - /* - * Report the worker failed during table synchronization. Abort - * the current transaction so that the stats message is sent in an - * idle state. - */ - AbortOutOfAnyTransaction(); - pgstat_report_subscription_error(MySubscription->oid, false); - PG_RE_THROW(); - } + proc_exit(0); } PG_END_TRY(); @@ -3935,20 +3966,27 @@ start_apply(XLogRecPtr origin_startpos) } PG_CATCH(); { + /* + * Emit the error message, and recover from the error state to an idle + * state + */ + HOLD_INTERRUPTS(); + + EmitErrorReport(); + AbortOutOfAnyTransaction(); + FlushErrorState(); + + RESUME_INTERRUPTS(); + + /* Report the worker failed while applying changes */ + pgstat_report_subscription_error(MySubscription->oid, + !am_tablesync_worker()); + + /* Set the retry flag. */ + set_subscription_retry(true); + if (MySubscription->disableonerr) DisableSubscriptionAndExit(); - else - { - /* - * Report the worker failed while applying changes. Abort the - * current transaction so that the stats message is sent in an - * idle state. - */ - AbortOutOfAnyTransaction(); - pgstat_report_subscription_error(MySubscription->oid, !am_tablesync_worker()); - - PG_RE_THROW(); - } } PG_END_TRY(); } @@ -4194,28 +4232,11 @@ ApplyWorkerMain(Datum main_arg) } /* - * After error recovery, disable the subscription in a new transaction - * and exit cleanly. + * Disable the subscription in a new transaction. */ static void DisableSubscriptionAndExit(void) { - /* - * Emit the error message, and recover from the error state to an idle - * state - */ - HOLD_INTERRUPTS(); - - EmitErrorReport(); - AbortOutOfAnyTransaction(); - FlushErrorState(); - - RESUME_INTERRUPTS(); - - /* Report the worker failed during either table synchronization or apply */ - pgstat_report_subscription_error(MyLogicalRepWorker->subid, - !am_tablesync_worker()); - /* Disable the subscription */ StartTransactionCommand(); DisableSubscription(MySubscription->oid); @@ -4225,8 +4246,6 @@ DisableSubscriptionAndExit(void) ereport(LOG, errmsg("logical replication subscription \"%s\" has been disabled due to an error", MySubscription->name)); - - proc_exit(0); } /* @@ -4461,3 +4480,63 @@ reset_apply_error_context_info(void) apply_error_callback_arg.remote_attnum = -1; set_apply_error_context_xact(InvalidTransactionId, InvalidXLogRecPtr); } + +/* + * Set subretry of pg_subscription catalog. + * + * If retry is true, subscriber is about to exit with an error. Otherwise, it + * means that the transaction was applied successfully. + */ +static void +set_subscription_retry(bool retry) +{ + Relation rel; + HeapTuple tup; + bool started_tx = false; + bool nulls[Natts_pg_subscription]; + bool replaces[Natts_pg_subscription]; + Datum values[Natts_pg_subscription]; + + if (MySubscription->retry == retry || + am_apply_bgworker()) + return; + + if (!IsTransactionState()) + { + StartTransactionCommand(); + started_tx = true; + } + + /* Look up the subscription in the catalog */ + rel = table_open(SubscriptionRelationId, RowExclusiveLock); + tup = SearchSysCacheCopy1(SUBSCRIPTIONOID, + ObjectIdGetDatum(MySubscription->oid)); + + if (!HeapTupleIsValid(tup)) + elog(ERROR, "subscription \"%s\" does not exist", MySubscription->name); + + LockSharedObject(SubscriptionRelationId, MySubscription->oid, 0, + AccessShareLock); + + /* Form a new tuple. */ + memset(values, 0, sizeof(values)); + memset(nulls, false, sizeof(nulls)); + memset(replaces, false, sizeof(replaces)); + + /* reset subretry */ + values[Anum_pg_subscription_subretry - 1] = BoolGetDatum(retry); + replaces[Anum_pg_subscription_subretry - 1] = true; + + tup = heap_modify_tuple(tup, RelationGetDescr(rel), values, nulls, + replaces); + + /* Update the catalog. */ + CatalogTupleUpdate(rel, &tup->t_self, tup); + + /* Cleanup. */ + heap_freetuple(tup); + table_close(rel, NoLock); + + if (started_tx) + CommitTransactionCommand(); +} diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 9099fe5f74..f6e257d06d 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -4478,8 +4478,9 @@ getSubscriptions(Archive *fout) ntups = PQntuples(res); /* - * Get subscription fields. We don't include subskiplsn in the dump as - * after restoring the dump this value may no longer be relevant. + * Get subscription fields. We don't include subskiplsn and subretry in + * the dump as after restoring the dump this value may no longer be + * relevant. */ i_tableoid = PQfnumber(res, "tableoid"); i_oid = PQfnumber(res, "oid"); diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h index d54540f5f5..5f4e058ec1 100644 --- a/src/include/catalog/pg_subscription.h +++ b/src/include/catalog/pg_subscription.h @@ -76,6 +76,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW bool subdisableonerr; /* True if a worker error should cause the * subscription to be disabled */ + bool subretry BKI_DEFAULT(f); /* True if the previous apply change failed. */ + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* Connection string to the publisher */ text subconninfo BKI_FORCE_NOT_NULL; @@ -116,6 +118,8 @@ typedef struct Subscription bool disableonerr; /* Indicates if the subscription should be * automatically disabled if a worker error * occurs */ + bool retry; /* Indicates if the previous apply change + * failed. */ char *conninfo; /* Connection string to the publisher */ char *slotname; /* Name of the replication slot */ char *synccommit; /* Synchronous commit setting for worker */ diff --git a/src/test/subscription/t/032_streaming_apply.pl b/src/test/subscription/t/032_streaming_apply.pl index eca4328676..1bcb4c65b7 100644 --- a/src/test/subscription/t/032_streaming_apply.pl +++ b/src/test/subscription/t/032_streaming_apply.pl @@ -57,8 +57,13 @@ $node_subscriber->safe_psql( $node_publisher->wait_for_catchup($appname); +# ============================================================================ # It is not allowed that the unique index on the publisher and the subscriber -# is different. Check the error reported by background worker in this case. +# is different. Check the error reported by background worker in this case. And +# after retrying in apply worker, we check if the data is replicated +# successfully. +# ============================================================================ + # First we check the unique index on normal table. $node_subscriber->safe_psql('postgres', "CREATE UNIQUE INDEX test_tab_b_idx ON test_tab (b)"); @@ -82,14 +87,15 @@ $node_subscriber->wait_for_log( qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab" in parallel mode/, $offset); -# Drop the unique index on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', "DROP INDEX test_tab_b_idx"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); my $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); -is($result, qq(5000), 'data replicated to subscriber after dropping index'); +is($result, qq(5000), 'data replicated to subscribers after retrying because of unique index'); + +# Drop the unique index on the subscriber. +$node_subscriber->safe_psql('postgres', "DROP INDEX test_tab_b_idx"); # Then we check the unique index on partition table. $node_subscriber->safe_psql('postgres', @@ -106,17 +112,20 @@ $node_subscriber->wait_for_log( qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab_partitioned" in parallel mode/, $offset); -# Drop the unique index on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', "DROP INDEX test_tab_b_partition_idx"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); -is($result, qq(5000), 'data replicated to subscriber after dropping index'); +is($result, qq(5000), 'data replicated to subscribers after retrying because of unique index'); + +# Drop the unique index on the subscriber. +$node_subscriber->safe_psql('postgres', "DROP INDEX test_tab_b_partition_idx"); # Triggers which execute non-immutable function are not allowed on the # subscriber side. Check the error reported by background worker in this case. +# And after retrying in apply worker, we check if the data is replicated +# successfully. # First we check the trigger function on normal table. $node_subscriber->safe_psql( 'postgres', qq{ @@ -140,15 +149,16 @@ $node_subscriber->wait_for_log( qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab" in parallel mode/, $offset); -# Drop the trigger on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "DROP TRIGGER insert_trig ON test_tab"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); -is($result, qq(0), 'data replicated to subscriber after dropping trigger'); +is($result, qq(0), 'data replicated to subscribers after retrying because of trigger'); + +# Drop the trigger on the subscriber. +$node_subscriber->safe_psql('postgres', + "DROP TRIGGER insert_trig ON test_tab"); # Then we check the trigger function on partition table. $node_subscriber->safe_psql( @@ -168,19 +178,24 @@ $node_subscriber->wait_for_log( qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab_partitioned" in parallel mode/, $offset); -# Drop the trigger on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "DROP TRIGGER insert_trig ON test_tab_partition"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); -is($result, qq(0), 'data replicated to subscriber after dropping trigger'); +is($result, qq(0), 'data replicated to subscribers after retrying because of trigger'); +# Drop the trigger on the subscriber. +$node_subscriber->safe_psql('postgres', + "DROP TRIGGER insert_trig ON test_tab_partition"); + +# ============================================================================ # It is not allowed that column default value expression contains a # non-immutable function on the subscriber side. Check the error reported by -# background worker in this case. +# background worker in this case. And after retrying in apply worker, we check +# if the data is replicated successfully. +# ============================================================================ + # First we check the column default value expression on normal table. $node_subscriber->safe_psql('postgres', "ALTER TABLE test_tab ALTER COLUMN b SET DEFAULT random()"); @@ -196,16 +211,17 @@ $node_subscriber->wait_for_log( qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab" in parallel mode/, $offset); -# Drop default value on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "ALTER TABLE test_tab ALTER COLUMN b DROP DEFAULT"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); is($result, qq(5000), - 'data replicated to subscriber after dropping default value expression'); + 'data replicated to subscribers after retrying because of column default value'); + +# Drop default value on the subscriber. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab ALTER COLUMN b DROP DEFAULT"); # Then we check the column default value expression on partition table. $node_subscriber->safe_psql('postgres', @@ -222,20 +238,25 @@ $node_subscriber->wait_for_log( qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab_partitioned" in parallel mode/, $offset); -# Drop default value on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "ALTER TABLE test_tab_partition ALTER COLUMN b DROP DEFAULT"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); is($result, qq(5000), - 'data replicated to subscriber after dropping default value expression'); + 'data replicated to subscribers after retrying because of column default value'); + +# Drop default value on the subscriber. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partition ALTER COLUMN b DROP DEFAULT"); +# ============================================================================ # It is not allowed that domain constraint expression contains a non-immutable # function on the subscriber side. Check the error reported by background -# worker in this case. +# worker in this case. And after retrying in apply worker, we check if the data +# is replicated successfully. +# ============================================================================ + # Because the column type of the partition table must be the same as its parent # table, only test normal table here. $node_subscriber->safe_psql( @@ -253,21 +274,26 @@ $node_subscriber->wait_for_log( qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab" in parallel mode/, $offset); -# Drop domain constraint expression on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "ALTER TABLE test_tab ALTER COLUMN a TYPE int"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); is($result, qq(0), - 'data replicated to subscriber after dropping domain constraint expression' + 'data replicated to subscribers after retrying because of domain' ); -# It is not allowed that constraint expression contains a non-immutable function -# on the subscriber side. Check the error reported by background worker in this -# case. +# Drop domain constraint expression on the subscriber. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab ALTER COLUMN a TYPE int"); + +# ============================================================================ +# It is not allowed that constraint expression contains a non-immutable +# function on the subscriber side. Check the error reported by background +# worker in this case. And after retrying in apply worker, we check if the data +# is replicated successfully. +# ============================================================================ + # First we check the constraint expression on normal table. $node_subscriber->safe_psql( 'postgres', qq{ @@ -285,16 +311,17 @@ $node_subscriber->wait_for_log( qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab" in parallel mode/, $offset); -# Drop constraint on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "ALTER TABLE test_tab DROP CONSTRAINT test_tab_con"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); is($result, qq(5000), - 'data replicated to subscriber after dropping constraint expression'); + 'data replicated to subscribers after retrying because of constraint'); + +# Drop constraint on the subscriber. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab DROP CONSTRAINT test_tab_con"); # Then we check the constraint expression on partition table. $node_subscriber->safe_psql( @@ -311,19 +338,24 @@ $node_subscriber->wait_for_log( qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab_partitioned" in parallel mode/, $offset); -# Drop constraint on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "ALTER TABLE test_tab_partition DROP CONSTRAINT test_tab_con"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); is($result, qq(0), - 'data replicated to subscriber after dropping constraint expression'); + 'data replicated to subscribers after retrying because of constraint'); +# Drop constraint on the subscriber. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partition DROP CONSTRAINT test_tab_con"); + +# ============================================================================ # It is not allowed that foreign key on the subscriber side. Check the error -# reported by background worker in this case. +# reported by background worker in this case. And after retrying in apply +# worker, we check if the data is replicated successfully. +# ============================================================================ + # First we check the foreign key on normal table. $node_publisher->safe_psql('postgres', "DELETE FROM test_tab"); $node_publisher->wait_for_catchup($appname); @@ -344,16 +376,17 @@ $node_subscriber->wait_for_log( qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab" in parallel mode/, $offset); -# Drop the foreign key constraint on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "ALTER TABLE test_tab DROP CONSTRAINT test_tabfk"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); is($result, qq(5000), - 'data replicated to subscriber after dropping the foreign key'); + 'data replicated to subscribers after retrying because of foreign key'); + +# Drop the foreign key constraint on the subscriber. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab DROP CONSTRAINT test_tabfk"); # Then we check the foreign key on partition table. $node_publisher->wait_for_catchup($appname); @@ -374,16 +407,17 @@ $node_subscriber->wait_for_log( qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab_partitioned" in parallel mode/, $offset); -# Drop the foreign key constraint on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "ALTER TABLE test_tab_partition DROP CONSTRAINT test_tab_patition_fk"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); is($result, qq(5000), - 'data replicated to subscriber after dropping the foreign key'); + 'data replicated to subscribers after retrying because of foreign key'); + +# Drop the foreign key constraint on the subscriber. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partition DROP CONSTRAINT test_tab_patition_fk"); $node_subscriber->stop; $node_publisher->stop; -- 2.23.0.windows.1 ^ permalink raw reply [nested|flat] 66+ messages in thread
* RE: Perform streaming logical transactions by background workers and parallel apply @ 2022-07-19 02:28 [email protected] <[email protected]> parent: Peter Smith <[email protected]> 0 siblings, 1 reply; 66+ messages in thread From: [email protected] @ 2022-07-19 02:28 UTC (permalink / raw) To: Peter Smith <[email protected]>; +Cc: Amit Kapila <[email protected]>; Masahiko Sawada <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; PostgreSQL Hackers <[email protected]> On Wed, Jul 13, 2022 at 13:49 PM Peter Smith <[email protected]> wrote: > Below are my review comments for the v16* patch set: Thanks for your comments. > ======== > v16-0001 > ======== > > 1.0 <general> > > There are places (comments, docs, errmsgs, etc) in the patch referring > to "parallel mode". I think every one of those references should be > found and renamed to "parallel streaming mode" or "streaming=parallel" > or at the very least match sure that "streaming" is in the same > sentence. IMO it's too vague just saying "parallel" without also > saying the context is for the "streaming" parameter. > > I have commented on some of those examples below, but please search > everything anyway (including the docs) to catch the ones I haven't > explicitly mentioned. I checked all places in the patch where the word "parallel" is used (case insensitive), and I think it is clear that the description is related to stream transactions. So I am not so sure. Could you please give me some examples? I will improve them later. > 1.2 .../replication/logical/applybgworker.c - apply_bgworker_start > > + if (list_length(ApplyWorkersFreeList) > 0) > + { > + wstate = (ApplyBgworkerState *) llast(ApplyWorkersFreeList); > + ApplyWorkersFreeList = list_delete_last(ApplyWorkersFreeList); > + Assert(wstate->pstate->status == APPLY_BGWORKER_FINISHED); > + } > > The Assert that the entries in the free-list are FINISHED seems like > unnecessary checking. IIUC, code is already doing the Assert that > entries are FINISHED before allowing them into the free-list in the > first place. Just for robustness. > 1.3 .../replication/logical/applybgworker.c - apply_bgworker_find > > + if (found) > + { > + char status = entry->wstate->pstate->status; > + > + /* If any workers (or the postmaster) have died, we have failed. */ > + if (status == APPLY_BGWORKER_EXIT) > + ereport(ERROR, > + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), > + errmsg("background worker %u failed to apply transaction %u", > + entry->wstate->pstate->n, > + entry->wstate->pstate->stream_xid))); > + > + Assert(status == APPLY_BGWORKER_BUSY); > + > + return entry->wstate; > + } > > Why not remove that Assert but change the condition to be: > > if (status != APPLY_BGWORKER_BUSY) > ereport(...) When I check "APPLY_BGWORKER_EXIT", I use the function "ereport" to report the error, because "APPLY_BGWORKER_EXIT" is a possible use case. But for "APPLY_BGWORKER_BUSY", this use case should not happen here. So I think it's fine to only check this for developers when the compile option "--enable-cassert" is specified. > ======== > v16-0003 > ======== > > 3.0 <general> > > Same comment about "parallel mode" as in comment #1.0 > > ====== Please refer to the reply to #1.0. > 3.5 src/backend/replication/logical/proto.c - logicalrep_read_attrs > > @@ -1012,11 +1062,14 @@ logicalrep_read_attrs(StringInfo in, > LogicalRepRelation *rel) > { > uint8 flags; > > - /* Check for replica identity column */ > + /* Check for replica identity and unique column */ > flags = pq_getmsgbyte(in); > - if (flags & LOGICALREP_IS_REPLICA_IDENTITY) > + if (flags & ATTR_IS_REPLICA_IDENTITY) > attkeys = bms_add_member(attkeys, i); > > + if (flags & ATTR_IS_UNIQUE) > + attunique = bms_add_member(attunique, i); > > The code comment really applies to all 3 statements so maybe better > not to have the blank line here. I think it looks a bit messy without the blank line. So I tried to improve it to the following: ``` /* Check for replica identity column */ flags = pq_getmsgbyte(in); if (flags & ATTR_IS_REPLICA_IDENTITY) attkeys = bms_add_member(attkeys, i); /* Check for unique column */ if (flags & ATTR_IS_UNIQUE) attunique = bms_add_member(attunique, i); ``` > 3.6 src/backend/replication/logical/relation.c - logicalrep_rel_mark_parallel > > 3.6.a > + /* Fast path if we marked 'parallel' flag. */ > + if (entry->parallel != PARALLEL_APPLY_UNKNOWN) > + return; > > SUGGESTED > Fast path if 'parallel' flag is already known. > > ~ > > 3.6.b > + /* Initialize the flag. */ > + entry->parallel = PARALLEL_APPLY_SAFE; > > I think it makes more sense if assigning SAFE is the very *last* thing > this function does – not the first thing. > > ~ > > 3.6.c > + /* > + * First, we check if the unique column in the relation on the > + * subscriber-side is also the unique column on the publisher-side. > + */ > > "First, we check..." -> "First, check..." > > ~ > > 3.6.d > + /* > + * Then, We check if there is any non-immutable function in the local > + * table. Look for functions in the following places: > > > "Then, We check..." -> "Then, check" =>3.6.a =>3.6.c =>3.6.d Improved as suggested. =>3.6.b Not sure about this. > 3.7 src/backend/replication/logical/relation.c - logicalrep_rel_mark_parallel > > From [3] you wrote: > Personally, I do not like to use the `goto` syntax if it is not necessary, > because the `goto` syntax will forcibly change the flow of code execution. > > Yes, but OTOH readability is a major consideration too, and in this > function by simply saying goto parallel_unsafe; you can have 3 returns > instead of 7 returns, and it will take ~10 lines less code to do the > same functionality. I am still not sure about this, I think I will change this if some more people think `goto` is better here. > 4.3 doc/src/sgml/ref/create_subscription.sgml > > + <literal>parallel</literal> mode is disregarded when retrying; > + instead the transaction will be applied using <literal>on</literal> > + mode. > > "on mode" etc sounds strange. > > SUGGESTION > During the retry the streaming=parallel mode is ignored. The retried > transaction will be applied using streaming=on mode. Since it's part of the streaming option document. I think it's fine to directly say "<literal>parallel</literal> mode" > 4.4 src/backend/replication/logical/worker.c - set_subscription_retry > > + if (MySubscription->retry == retry || > + am_apply_bgworker()) > + return; > + > > Somehow I feel that this quick exit condition is not quite what it > seems. IIUC the purpose of this is really to avoid doing the tuple > updates if it is not necessary to do them. So if retry was already set > true then there is no need to update tuple to true again. So if retry > was already set false then there is no need to update the tuple to > false. But I just don't see how the (hypothetical) code below can work > as expected, because where is the code updating the value of > MySubscription->retry ??? > > set_subscription_retry(true); > set_subscription_retry(true); > > I think at least there needs to be some detailed comments explaining > what this quick exit is really doing because my guess is that > currently it is not quite working as expected. The subscription cache is be updated in maybe_reread_subscription() and is invoked at every transaction. And we reset the retry flag at transaction end, so it should be fine. And I think the quick exit check code is similar to clear_subscription_skip_lsn. Attach the news patches. [1] - https://www.postgresql.org/message-id/CAHut%2BPv0yWynWTmp4o34s0d98xVubys9fy%3Dp0YXsZ5_sUcNnMw%40mail... Regards, Wang wei Attachments: [application/octet-stream] v18-0001-Perform-streaming-logical-transactions-by-backgr.patch (106.6K, ../../OS3PR01MB62758A6AAED27B3A848CEB7A9E8F9@OS3PR01MB6275.jpnprd01.prod.outlook.com/2-v18-0001-Perform-streaming-logical-transactions-by-backgr.patch) download | inline diff: From b0530ad3660d5571b56478100c94b702307cd09e Mon Sep 17 00:00:00 2001 From: "houzj.fnst" <[email protected]> Date: Wed, 20 Apr 2022 16:45:07 +0800 Subject: [PATCH v18 1/4] Perform streaming logical transactions by background workers Currently, for large transactions, the publisher sends the data in multiple streams (changes divided into chunks depending upon logical_decoding_work_mem), and then on the subscriber-side, the apply worker writes the changes into temporary files and once it receives the commit, it reads from the file and applies the entire transaction. To improve the performance of such transactions, we can instead allow them to be applied via background workers. In this approach, we assign a new apply background worker (if available) as soon as the xact's first stream is received and the main apply worker will send changes to this new worker via shared memory. The apply background worker will directly apply the change instead of writing it to temporary files. We keep this worker assigned till the transaction commit is received and also wait for the worker to finish at commit. This preserves commit ordering and avoids writing to and reading from file in most cases. We still need to spill if there is no worker available. This patch also extends the SUBSCRIPTION 'streaming' parameter so that the user can control whether to apply the streaming transaction in an apply background worker or spill the change to disk. The user can set the streaming parameter to 'on/off', 'parallel'. The parameter value 'parallel' means the streaming will be applied via an apply background worker, if available. The parameter value 'on' means the streaming transaction will be spilled to disk. The default value is 'off' (same as current behaviour). --- doc/src/sgml/catalogs.sgml | 10 +- doc/src/sgml/config.sgml | 25 + doc/src/sgml/logical-replication.sgml | 10 + doc/src/sgml/protocol.sgml | 19 + doc/src/sgml/ref/create_subscription.sgml | 24 +- src/backend/access/transam/xact.c | 13 + src/backend/commands/subscriptioncmds.c | 66 +- src/backend/postmaster/bgworker.c | 3 + src/backend/replication/logical/Makefile | 1 + .../replication/logical/applybgworker.c | 802 ++++++++++++++++++ src/backend/replication/logical/decode.c | 10 +- src/backend/replication/logical/launcher.c | 130 ++- src/backend/replication/logical/origin.c | 26 +- src/backend/replication/logical/proto.c | 41 +- .../replication/logical/reorderbuffer.c | 10 +- src/backend/replication/logical/tablesync.c | 10 +- src/backend/replication/logical/worker.c | 692 +++++++++++---- src/backend/replication/pgoutput/pgoutput.c | 6 +- src/backend/utils/activity/wait_event.c | 3 + src/backend/utils/misc/guc.c | 12 + src/backend/utils/misc/postgresql.conf.sample | 1 + src/bin/pg_dump/pg_dump.c | 6 +- src/include/catalog/pg_subscription.h | 21 +- src/include/replication/logicallauncher.h | 1 + src/include/replication/logicalproto.h | 27 +- src/include/replication/logicalworker.h | 1 + src/include/replication/origin.h | 2 +- src/include/replication/reorderbuffer.h | 7 +- src/include/replication/worker_internal.h | 102 ++- src/include/utils/wait_event.h | 1 + src/test/regress/expected/subscription.out | 2 +- src/tools/pgindent/typedefs.list | 5 + 32 files changed, 1869 insertions(+), 220 deletions(-) create mode 100644 src/backend/replication/logical/applybgworker.c diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index 670a5406d6..13c9e8eca1 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -7873,11 +7873,15 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l <row> <entry role="catalog_table_entry"><para role="column_definition"> - <structfield>substream</structfield> <type>bool</type> + <structfield>substream</structfield> <type>char</type> </para> <para> - If true, the subscription will allow streaming of in-progress - transactions + Controls how to handle the streaming of in-progress transactions: + <literal>f</literal> = disallow streaming of in-progress transactions, + <literal>t</literal> = spill the changes of in-progress transactions to + disk and apply at once after the transaction is committed on the + publisher, + <literal>p</literal> = apply changes directly using a background worker </para></entry> </row> diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml index 37fd80388c..6bbd986195 100644 --- a/doc/src/sgml/config.sgml +++ b/doc/src/sgml/config.sgml @@ -4970,6 +4970,31 @@ ANY <replaceable class="parameter">num_sync</replaceable> ( <replaceable class=" </listitem> </varlistentry> + <varlistentry id="guc-max-apply-bgworkers-per-subscription" xreflabel="max_apply_bgworkers_per_subscription"> + <term><varname>max_apply_bgworkers_per_subscription</varname> (<type>integer</type>) + <indexterm> + <primary><varname>max_apply_bgworkers_per_subscription</varname> configuration parameter</primary> + </indexterm> + </term> + <listitem> + <para> + Maximum number of apply background workers per subscription. This + parameter controls the amount of parallelism of the streaming of + in-progress transactions when subscription parameter + <literal>streaming = parallel</literal>. + </para> + <para> + The apply background workers are taken from the pool defined by + <varname>max_logical_replication_workers</varname>. + </para> + <para> + The default value is 2. This parameter can only be set in the + <filename>postgresql.conf</filename> file or on the server command + line. + </para> + </listitem> + </varlistentry> + </variablelist> </sect2> diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml index bdf1e7b727..92997f9299 100644 --- a/doc/src/sgml/logical-replication.sgml +++ b/doc/src/sgml/logical-replication.sgml @@ -1153,6 +1153,16 @@ CONTEXT: processing remote data for replication origin "pg_16395" during "INSER might not violate any constraint. This can easily make the subscriber inconsistent. </para> + + <para> + When the streaming mode is <literal>parallel</literal>, the finish LSN of + failed transactions may not be logged. In that case, it may be necessary to + change the streaming mode to <literal>on</literal> and cause the same + conflicts again so the finish LSN of the failed transaction will be written + to the server log. For the usage of finish LSN, please refer to <link + linkend="sql-altersubscription"><command>ALTER SUBSCRIPTION ... + SKIP</command></link>. + </para> </sect1> <sect1 id="logical-replication-restrictions"> diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml index c0b89a3c01..7e88ba9631 100644 --- a/doc/src/sgml/protocol.sgml +++ b/doc/src/sgml/protocol.sgml @@ -6809,6 +6809,25 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;" </listitem> </varlistentry> + <varlistentry> + <term>Int64 (XLogRecPtr)</term> + <listitem> + <para> + The LSN of the abort. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Int64 (TimestampTz)</term> + <listitem> + <para> + Abort timestamp of the transaction. The value is in number + of microseconds since PostgreSQL epoch (2000-01-01). + </para> + </listitem> + </varlistentry> + <varlistentry> <term>Int32 (TransactionId)</term> <listitem> diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml index 34b3264b26..71dd4aca81 100644 --- a/doc/src/sgml/ref/create_subscription.sgml +++ b/doc/src/sgml/ref/create_subscription.sgml @@ -217,13 +217,29 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl </varlistentry> <varlistentry> - <term><literal>streaming</literal> (<type>boolean</type>)</term> + <term><literal>streaming</literal> (<type>enum</type>)</term> <listitem> <para> Specifies whether to enable streaming of in-progress transactions - for this subscription. By default, all transactions - are fully decoded on the publisher and only then sent to the - subscriber as a whole. + for this subscription. The default value is <literal>off</literal>, + meaning all transactions are fully decoded on the publisher and only + then sent to the subscriber as a whole. + </para> + + <para> + If set to <literal>on</literal>, the incoming changes are written to + temporary files and then applied only after the transaction is + committed on the publisher. + </para> + + <para> + If set to <literal>parallel</literal>, incoming changes are directly + applied via one of the apply background workers, if available. If no + background worker is free to handle streaming transaction then the + changes are written to temporary files and applied after the + transaction is committed. Note that if an error happens when + applying changes in a background worker, the finish LSN of the + remote transaction might not be reported in the server log. </para> </listitem> </varlistentry> diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c index 116de1175b..3e61a57b50 100644 --- a/src/backend/access/transam/xact.c +++ b/src/backend/access/transam/xact.c @@ -1711,6 +1711,7 @@ RecordTransactionAbort(bool isSubXact) int nchildren; TransactionId *children; TimestampTz xact_time; + bool replorigin; /* * If we haven't been assigned an XID, nobody will care whether we aborted @@ -1741,6 +1742,13 @@ RecordTransactionAbort(bool isSubXact) elog(PANIC, "cannot abort transaction %u, it was already committed", xid); + /* + * Are we using the replication origins feature? Or, in other words, + * are we replaying remote actions? + */ + replorigin = (replorigin_session_origin != InvalidRepOriginId && + replorigin_session_origin != DoNotReplicateId); + /* Fetch the data we need for the abort record */ nrels = smgrGetPendingDeletes(false, &rels); nchildren = xactGetCommittedChildren(&children); @@ -1765,6 +1773,11 @@ RecordTransactionAbort(bool isSubXact) MyXactFlags, InvalidTransactionId, NULL); + if (replorigin) + /* Move LSNs forward for this replication origin */ + replorigin_session_advance(replorigin_session_origin_lsn, + XactLastRecEnd); + /* * Report the latest async abort LSN, so that the WAL writer knows to * flush this abort. There's nothing to be gained by delaying this, since diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c index bdc1208724..d4b2616ee6 100644 --- a/src/backend/commands/subscriptioncmds.c +++ b/src/backend/commands/subscriptioncmds.c @@ -83,7 +83,7 @@ typedef struct SubOpts bool copy_data; bool refresh; bool binary; - bool streaming; + char streaming; bool twophase; bool disableonerr; XLogRecPtr lsn; @@ -95,6 +95,62 @@ static List *merge_publications(List *oldpublist, List *newpublist, bool addpub, static void ReportSlotConnectionError(List *rstates, Oid subid, char *slotname, char *err); +/* + * Extract the streaming mode value from a DefElem. This is like + * defGetBoolean() but also accepts the special value of "parallel". + */ +static char +defGetStreamingMode(DefElem *def) +{ + /* + * If no parameter value given, assume "true" is meant. + */ + if (def->arg == NULL) + return SUBSTREAM_ON; + + /* + * Allow 0, 1, "false", "true", "off", "on" or "parallel". + */ + switch (nodeTag(def->arg)) + { + case T_Integer: + switch (intVal(def->arg)) + { + case 0: + return SUBSTREAM_OFF; + case 1: + return SUBSTREAM_ON; + default: + /* otherwise, error out below */ + break; + } + break; + default: + { + char *sval = defGetString(def); + + /* + * The set of strings accepted here should match up with the + * grammar's opt_boolean_or_string production. + */ + if (pg_strcasecmp(sval, "false") == 0 || + pg_strcasecmp(sval, "off") == 0) + return SUBSTREAM_OFF; + if (pg_strcasecmp(sval, "true") == 0 || + pg_strcasecmp(sval, "on") == 0) + return SUBSTREAM_ON; + if (pg_strcasecmp(sval, "parallel") == 0) + return SUBSTREAM_PARALLEL; + } + break; + } + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s requires a Boolean value or \"parallel\"", + def->defname))); + return SUBSTREAM_OFF; /* keep compiler quiet */ +} + /* * Common option parsing function for CREATE and ALTER SUBSCRIPTION commands. * @@ -132,7 +188,7 @@ parse_subscription_options(ParseState *pstate, List *stmt_options, if (IsSet(supported_opts, SUBOPT_BINARY)) opts->binary = false; if (IsSet(supported_opts, SUBOPT_STREAMING)) - opts->streaming = false; + opts->streaming = SUBSTREAM_OFF; if (IsSet(supported_opts, SUBOPT_TWOPHASE_COMMIT)) opts->twophase = false; if (IsSet(supported_opts, SUBOPT_DISABLE_ON_ERR)) @@ -233,7 +289,7 @@ parse_subscription_options(ParseState *pstate, List *stmt_options, errorConflictingDefElem(defel, pstate); opts->specified_opts |= SUBOPT_STREAMING; - opts->streaming = defGetBoolean(defel); + opts->streaming = defGetStreamingMode(defel); } else if (strcmp(defel->defname, "two_phase") == 0) { @@ -600,7 +656,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt, values[Anum_pg_subscription_subowner - 1] = ObjectIdGetDatum(owner); values[Anum_pg_subscription_subenabled - 1] = BoolGetDatum(opts.enabled); values[Anum_pg_subscription_subbinary - 1] = BoolGetDatum(opts.binary); - values[Anum_pg_subscription_substream - 1] = BoolGetDatum(opts.streaming); + values[Anum_pg_subscription_substream - 1] = CharGetDatum(opts.streaming); values[Anum_pg_subscription_subtwophasestate - 1] = CharGetDatum(opts.twophase ? LOGICALREP_TWOPHASE_STATE_PENDING : @@ -1059,7 +1115,7 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt, if (IsSet(opts.specified_opts, SUBOPT_STREAMING)) { values[Anum_pg_subscription_substream - 1] = - BoolGetDatum(opts.streaming); + CharGetDatum(opts.streaming); replaces[Anum_pg_subscription_substream - 1] = true; } diff --git a/src/backend/postmaster/bgworker.c b/src/backend/postmaster/bgworker.c index 40601aefd9..40ccb8993c 100644 --- a/src/backend/postmaster/bgworker.c +++ b/src/backend/postmaster/bgworker.c @@ -128,6 +128,9 @@ static const struct }, { "ApplyWorkerMain", ApplyWorkerMain + }, + { + "ApplyBgworkerMain", ApplyBgworkerMain } }; diff --git a/src/backend/replication/logical/Makefile b/src/backend/replication/logical/Makefile index c4e2fdeb71..cbfb5d794e 100644 --- a/src/backend/replication/logical/Makefile +++ b/src/backend/replication/logical/Makefile @@ -15,6 +15,7 @@ include $(top_builddir)/src/Makefile.global override CPPFLAGS := -I$(srcdir) $(CPPFLAGS) OBJS = \ + applybgworker.o \ decode.o \ launcher.o \ logical.o \ diff --git a/src/backend/replication/logical/applybgworker.c b/src/backend/replication/logical/applybgworker.c new file mode 100644 index 0000000000..aa222490a0 --- /dev/null +++ b/src/backend/replication/logical/applybgworker.c @@ -0,0 +1,802 @@ +/*------------------------------------------------------------------------- + * applybgworker.c + * Support routines for applying xact by apply background worker + * + * Copyright (c) 2016-2022, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/replication/logical/applybgworker.c + * + * This file contains routines that are intended to support setting up, using, + * and tearing down a ApplyBgworkerState. + * + * Refer to the comments in file header of logical/worker.c to see more + * information about apply background worker. + * + *------------------------------------------------------------------------- + */ + +#include "postgres.h" + +#include "libpq/pqformat.h" +#include "mb/pg_wchar.h" +#include "pgstat.h" +#include "postmaster/interrupt.h" +#include "replication/logicallauncher.h" +#include "replication/logicalworker.h" +#include "replication/origin.h" +#include "replication/walreceiver.h" +#include "replication/worker_internal.h" +#include "storage/ipc.h" +#include "storage/procarray.h" +#include "tcop/tcopprot.h" +#include "utils/inval.h" +#include "utils/memutils.h" +#include "utils/resowner.h" +#include "utils/syscache.h" + +#define PG_LOGICAL_APPLY_SHM_MAGIC 0x79fb2447 // TODO Consider change + +/* + * DSM keys for apply background worker. Unlike other parallel execution code, + * since we don't need to worry about DSM keys conflicting with plan_node_id we + * can use small integers. + */ +#define APPLY_BGWORKER_KEY_SHARED 1 +#define APPLY_BGWORKER_KEY_MQ 2 + +/* Queue size of DSM, 16 MB for now. */ +#define DSM_QUEUE_SIZE 160000000 + +/* + * There are three fields in message: start_lsn, end_lsn and send_time. Because + * we have updated these statistics in apply worker, we could ignore these + * fields in apply background worker. (see function LogicalRepApplyLoop) + */ +#define IGNORE_SIZE_IN_MESSAGE (3 * sizeof(uint64)) + +/* + * Entry for a hash table we use to map from xid to our apply background worker + * state. + */ +typedef struct ApplyBgworkerEntry +{ + TransactionId xid; + ApplyBgworkerState *wstate; +} ApplyBgworkerEntry; + +/* Apply background workers hash table (initialized on first use) */ +static HTAB *ApplyWorkersHash = NULL; +static List *ApplyWorkersFreeList = NIL; +static List *ApplyWorkersList = NIL; + +/* + * Information shared between main apply worker and apply background worker. + */ +volatile ApplyBgworkerShared *MyParallelShared = NULL; + +List *subxactlist = NIL; + +static bool apply_bgworker_can_start(TransactionId xid); +static ApplyBgworkerState *apply_bgworker_setup(void); +static void apply_bgworker_setup_dsm(ApplyBgworkerState *wstate); + +/* + * Check if starting a new apply background worker is allowed. + */ +static bool +apply_bgworker_can_start(TransactionId xid) +{ + if (!TransactionIdIsValid(xid)) + return false; + + /* + * Don't start a new background worker if not in streaming parallel mode. + */ + if (MySubscription->stream != SUBSTREAM_PARALLEL) + return false; + + /* + * Don't start a new background worker if user has set skiplsn as it's + * possible that user want to skip the streaming transaction. For + * streaming transaction, we need to spill the transaction to disk so that + * we can get the last LSN of the transaction to judge whether to skip + * before starting to apply the change. + */ + if (!XLogRecPtrIsInvalid(MySubscription->skiplsn)) + return false; + + /* + * For streaming transactions that are being applied in apply background + * worker, we cannot decide whether to apply the change for a relation + * that is not in the READY state (see should_apply_changes_for_rel) as we + * won't know remote_final_lsn by that time. So, we don't start new apply + * background worker in this case. + */ + if (!AllTablesyncsReady()) + return false; + + return true; +} + +/* + * Try to start an apply background worker and, if successful, cache it in + * ApplyWorkersHash keyed by the specified xid. + */ +ApplyBgworkerState * +apply_bgworker_start(TransactionId xid) +{ + bool found; + int server_version; + ApplyBgworkerState *wstate; + ApplyBgworkerEntry *entry = NULL; + + if (!apply_bgworker_can_start(xid)) + return NULL; + + /* First time through, initialize apply workers hashtable */ + if (ApplyWorkersHash == NULL) + { + HASHCTL ctl; + + MemSet(&ctl, 0, sizeof(ctl)); + ctl.keysize = sizeof(TransactionId); + ctl.entrysize = sizeof(ApplyBgworkerEntry); + ctl.hcxt = ApplyContext; + + ApplyWorkersHash = hash_create("logical apply workers hash", 8, &ctl, + HASH_ELEM | HASH_BLOBS | HASH_CONTEXT); + } + + /* + * Now, we try to get an apply background worker. If there is at least one + * worker in the free list, then take one. Otherwise, we try to start a + * new apply background worker. + */ + if (list_length(ApplyWorkersFreeList) > 0) + { + wstate = (ApplyBgworkerState *) llast(ApplyWorkersFreeList); + ApplyWorkersFreeList = list_delete_last(ApplyWorkersFreeList); + Assert(wstate->shared->status == APPLY_BGWORKER_FINISHED); + } + else + { + wstate = apply_bgworker_setup(); + + if (wstate == NULL) + return NULL; + } + + /* + * Create entry for requested transaction. + */ + entry = hash_search(ApplyWorkersHash, &xid, HASH_ENTER, &found); + if (found) + elog(ERROR, "hash table corrupted"); + + /* Fill up the hash entry */ + wstate->shared->status = APPLY_BGWORKER_BUSY; + + server_version = walrcv_server_version(LogRepWorkerWalRcvConn); + wstate->shared->server_version = + server_version >= 160000 ? LOGICALREP_PROTO_STREAM_PARALLEL_VERSION_NUM : + server_version >= 150000 ? LOGICALREP_PROTO_TWOPHASE_VERSION_NUM : + server_version >= 140000 ? LOGICALREP_PROTO_STREAM_VERSION_NUM : + LOGICALREP_PROTO_VERSION_NUM; + + wstate->shared->stream_xid = xid; + entry->wstate = wstate; + entry->xid = xid; + + return wstate; +} + +/* + * Try to look up worker inside ApplyWorkersHash for requested xid. + */ +ApplyBgworkerState * +apply_bgworker_find(TransactionId xid) +{ + bool found; + ApplyBgworkerEntry *entry = NULL; + + if (!TransactionIdIsValid(xid)) + return NULL; + + if (ApplyWorkersHash == NULL) + return NULL; + + /* + * Find entry for requested transaction. + */ + entry = hash_search(ApplyWorkersHash, &xid, HASH_FIND, &found); + if (found) + { + char status = entry->wstate->shared->status; + + /* If any workers (or the postmaster) have died, we have failed. */ + if (status == APPLY_BGWORKER_EXIT) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("background worker %u failed to apply transaction %u", + entry->wstate->shared->n, + entry->wstate->shared->stream_xid))); + + Assert(status == APPLY_BGWORKER_BUSY); + + return entry->wstate; + } + else + return NULL; +} + +/* + * Add the worker to the free list and remove the entry from the hash table. + */ +void +apply_bgworker_free(ApplyBgworkerState *wstate) +{ + MemoryContext oldctx; + TransactionId xid = wstate->shared->stream_xid; + + Assert(wstate->shared->status == APPLY_BGWORKER_FINISHED); + + oldctx = MemoryContextSwitchTo(ApplyContext); + + hash_search(ApplyWorkersHash, &xid, HASH_REMOVE, NULL); + + elog(DEBUG1, "adding finished apply worker #%u for xid %u to the free list", + wstate->shared->n, wstate->shared->stream_xid); + + ApplyWorkersFreeList = lappend(ApplyWorkersFreeList, wstate); + + MemoryContextSwitchTo(oldctx); +} + +/* Apply Background Worker main loop */ +static void +LogicalApplyBgwLoop(shm_mq_handle *mqh, volatile ApplyBgworkerShared *shared) +{ + shm_mq_result shmq_res; + PGPROC *registrant; + ErrorContextCallback errcallback; + + registrant = BackendPidGetProc(MyBgworkerEntry->bgw_notify_pid); + SetLatch(®istrant->procLatch); + + /* + * Push apply error context callback. Fields will be filled applying a + * change. + */ + errcallback.callback = apply_error_callback; + errcallback.previous = error_context_stack; + error_context_stack = &errcallback; + + for (;;) + { + void *data; + Size len; + int c; + StringInfoData s; + MemoryContext oldctx; + + CHECK_FOR_INTERRUPTS(); + + /* Ensure we are reading the data into our memory context. */ + oldctx = MemoryContextSwitchTo(ApplyMessageContext); + + shmq_res = shm_mq_receive(mqh, &len, &data, false); + + if (shmq_res != SHM_MQ_SUCCESS) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("lost connection to the main apply worker"))); + + if (len == 0) + break; + + s.cursor = 0; + s.maxlen = -1; + s.data = (char *) data; + s.len = len; + + /* + * We use first byte of message for additional communication between + * main Logical replication worker and apply background workers, so if + * it differs from 'w', then process it first. + */ + c = pq_getmsgbyte(&s); + switch (c) + { + /* End message of streaming chunk */ + case LOGICAL_REP_MSG_STREAM_STOP: + elog(DEBUG1, "[Apply BGW #%u] ended processing streaming chunk," + "waiting on shm_mq_receive", shared->n); + + in_streamed_transaction = false; + pgstat_report_activity(STATE_IDLEINTRANSACTION, NULL); + continue; + case 'w': + break; + default: + elog(ERROR, "[Apply BGW #%u] unexpected message \"%c\"", + shared->n, c); + break; + } + + /* Ignore statistics fields that have been updated. */ + s.cursor += IGNORE_SIZE_IN_MESSAGE; + + apply_dispatch(&s); + + if (ConfigReloadPending) + { + ConfigReloadPending = false; + ProcessConfigFile(PGC_SIGHUP); + } + + MemoryContextSwitchTo(oldctx); + MemoryContextReset(ApplyMessageContext); + } + + MemoryContextSwitchTo(TopMemoryContext); + MemoryContextReset(ApplyContext); + + /* Pop the error context stack */ + error_context_stack = errcallback.previous; + + elog(DEBUG1, "[Apply BGW #%u] exiting", shared->n); + + /* Signal main process that we are done. */ + SetLatch(®istrant->procLatch); +} + +/* + * Set the exit status so that the main apply worker can realize we have + * shutdown. + */ +static void +apply_bgworker_shutdown(int code, Datum arg) +{ + SpinLockAcquire(&MyParallelShared->mutex); + MyParallelShared->status = APPLY_BGWORKER_EXIT; + SpinLockRelease(&MyParallelShared->mutex); + + dsm_detach((dsm_segment *) DatumGetPointer(arg)); +} + +/* + * Apply Background Worker entry point + */ +void +ApplyBgworkerMain(Datum main_arg) +{ + volatile ApplyBgworkerShared *shared; + + dsm_handle handle; + dsm_segment *seg; + shm_toc *toc; + shm_mq *mq; + shm_mq_handle *mqh; + MemoryContext oldcontext; + RepOriginId originid; + int worker_slot = DatumGetInt32(main_arg); + char originname[NAMEDATALEN]; + + MemoryContextSwitchTo(TopMemoryContext); + + /* Init the memory context for the apply background worker to work in. */ + ApplyContext = AllocSetContextCreate(TopMemoryContext, + "ApplyContext", + ALLOCSET_DEFAULT_SIZES); + + /* + * Init the ApplyMessageContext which we clean up after each replication + * protocol message. + */ + ApplyMessageContext = AllocSetContextCreate(ApplyContext, + "ApplyMessageContext", + ALLOCSET_DEFAULT_SIZES); + + /* Setup signal handling */ + pqsignal(SIGHUP, SignalHandlerForConfigReload); + pqsignal(SIGTERM, die); + BackgroundWorkerUnblockSignals(); + + /* + * Connect to the dynamic shared memory segment. + * + * The backend that registered this worker passed us the ID of a shared + * memory segment to which we must attach for further instructions. In + * order to attach to dynamic shared memory, we need a resource owner. + * Once we've mapped the segment in our address space, attach to the table + * of contents so we can locate the various data structures we'll need to + * find within the segment. + */ + CurrentResourceOwner = ResourceOwnerCreate(NULL, "Logical apply worker"); + memcpy(&handle, MyBgworkerEntry->bgw_extra, sizeof(dsm_handle)); + seg = dsm_attach(handle); + if (seg == NULL) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("unable to map dynamic shared memory segment"))); + toc = shm_toc_attach(PG_LOGICAL_APPLY_SHM_MAGIC, dsm_segment_address(seg)); + if (toc == NULL) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("bad magic number in dynamic shared memory segment"))); + + before_shmem_exit(apply_bgworker_shutdown, PointerGetDatum(seg)); + + /* Look up the shared information. */ + shared = shm_toc_lookup(toc, APPLY_BGWORKER_KEY_SHARED, false); + MyParallelShared = shared; + + /* + * Attach to the message queue. + */ + mq = shm_toc_lookup(toc, APPLY_BGWORKER_KEY_MQ, false); + shm_mq_set_receiver(mq, MyProc); + mqh = shm_mq_attach(mq, seg, NULL); + + /* Run as replica session replication role. */ + SetConfigOption("session_replication_role", "replica", + PGC_SUSET, PGC_S_OVERRIDE); + + /* + * Now, we have initialized DSM. Attach to slot. + */ + logicalrep_worker_attach(worker_slot); + + /* Connect to our database. */ + BackgroundWorkerInitializeConnectionByOid(MyLogicalRepWorker->dbid, + MyLogicalRepWorker->userid, + 0); + + /* + * Set always-secure search path, so malicious users can't redirect user + * code (e.g. pg_index.indexprs). + */ + SetConfigOption("search_path", "", PGC_SUSET, PGC_S_OVERRIDE); + + /* + * Set the client encoding to the database encoding, since that is what + * the leader will expect. + */ + SetClientEncoding(GetDatabaseEncoding()); + + stream_xid = shared->stream_xid; + + StartTransactionCommand(); + oldcontext = MemoryContextSwitchTo(ApplyContext); + + MySubscription = GetSubscription(MyLogicalRepWorker->subid, true); + if (!MySubscription) + { + ereport(LOG, + (errmsg("logical replication apply worker for subscription %u will not " + "start because the subscription was removed during startup", + MyLogicalRepWorker->subid))); + proc_exit(0); + } + + MySubscriptionValid = true; + MemoryContextSwitchTo(oldcontext); + + /* Setup synchronous commit according to the user's wishes */ + SetConfigOption("synchronous_commit", MySubscription->synccommit, + PGC_BACKEND, PGC_S_OVERRIDE); + + /* Keep us informed about subscription changes. */ + CacheRegisterSyscacheCallback(SUBSCRIPTIONOID, + subscription_change_cb, + (Datum) 0); + + CommitTransactionCommand(); + + /* Setup replication origin tracking. */ + StartTransactionCommand(); + snprintf(originname, sizeof(originname), "pg_%u", MySubscription->oid); + originid = replorigin_by_name(originname, true); + if (!OidIsValid(originid)) + originid = replorigin_create(originname); + + /* + * The apply background worker doesn't need to monopolize this replication + * origin which was already acquired by its leader process. + */ + replorigin_session_setup(originid, false); + replorigin_session_origin = originid; + CommitTransactionCommand(); + + /* + * Allocate the origin name in long-lived context for error context + * message. + */ + apply_error_callback_arg.origin_name = MemoryContextStrdup(ApplyContext, + originname); + + elog(DEBUG1, "[Apply BGW #%u] started", shared->n); + + LogicalApplyBgwLoop(mqh, shared); + + /* + * We're done. Explicitly detach the shared memory segment so that we + * don't get a resource leak warning at commit time. This will fire any + * on_dsm_detach callbacks we've registered, as well. Once that's done, + * we can go ahead and exit. + */ + dsm_detach(seg); + proc_exit(0); +} + +/* + * Set up a dynamic shared memory segment. + * + * We set up a control region that contains a ApplyBgworkerShared, + * plus one region per message queue. There are as many message queues as + * the number of workers. + */ +static void +apply_bgworker_setup_dsm(ApplyBgworkerState *wstate) +{ + shm_toc_estimator e; + Size segsize; + dsm_segment *seg; + shm_toc *toc; + ApplyBgworkerShared *shared; + shm_mq *mq; + int64 queue_size = DSM_QUEUE_SIZE; + int server_version; + + /* + * Estimate how much shared memory we need. + * + * Because the TOC machinery may choose to insert padding of oddly-sized + * requests, we must estimate each chunk separately. + * + * We need one key to register the location of the header, and we need + * another key to track the location of the message queue. + */ + shm_toc_initialize_estimator(&e); + shm_toc_estimate_chunk(&e, sizeof(ApplyBgworkerShared)); + shm_toc_estimate_chunk(&e, (Size) queue_size); + + shm_toc_estimate_keys(&e, 1 + 1); + segsize = shm_toc_estimate(&e); + + /* Create the shared memory segment and establish a table of contents. */ + seg = dsm_create(shm_toc_estimate(&e), 0); + toc = shm_toc_create(PG_LOGICAL_APPLY_SHM_MAGIC, dsm_segment_address(seg), + segsize); + + /* Set up the header region. */ + shared = shm_toc_allocate(toc, sizeof(ApplyBgworkerShared)); + SpinLockInit(&shared->mutex); + shared->status = APPLY_BGWORKER_BUSY; + + server_version = walrcv_server_version(LogRepWorkerWalRcvConn); + shared->server_version = + server_version >= 160000 ? LOGICALREP_PROTO_STREAM_PARALLEL_VERSION_NUM : + server_version >= 150000 ? LOGICALREP_PROTO_TWOPHASE_VERSION_NUM : + server_version >= 140000 ? LOGICALREP_PROTO_STREAM_VERSION_NUM : + LOGICALREP_PROTO_VERSION_NUM; + + shared->stream_xid = stream_xid; + shared->n = list_length(ApplyWorkersList) + 1; + + shm_toc_insert(toc, APPLY_BGWORKER_KEY_SHARED, shared); + + /* Set up message queue for the worker. */ + mq = shm_mq_create(shm_toc_allocate(toc, (Size) queue_size), + (Size) queue_size); + shm_toc_insert(toc, APPLY_BGWORKER_KEY_MQ, mq); + shm_mq_set_sender(mq, MyProc); + + /* Attach the queue. */ + wstate->mq_handle = shm_mq_attach(mq, seg, NULL); + + /* Return results to caller. */ + wstate->dsm_seg = seg; + wstate->shared = shared; +} + +/* + * Start apply background worker process and allocate shared memory for it. + */ +static ApplyBgworkerState * +apply_bgworker_setup(void) +{ + MemoryContext oldcontext; + bool launched; + ApplyBgworkerState *wstate; + int napplyworkers; + + elog(DEBUG1, "setting up apply worker #%u", list_length(ApplyWorkersList) + 1); + + /* Check if there are free worker slot(s) */ + LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); + napplyworkers = logicalrep_apply_bgworker_count(MyLogicalRepWorker->subid); + LWLockRelease(LogicalRepWorkerLock); + if (napplyworkers >= max_apply_bgworkers_per_subscription) + return NULL; + + oldcontext = MemoryContextSwitchTo(ApplyContext); + + wstate = (ApplyBgworkerState *) palloc0(sizeof(ApplyBgworkerState)); + + /* Setup shared memory */ + apply_bgworker_setup_dsm(wstate); + + launched = logicalrep_worker_launch(MyLogicalRepWorker->dbid, + MySubscription->oid, + MySubscription->name, + MyLogicalRepWorker->userid, + InvalidOid, + dsm_segment_handle(wstate->dsm_seg)); + + if (launched) + ApplyWorkersList = lappend(ApplyWorkersList, wstate); + else + { + dsm_detach(wstate->dsm_seg); + wstate->dsm_seg = NULL; + + pfree(wstate); + wstate = NULL; + } + + MemoryContextSwitchTo(oldcontext); + + return wstate; +} + +/* + * Send the data to the specified apply background worker via shared-memory queue. + */ +void +apply_bgworker_send_data(ApplyBgworkerState *wstate, Size nbytes, const void *data) +{ + shm_mq_result result; + + result = shm_mq_send(wstate->mq_handle, nbytes, data, false, true); + + if (result != SHM_MQ_SUCCESS) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("could not send tuples to shared-memory queue"))); +} + +/* + * Wait until the status of apply background worker reaches the + * 'wait_for_status' + */ +void +apply_bgworker_wait_for(ApplyBgworkerState *wstate, + ApplyBgworkerStatus wait_for_status) +{ + for (;;) + { + char status; + + SpinLockAcquire(&wstate->shared->mutex); + status = wstate->shared->status; + SpinLockRelease(&wstate->shared->mutex); + + /* Done if already in correct status. */ + if (status == wait_for_status) + break; + + /* If any workers (or the postmaster) have died, we have failed. */ + if (status == APPLY_BGWORKER_EXIT) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("background worker %u failed to apply transaction %u", + wstate->shared->n, wstate->shared->stream_xid))); + + /* Wait to be signalled. */ + WaitLatch(MyLatch, WL_LATCH_SET | WL_EXIT_ON_PM_DEATH, 0, + WAIT_EVENT_LOGICAL_APPLY_WORKER_STATE_CHANGE); + + /* Reset the latch so we don't spin. */ + ResetLatch(MyLatch); + + /* An interrupt may have occurred while we were waiting. */ + CHECK_FOR_INTERRUPTS(); + } +} + +/* + * Check the status of workers and report an error if any apply background + * worker has exited unexpectedly. + */ +void +apply_bgworker_check_status(void) +{ + ListCell *lc; + + if (am_apply_bgworker() || MySubscription->stream != SUBSTREAM_PARALLEL) + return; + + foreach(lc, ApplyWorkersList) + { + ApplyBgworkerState *wstate = (ApplyBgworkerState *) lfirst(lc); + + /* + * We don't lock here as in the worst case we will just detect the + * failure of worker a bit later. + */ + if (wstate->shared->status == APPLY_BGWORKER_EXIT) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("background worker %u exited unexpectedly", + wstate->shared->n))); + } + + /* + * Exit if any relation is not in the READY state and if any worker is + * handling the streaming transaction at the same time. Because for + * streaming transactions that is being applied in apply background + * worker, we cannot decide whether to apply the change for a relation + * that is not in the READY state (see should_apply_changes_for_rel) as we + * won't know remote_final_lsn by that time. + */ + if (list_length(ApplyWorkersFreeList) != list_length(ApplyWorkersList) && + !AllTablesyncsReady()) + { + ereport(LOG, + (errmsg("logical replication apply workers for subscription \"%s\" will restart", + MySubscription->name), + errdetail("Cannot handle streamed replication transaction by apply " + "background workers until all tables are synchronized"))); + + proc_exit(0); + } +} + +/* Set the apply background worker status */ +void +apply_bgworker_set_status(ApplyBgworkerStatus status) +{ + if (!am_apply_bgworker()) + return; + + elog(DEBUG1, "[Apply BGW #%u] set status to %d", MyParallelShared->n, status); + + SpinLockAcquire(&MyParallelShared->mutex); + MyParallelShared->status = status; + SpinLockRelease(&MyParallelShared->mutex); +} + +/* + * Define a savepoint for a subxact in apply background worker if needed. + * + * Inside apply background worker we can figure out that new subtransaction was + * started if new change arrived with different xid. In that case we can define + * named savepoint, so that we were able to commit/rollback it separately + * later. + * Special case is if the first change comes from subtransaction, then + * we check that current_xid differs from stream_xid. + */ +void +apply_bgworker_subxact_info_add(TransactionId current_xid) +{ + if (current_xid != stream_xid && + !list_member_int(subxactlist, (int) current_xid)) + { + MemoryContext oldctx; + char spname[MAXPGPATH]; + + snprintf(spname, MAXPGPATH, "savepoint_for_xid_%u", current_xid); + + elog(DEBUG1, "[Apply BGW #%u] defining savepoint %s", + MyParallelShared->n, spname); + + DefineSavepoint(spname); + CommitTransactionCommand(); + + oldctx = MemoryContextSwitchTo(ApplyContext); + subxactlist = lappend_int(subxactlist, (int) current_xid); + MemoryContextSwitchTo(oldctx); + } +} diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c index c5c6a2ba68..d4d5093a0b 100644 --- a/src/backend/replication/logical/decode.c +++ b/src/backend/replication/logical/decode.c @@ -651,9 +651,10 @@ DecodeCommit(LogicalDecodingContext *ctx, XLogRecordBuffer *buf, { for (i = 0; i < parsed->nsubxacts; i++) { - ReorderBufferForget(ctx->reorder, parsed->subxacts[i], buf->origptr); + ReorderBufferForget(ctx->reorder, parsed->subxacts[i], buf->origptr, + commit_time); } - ReorderBufferForget(ctx->reorder, xid, buf->origptr); + ReorderBufferForget(ctx->reorder, xid, buf->origptr, commit_time); return; } @@ -821,10 +822,11 @@ DecodeAbort(LogicalDecodingContext *ctx, XLogRecordBuffer *buf, for (i = 0; i < parsed->nsubxacts; i++) { ReorderBufferAbort(ctx->reorder, parsed->subxacts[i], - buf->record->EndRecPtr); + buf->record->EndRecPtr, abort_time); } - ReorderBufferAbort(ctx->reorder, xid, buf->record->EndRecPtr); + ReorderBufferAbort(ctx->reorder, xid, buf->record->EndRecPtr, + abort_time); } /* update the decoding stats */ diff --git a/src/backend/replication/logical/launcher.c b/src/backend/replication/logical/launcher.c index 3bbd522724..d92bfaf6d6 100644 --- a/src/backend/replication/logical/launcher.c +++ b/src/backend/replication/logical/launcher.c @@ -54,6 +54,7 @@ int max_logical_replication_workers = 4; int max_sync_workers_per_subscription = 2; +int max_apply_bgworkers_per_subscription = 2; LogicalRepWorker *MyLogicalRepWorker = NULL; @@ -73,6 +74,7 @@ static void logicalrep_launcher_onexit(int code, Datum arg); static void logicalrep_worker_onexit(int code, Datum arg); static void logicalrep_worker_detach(void); static void logicalrep_worker_cleanup(LogicalRepWorker *worker); +static void logicalrep_worker_stop_internal(LogicalRepWorker *worker); static bool on_commit_launcher_wakeup = false; @@ -151,8 +153,10 @@ get_subscription_list(void) * * This is only needed for cleaning up the shared memory in case the worker * fails to attach. + * + * Return false if the attach fails. Otherwise return true. */ -static void +static bool WaitForReplicationWorkerAttach(LogicalRepWorker *worker, uint16 generation, BackgroundWorkerHandle *handle) @@ -168,11 +172,11 @@ WaitForReplicationWorkerAttach(LogicalRepWorker *worker, LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); - /* Worker either died or has started; no need to do anything. */ + /* Worker either died or has started. Return false if died. */ if (!worker->in_use || worker->proc) { LWLockRelease(LogicalRepWorkerLock); - return; + return worker->in_use; } LWLockRelease(LogicalRepWorkerLock); @@ -187,7 +191,7 @@ WaitForReplicationWorkerAttach(LogicalRepWorker *worker, if (generation == worker->generation) logicalrep_worker_cleanup(worker); LWLockRelease(LogicalRepWorkerLock); - return; + return false; } /* @@ -223,6 +227,13 @@ logicalrep_worker_find(Oid subid, Oid relid, bool only_running) { LogicalRepWorker *w = &LogicalRepCtx->workers[i]; + /* + * We are only interested in the main apply worker or table sync worker + * here. + */ + if (w->subworker) + continue; + if (w->in_use && w->subid == subid && w->relid == relid && (!only_running || w->proc)) { @@ -259,11 +270,11 @@ logicalrep_workers_find(Oid subid, bool only_running) } /* - * Start new apply background worker, if possible. + * Start new background worker, if possible. */ -void +bool logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, Oid userid, - Oid relid) + Oid relid, dsm_handle subworker_dsm) { BackgroundWorker bgw; BackgroundWorkerHandle *bgw_handle; @@ -273,6 +284,10 @@ logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, Oid userid, LogicalRepWorker *worker = NULL; int nsyncworkers; TimestampTz now; + bool is_subworker = (subworker_dsm != DSM_HANDLE_INVALID); + + /* Sanity check : we don't support table sync in subworker. */ + Assert(!(is_subworker && OidIsValid(relid))); ereport(DEBUG1, (errmsg_internal("starting logical replication worker for subscription \"%s\"", @@ -350,7 +365,7 @@ retry: if (OidIsValid(relid) && nsyncworkers >= max_sync_workers_per_subscription) { LWLockRelease(LogicalRepWorkerLock); - return; + return false; } /* @@ -364,7 +379,7 @@ retry: (errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED), errmsg("out of logical replication worker slots"), errhint("You might need to increase max_logical_replication_workers."))); - return; + return false; } /* Prepare the worker slot. */ @@ -379,6 +394,7 @@ retry: worker->relstate = SUBREL_STATE_UNKNOWN; worker->relstate_lsn = InvalidXLogRecPtr; worker->stream_fileset = NULL; + worker->subworker = is_subworker; worker->last_lsn = InvalidXLogRecPtr; TIMESTAMP_NOBEGIN(worker->last_send_time); TIMESTAMP_NOBEGIN(worker->last_recv_time); @@ -396,19 +412,31 @@ retry: BGWORKER_BACKEND_DATABASE_CONNECTION; bgw.bgw_start_time = BgWorkerStart_RecoveryFinished; snprintf(bgw.bgw_library_name, BGW_MAXLEN, "postgres"); - snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyWorkerMain"); + + if (is_subworker) + snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyBgworkerMain"); + else + snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyWorkerMain"); + if (OidIsValid(relid)) snprintf(bgw.bgw_name, BGW_MAXLEN, "logical replication worker for subscription %u sync %u", subid, relid); + else if (is_subworker) + snprintf(bgw.bgw_name, BGW_MAXLEN, + "logical replication apply background worker for subscription %u", subid); else snprintf(bgw.bgw_name, BGW_MAXLEN, "logical replication worker for subscription %u", subid); + snprintf(bgw.bgw_type, BGW_MAXLEN, "logical replication worker"); bgw.bgw_restart_time = BGW_NEVER_RESTART; bgw.bgw_notify_pid = MyProcPid; bgw.bgw_main_arg = Int32GetDatum(slot); + if (is_subworker) + memcpy(bgw.bgw_extra, &subworker_dsm, sizeof(dsm_handle)); + if (!RegisterDynamicBackgroundWorker(&bgw, &bgw_handle)) { /* Failed to start worker, so clean up the worker slot. */ @@ -421,11 +449,11 @@ retry: (errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED), errmsg("out of background worker slots"), errhint("You might need to increase max_worker_processes."))); - return; + return false; } /* Now wait until it attaches. */ - WaitForReplicationWorkerAttach(worker, generation, bgw_handle); + return WaitForReplicationWorkerAttach(worker, generation, bgw_handle); } /* @@ -436,18 +464,27 @@ void logicalrep_worker_stop(Oid subid, Oid relid) { LogicalRepWorker *worker; - uint16 generation; LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); worker = logicalrep_worker_find(subid, relid, false); - /* No worker, nothing to do. */ - if (!worker) - { - LWLockRelease(LogicalRepWorkerLock); - return; - } + if (worker) + logicalrep_worker_stop_internal(worker); + + LWLockRelease(LogicalRepWorkerLock); +} + +/* + * Workhorse for logicalrep_worker_stop() and logicalrep_worker_detach(). Stop + * the worker and wait for it to die. + */ +static void +logicalrep_worker_stop_internal(LogicalRepWorker *worker) +{ + uint16 generation; + + Assert(LWLockHeldByMe(LogicalRepWorkerLock)); /* * Remember which generation was our worker so we can check if what we see @@ -485,10 +522,7 @@ logicalrep_worker_stop(Oid subid, Oid relid) * different, meaning that a different worker has taken the slot. */ if (!worker->in_use || worker->generation != generation) - { - LWLockRelease(LogicalRepWorkerLock); return; - } /* Worker has assigned proc, so it has started. */ if (worker->proc) @@ -522,8 +556,6 @@ logicalrep_worker_stop(Oid subid, Oid relid) LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); } - - LWLockRelease(LogicalRepWorkerLock); } /* @@ -599,6 +631,29 @@ logicalrep_worker_attach(int slot) static void logicalrep_worker_detach(void) { + /* + * If we are the main apply worker, stop all the apply background workers + * we started before. + */ + if (!MyLogicalRepWorker->subworker) + { + List *workers; + ListCell *lc; + + LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); + + workers = logicalrep_workers_find(MyLogicalRepWorker->subid, true); + foreach(lc, workers) + { + LogicalRepWorker *w = (LogicalRepWorker *) lfirst(lc); + + if (w->subworker) + logicalrep_worker_stop_internal(w); + } + + LWLockRelease(LogicalRepWorkerLock); + } + /* Block concurrent access. */ LWLockAcquire(LogicalRepWorkerLock, LW_EXCLUSIVE); @@ -621,6 +676,7 @@ logicalrep_worker_cleanup(LogicalRepWorker *worker) worker->userid = InvalidOid; worker->subid = InvalidOid; worker->relid = InvalidOid; + worker->subworker = false; } /* @@ -679,6 +735,30 @@ logicalrep_sync_worker_count(Oid subid) return res; } +/* + * Count the number of registered (not necessarily running) apply background + * workers for a subscription. + */ +int +logicalrep_apply_bgworker_count(Oid subid) +{ + int i; + int res = 0; + + Assert(LWLockHeldByMe(LogicalRepWorkerLock)); + + /* Search for attached worker for a given subscription id. */ + for (i = 0; i < max_logical_replication_workers; i++) + { + LogicalRepWorker *w = &LogicalRepCtx->workers[i]; + + if (w->subid == subid && w->subworker) + res++; + } + + return res; +} + /* * ApplyLauncherShmemSize * Compute space needed for replication launcher shared memory @@ -868,7 +948,7 @@ ApplyLauncherMain(Datum main_arg) wait_time = wal_retrieve_retry_interval; logicalrep_worker_launch(sub->dbid, sub->oid, sub->name, - sub->owner, InvalidOid); + sub->owner, InvalidOid, DSM_HANDLE_INVALID); } } diff --git a/src/backend/replication/logical/origin.c b/src/backend/replication/logical/origin.c index 21937ab2d3..50c567fb6e 100644 --- a/src/backend/replication/logical/origin.c +++ b/src/backend/replication/logical/origin.c @@ -1063,12 +1063,21 @@ ReplicationOriginExitCleanup(int code, Datum arg) * array doesn't have to be searched when calling * replorigin_session_advance(). * - * Obviously only one such cached origin can exist per process and the current + * Normally only one such cached origin can exist per process and the current * cached value can only be set again after the previous value is torn down * with replorigin_session_reset(). + * + * However, if the function parameter 'must_acquire' is false, we allow the + * process to use the same slot already acquired by another process. It's safe + * because 1) The only caller (apply background workers) will maintain the + * commit order by allowing only one process to commit at a time, so no two + * workers will be operating on the same origin at the same time (see comments + * in logical/worker.c). 2) Even though we try to advance the session's origin + * concurrently, it's safe to do so as we change/advance the session_origin + * LSNs under replicate_state LWLock. */ void -replorigin_session_setup(RepOriginId node) +replorigin_session_setup(RepOriginId node, bool must_acquire) { static bool registered_cleanup; int i; @@ -1110,7 +1119,7 @@ replorigin_session_setup(RepOriginId node) if (curstate->roident != node) continue; - else if (curstate->acquired_by != 0) + else if (curstate->acquired_by != 0 && must_acquire) { ereport(ERROR, (errcode(ERRCODE_OBJECT_IN_USE), @@ -1141,7 +1150,14 @@ replorigin_session_setup(RepOriginId node) Assert(session_replication_state->roident != InvalidRepOriginId); - session_replication_state->acquired_by = MyProcPid; + if (must_acquire) + session_replication_state->acquired_by = MyProcPid; + else if (session_replication_state->acquired_by == 0) + ereport(ERROR, + (errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED), + errmsg("apply background worker could not find replication state slot for replication origin with OID %u", + node), + errdetail("There is no replication state slot set by its main apply worker."))); LWLockRelease(ReplicationOriginLock); @@ -1321,7 +1337,7 @@ pg_replication_origin_session_setup(PG_FUNCTION_ARGS) name = text_to_cstring((text *) DatumGetPointer(PG_GETARG_DATUM(0))); origin = replorigin_by_name(name, false); - replorigin_session_setup(origin); + replorigin_session_setup(origin, true); replorigin_session_origin = origin; diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c index ff8513e2d2..47bd811fb7 100644 --- a/src/backend/replication/logical/proto.c +++ b/src/backend/replication/logical/proto.c @@ -1163,31 +1163,56 @@ logicalrep_read_stream_commit(StringInfo in, LogicalRepCommitData *commit_data) /* * Write STREAM ABORT to the output stream. Note that xid and subxid will be * same for the top-level transaction abort. + * + * If write_abort_lsn is true, send the abort_lsn and abort_time fields, + * otherwise don't. */ void logicalrep_write_stream_abort(StringInfo out, TransactionId xid, - TransactionId subxid) + ReorderBufferTXN *txn, XLogRecPtr abort_lsn, + bool write_abort_lsn) { pq_sendbyte(out, LOGICAL_REP_MSG_STREAM_ABORT); - Assert(TransactionIdIsValid(xid) && TransactionIdIsValid(subxid)); + Assert(TransactionIdIsValid(xid) && TransactionIdIsValid(txn->xid)); /* transaction ID */ pq_sendint32(out, xid); - pq_sendint32(out, subxid); + pq_sendint32(out, txn->xid); + + if (write_abort_lsn) + { + pq_sendint64(out, abort_lsn); + pq_sendint64(out, txn->xact_time.abort_time); + } } /* * Read STREAM ABORT from the output stream. + * + * If read_abort_lsn is true, try to read the abort_lsn and abort_time fields, + * otherwise don't. */ void -logicalrep_read_stream_abort(StringInfo in, TransactionId *xid, - TransactionId *subxid) +logicalrep_read_stream_abort(StringInfo in, + LogicalRepStreamAbortData *abort_data, + bool read_abort_lsn) { - Assert(xid && subxid); + Assert(abort_data); - *xid = pq_getmsgint(in, 4); - *subxid = pq_getmsgint(in, 4); + abort_data->xid = pq_getmsgint(in, 4); + abort_data->subxid = pq_getmsgint(in, 4); + + if (read_abort_lsn) + { + abort_data->abort_lsn = pq_getmsgint64(in); + abort_data->abort_time = pq_getmsgint64(in); + } + else + { + abort_data->abort_lsn = InvalidXLogRecPtr; + abort_data->abort_time = 0; + } } /* diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c index 88a37fde72..8989328046 100644 --- a/src/backend/replication/logical/reorderbuffer.c +++ b/src/backend/replication/logical/reorderbuffer.c @@ -2826,7 +2826,8 @@ ReorderBufferFinishPrepared(ReorderBuffer *rb, TransactionId xid, * disk. */ void -ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn) +ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn, + TimestampTz abort_time) { ReorderBufferTXN *txn; @@ -2837,6 +2838,8 @@ ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn) if (txn == NULL) return; + txn->xact_time.abort_time = abort_time; + /* For streamed transactions notify the remote node about the abort. */ if (rbtxn_is_streamed(txn)) { @@ -2911,7 +2914,8 @@ ReorderBufferAbortOld(ReorderBuffer *rb, TransactionId oldestRunningXid) * to this xid might re-create the transaction incompletely. */ void -ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn) +ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn, + TimestampTz abort_time) { ReorderBufferTXN *txn; @@ -2922,6 +2926,8 @@ ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn) if (txn == NULL) return; + txn->xact_time.abort_time = abort_time; + /* For streamed transactions notify the remote node about the abort. */ if (rbtxn_is_streamed(txn)) rb->stream_abort(rb, txn, lsn); diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c index 670c6fcada..8ffba7e2e5 100644 --- a/src/backend/replication/logical/tablesync.c +++ b/src/backend/replication/logical/tablesync.c @@ -568,7 +568,8 @@ process_syncing_tables_for_apply(XLogRecPtr current_lsn) MySubscription->oid, MySubscription->name, MyLogicalRepWorker->userid, - rstate->relid); + rstate->relid, + DSM_HANDLE_INVALID); hentry->last_start_time = now; } } @@ -589,6 +590,9 @@ process_syncing_tables_for_apply(XLogRecPtr current_lsn) void process_syncing_tables(XLogRecPtr current_lsn) { + if (MyLogicalRepWorker->subworker) + return; + if (am_tablesync_worker()) process_syncing_tables_for_sync(current_lsn); else @@ -1273,7 +1277,7 @@ LogicalRepSyncTableStart(XLogRecPtr *origin_startpos) * time this tablesync was launched. */ originid = replorigin_by_name(originname, false); - replorigin_session_setup(originid); + replorigin_session_setup(originid, true); replorigin_session_origin = originid; *origin_startpos = replorigin_session_get_progress(false); @@ -1384,7 +1388,7 @@ LogicalRepSyncTableStart(XLogRecPtr *origin_startpos) true /* go backward */ , true /* WAL log */ ); UnlockRelationOid(ReplicationOriginRelationId, RowExclusiveLock); - replorigin_session_setup(originid); + replorigin_session_setup(originid, true); replorigin_session_origin = originid; } else diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c index 38e3b1c1b3..b5aae0e19a 100644 --- a/src/backend/replication/logical/worker.c +++ b/src/backend/replication/logical/worker.c @@ -22,8 +22,28 @@ * STREAMED TRANSACTIONS * --------------------- * Streamed transactions (large transactions exceeding a memory limit on the - * upstream) are not applied immediately, but instead, the data is written - * to temporary files and then applied at once when the final commit arrives. + * upstream) are applied using one of two approaches. + * + * 1) Separate background workers + * + * If streaming = parallel, we assign a new apply background worker (if + * available) as soon as the xact's first stream is received. The main apply + * worker will send changes to this new worker via shared memory. We keep this + * worker assigned till the transaction commit is received and also wait for + * the worker to finish at commit. This preserves commit ordering and avoids + * file I/O in most cases. We still need to spill to a file if there is no + * worker available. It is important to maintain commit order to avoid failures + * due to (a) transaction dependencies, say if we insert a row in the first + * transaction and update it in the second transaction then allowing to apply + * both in parallel can lead to failure in the update. (b) deadlocks, allowing + * transactions that update the same set of rows/tables in opposite order to be + * applied in parallel can lead to deadlocks. + * + * 2) Write to temporary files and apply when the final commit arrives + * + * If no worker is available to handle streamed transaction, the data is + * written to temporary files and then applied at once when the final commit + * arrives. * * Unlike the regular (non-streamed) case, handling streamed transactions has * to handle aborts of both the toplevel transaction and subtransactions. This @@ -219,20 +239,8 @@ typedef struct ApplyExecutionData PartitionTupleRouting *proute; /* partition routing info */ } ApplyExecutionData; -/* Struct for saving and restoring apply errcontext information */ -typedef struct ApplyErrorCallbackArg -{ - LogicalRepMsgType command; /* 0 if invalid */ - LogicalRepRelMapEntry *rel; - - /* Remote node information */ - int remote_attnum; /* -1 if invalid */ - TransactionId remote_xid; - XLogRecPtr finish_lsn; - char *origin_name; -} ApplyErrorCallbackArg; - -static ApplyErrorCallbackArg apply_error_callback_arg = +/* errcontext tracker */ +ApplyErrorCallbackArg apply_error_callback_arg = { .command = 0, .rel = NULL, @@ -242,7 +250,7 @@ static ApplyErrorCallbackArg apply_error_callback_arg = .origin_name = NULL, }; -static MemoryContext ApplyMessageContext = NULL; +MemoryContext ApplyMessageContext = NULL; MemoryContext ApplyContext = NULL; /* per stream context for streaming transactions */ @@ -251,27 +259,39 @@ static MemoryContext LogicalStreamingContext = NULL; WalReceiverConn *LogRepWorkerWalRcvConn = NULL; Subscription *MySubscription = NULL; -static bool MySubscriptionValid = false; +bool MySubscriptionValid = false; bool in_remote_transaction = false; static XLogRecPtr remote_final_lsn = InvalidXLogRecPtr; /* fields valid only when processing streamed transaction */ -static bool in_streamed_transaction = false; +bool in_streamed_transaction = false; + +TransactionId stream_xid = InvalidTransactionId; +static ApplyBgworkerState *stream_apply_worker = NULL; -static TransactionId stream_xid = InvalidTransactionId; +/* Check if we are applying the transaction in an apply background worker */ +#define apply_bgworker_active() (in_streamed_transaction && stream_apply_worker != NULL) + +/* + * The number of changes during one streaming block (only for apply background + * workers) + */ +static uint32 nchanges = 0; /* * We enable skipping all data modification changes (INSERT, UPDATE, etc.) for * the subscription if the remote transaction's finish LSN matches the subskiplsn. * Once we start skipping changes, we don't stop it until we skip all changes of * the transaction even if pg_subscription is updated and MySubscription->skiplsn - * gets changed or reset during that. Also, in streaming transaction cases, we - * don't skip receiving and spooling the changes since we decide whether or not + * gets changed or reset during that. Also, in streaming transaction cases (streaming = on), + * we don't skip receiving and spooling the changes since we decide whether or not * to skip applying the changes when starting to apply changes. The subskiplsn is * cleared after successfully skipping the transaction or applying non-empty * transaction. The latter prevents the mistakenly specified subskiplsn from - * being left. + * being left. Note that we cannot skip the streaming transactions when using + * apply background workers because we cannot get the finish LSN before + * applying the changes. */ static XLogRecPtr skip_xact_finish_lsn = InvalidXLogRecPtr; #define is_skipping_changes() (unlikely(!XLogRecPtrIsInvalid(skip_xact_finish_lsn))) @@ -324,9 +344,6 @@ static void maybe_reread_subscription(void); static void DisableSubscriptionAndExit(void); -/* prototype needed because of stream_commit */ -static void apply_dispatch(StringInfo s); - static void apply_handle_commit_internal(LogicalRepCommitData *commit_data); static void apply_handle_insert_internal(ApplyExecutionData *edata, ResultRelInfo *relinfo, @@ -359,7 +376,6 @@ static void stop_skipping_changes(void); static void clear_subscription_skip_lsn(XLogRecPtr finish_lsn); /* Functions for apply error callback */ -static void apply_error_callback(void *arg); static inline void set_apply_error_context_xact(TransactionId xid, XLogRecPtr lsn); static inline void reset_apply_error_context_info(void); @@ -426,40 +442,85 @@ end_replication_step(void) } /* - * Handle streamed transactions. + * Handle streamed transactions for both the main apply worker and the apply + * background workers. + * + * In streaming case (receiving a block of streamed transaction), for + * SUBSTREAM_ON mode, simply redirect it to a file for the proper toplevel + * transaction, and for SUBSTREAM_PARALLEL mode, send the changes to apply + * background workers (LOGICAL_REP_MSG_RELATION or LOGICAL_REP_MSG_TYPE changes + * will also be applied in main apply worker). * - * If in streaming mode (receiving a block of streamed transaction), we - * simply redirect it to a file for the proper toplevel transaction. + * For non-streamed transactions, returns false; + * For streamed transactions, returns true if in main apply worker, false + * otherwise. * - * Returns true for streamed transactions, false otherwise (regular mode). + * Exception: When the main apply worker is applying streaming transactions in + * parallel mode (e.g. when addressing LOGICAL_REP_MSG_RELATION or + * LOGICAL_REP_MSG_TYPE changes), then return false. */ static bool handle_streamed_transaction(LogicalRepMsgType action, StringInfo s) { - TransactionId xid; + TransactionId current_xid = InvalidTransactionId; - /* not in streaming mode */ - if (!in_streamed_transaction) + /* Not in streaming mode */ + if (!(in_streamed_transaction || am_apply_bgworker())) return false; - Assert(stream_fd != NULL); Assert(TransactionIdIsValid(stream_xid)); /* * We should have received XID of the subxact as the first part of the * message, so extract it. */ - xid = pq_getmsgint(s, 4); + current_xid = pq_getmsgint(s, 4); - if (!TransactionIdIsValid(xid)) + if (!TransactionIdIsValid(current_xid)) ereport(ERROR, (errcode(ERRCODE_PROTOCOL_VIOLATION), errmsg_internal("invalid transaction ID in streamed replication transaction"))); - /* Add the new subxact to the array (unless already there). */ - subxact_info_add(xid); + if (am_apply_bgworker()) + { + /* Define a savepoint for a subxact if needed. */ + apply_bgworker_subxact_info_add(current_xid); + + return false; + } + + if (apply_bgworker_active()) + { + /* + * This is the main apply worker, but there is an apply background + * worker, so apply the changes of this transaction in that background + * worker. Pass the data to the worker. + */ + apply_bgworker_send_data(stream_apply_worker, s->len, s->data); + nchanges += 1; + + /* + * XXX The publisher side doesn't always send relation/type update + * messages after the streaming transaction, so also update the + * relation/type in main apply worker here. See function + * cleanup_rel_sync_cache. + */ + if (action == LOGICAL_REP_MSG_RELATION || + action == LOGICAL_REP_MSG_TYPE) + return false; + + return true; + } + + /* + * This is the main apply worker, but there is no apply background worker, + * so write to temporary files and apply when the final commit arrives. + * + * Add the new subxact to the array (unless already there). + */ + subxact_info_add(current_xid); - /* write the change to the current file */ + /* Write the change to the current file */ stream_write_change(action, s); return true; @@ -844,6 +905,9 @@ apply_handle_commit(StringInfo s) apply_handle_commit_internal(&commit_data); + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); + /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(commit_data.end_lsn); @@ -898,7 +962,9 @@ apply_handle_prepare_internal(LogicalRepPreparedTxnData *prepare_data) * BeginTransactionBlock is necessary to balance the EndTransactionBlock * called within the PrepareTransactionBlock below. */ - BeginTransactionBlock(); + if (!IsTransactionBlock()) + BeginTransactionBlock(); + CommitTransactionCommand(); /* Completes the preceding Begin command. */ /* @@ -950,6 +1016,9 @@ apply_handle_prepare(StringInfo s) in_remote_transaction = false; + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); + /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(prepare_data.end_lsn); @@ -1064,10 +1133,6 @@ apply_handle_rollback_prepared(StringInfo s) /* * Handle STREAM PREPARE. - * - * Logic is in two parts: - * 1. Replay all the spooled operations - * 2. Mark the transaction as prepared */ static void apply_handle_stream_prepare(StringInfo s) @@ -1088,24 +1153,78 @@ apply_handle_stream_prepare(StringInfo s) logicalrep_read_stream_prepare(s, &prepare_data); set_apply_error_context_xact(prepare_data.xid, prepare_data.prepare_lsn); - elog(DEBUG1, "received prepare for streamed transaction %u", prepare_data.xid); + if (am_apply_bgworker()) + { + /* Mark the transaction as prepared. */ + apply_handle_prepare_internal(&prepare_data); - /* Replay all the spooled operations. */ - apply_spooled_messages(prepare_data.xid, prepare_data.prepare_lsn); + CommitTransactionCommand(); - /* Mark the transaction as prepared. */ - apply_handle_prepare_internal(&prepare_data); + pgstat_report_stat(false); - CommitTransactionCommand(); + list_free(subxactlist); + subxactlist = NIL; - pgstat_report_stat(false); + apply_bgworker_set_status(APPLY_BGWORKER_FINISHED); + } + else + { + /* This is the main apply worker. */ + ApplyBgworkerState *wstate = apply_bgworker_find(prepare_data.xid); - store_flush_position(prepare_data.end_lsn); + elog(DEBUG1, "received prepare for streamed transaction %u", + prepare_data.xid); + + /* + * Check if we are processing this transaction in an apply background + * worker and if so, send the changes to that worker. + */ + if (wstate) + { + /* Send STREAM PREPARE message to the apply background worker. */ + apply_bgworker_send_data(wstate, s->len, s->data); + + /* + * After sending the data to the apply background worker, wait for + * that worker to finish. This is necessary to maintain commit + * order which avoids failures due to transaction dependencies and + * deadlocks. + */ + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_FINISHED); + apply_bgworker_free(wstate); + + pgstat_report_stat(false); + store_flush_position(prepare_data.end_lsn); + } + else + { + /* + * The transaction has been serialized to file, so replay all the + * spooled operations. + */ + apply_spooled_messages(prepare_data.xid, prepare_data.prepare_lsn); + + /* Mark the transaction as prepared. */ + apply_handle_prepare_internal(&prepare_data); + + CommitTransactionCommand(); + + pgstat_report_stat(false); + + store_flush_position(prepare_data.end_lsn); + + in_remote_transaction = false; + + /* Unlink the files with serialized changes and subxact info. */ + stream_cleanup_files(MyLogicalRepWorker->subid, prepare_data.xid); + } + } in_remote_transaction = false; + stream_apply_worker = NULL; - /* unlink the files with serialized changes and subxact info. */ - stream_cleanup_files(MyLogicalRepWorker->subid, prepare_data.xid); + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(prepare_data.end_lsn); @@ -1155,15 +1274,6 @@ apply_handle_stream_start(StringInfo s) (errcode(ERRCODE_PROTOCOL_VIOLATION), errmsg_internal("duplicate STREAM START message"))); - /* - * Start a transaction on stream start, this transaction will be committed - * on the stream stop unless it is a tablesync worker in which case it - * will be committed after processing all the messages. We need the - * transaction for handling the buffile, used for serializing the - * streaming data and subxact info. - */ - begin_replication_step(); - /* notify handle methods we're processing a remote transaction */ in_streamed_transaction = true; @@ -1177,36 +1287,93 @@ apply_handle_stream_start(StringInfo s) set_apply_error_context_xact(stream_xid, InvalidXLogRecPtr); - /* - * Initialize the worker's stream_fileset if we haven't yet. This will be - * used for the entire duration of the worker so create it in a permanent - * context. We create this on the very first streaming message from any - * transaction and then use it for this and other streaming transactions. - * Now, we could create a fileset at the start of the worker as well but - * then we won't be sure that it will ever be used. - */ - if (MyLogicalRepWorker->stream_fileset == NULL) + if (am_apply_bgworker()) { - MemoryContext oldctx; - - oldctx = MemoryContextSwitchTo(ApplyContext); + /* + * Make sure the handle apply_dispatch methods are aware we're in a + * remote transaction. + */ + in_remote_transaction = true; - MyLogicalRepWorker->stream_fileset = palloc(sizeof(FileSet)); - FileSetInit(MyLogicalRepWorker->stream_fileset); + /* Begin the transaction. */ + AcceptInvalidationMessages(); + maybe_reread_subscription(); - MemoryContextSwitchTo(oldctx); + StartTransactionCommand(); + BeginTransactionBlock(); + CommitTransactionCommand(); } + else + { + /* + * This is the main apply worker. Check if there is any free apply + * background worker we can use to process this transaction. + */ + if (first_segment) + stream_apply_worker = apply_bgworker_start(stream_xid); + else + stream_apply_worker = apply_bgworker_find(stream_xid); - /* open the spool file for this transaction */ - stream_open_file(MyLogicalRepWorker->subid, stream_xid, first_segment); + if (stream_apply_worker) + { + /* + * If we have found a free worker or if we are already applying this + * transaction in an apply background worker, then we pass the data to + * that worker. + */ + if (first_segment) + apply_bgworker_send_data(stream_apply_worker, s->len, s->data); - /* if this is not the first segment, open existing subxact file */ - if (!first_segment) - subxact_info_read(MyLogicalRepWorker->subid, stream_xid); + nchanges = 0; + elog(DEBUG1, "starting streaming of xid %u", stream_xid); + } + else + { + /* + * Since no apply background worker is available for the first + * stream start, serialize all the changes of the transaction. + * + * Start a transaction on stream start, this transaction will be + * committed on the stream stop unless it is a tablesync worker in + * which case it will be committed after processing all the + * messages. We need the transaction for handling the buffile, + * used for serializing the streaming data and subxact info. + */ + begin_replication_step(); - pgstat_report_activity(STATE_RUNNING, NULL); + /* + * Initialize the worker's stream_fileset if we haven't yet. This will + * be used for the entire duration of the worker so create it in a + * permanent context. We create this on the very first streaming + * message from any transaction and then use it for this and other + * streaming transactions. Now, we could create a fileset at the start + * of the worker as well but then we won't be sure that it will ever + * be used. + */ + if (MyLogicalRepWorker->stream_fileset == NULL) + { + MemoryContext oldctx; - end_replication_step(); + oldctx = MemoryContextSwitchTo(ApplyContext); + + MyLogicalRepWorker->stream_fileset = palloc(sizeof(FileSet)); + FileSetInit(MyLogicalRepWorker->stream_fileset); + + MemoryContextSwitchTo(oldctx); + } + + /* Open the spool file for this transaction. */ + stream_open_file(MyLogicalRepWorker->subid, stream_xid, first_segment); + + /* If this is not the first segment, open existing subxact file. */ + if (!first_segment) + subxact_info_read(MyLogicalRepWorker->subid, stream_xid); + + end_replication_step(); + } + } + + pgstat_report_activity(STATE_RUNNING, NULL); } /* @@ -1220,53 +1387,52 @@ apply_handle_stream_stop(StringInfo s) (errcode(ERRCODE_PROTOCOL_VIOLATION), errmsg_internal("STREAM STOP message without STREAM START"))); - /* - * Close the file with serialized changes, and serialize information about - * subxacts for the toplevel transaction. - */ - subxact_info_write(MyLogicalRepWorker->subid, stream_xid); - stream_close_file(); + if (apply_bgworker_active()) + { + char action = LOGICAL_REP_MSG_STREAM_STOP; - /* We must be in a valid transaction state */ - Assert(IsTransactionState()); + apply_bgworker_send_data(stream_apply_worker, 1, &action); - /* Commit the per-stream transaction */ - CommitTransactionCommand(); + elog(DEBUG1, "stopped streaming of xid %u, %u changes streamed", stream_xid, nchanges); + } + else + { + /* + * Close the file with serialized changes, and serialize information + * about subxacts for the toplevel transaction. + */ + subxact_info_write(MyLogicalRepWorker->subid, stream_xid); + stream_close_file(); - in_streamed_transaction = false; + /* We must be in a valid transaction state */ + Assert(IsTransactionState()); - /* Reset per-stream context */ - MemoryContextReset(LogicalStreamingContext); + /* Commit the per-stream transaction */ + CommitTransactionCommand(); + + /* Reset per-stream context */ + MemoryContextReset(LogicalStreamingContext); + } + + in_streamed_transaction = false; + stream_apply_worker = NULL; pgstat_report_activity(STATE_IDLE, NULL); reset_apply_error_context_info(); } /* - * Handle STREAM abort message. + * Handle STREAM ABORT message when the transaction was spilled to disk. */ static void -apply_handle_stream_abort(StringInfo s) +serialize_stream_abort(TransactionId xid, TransactionId subxid) { - TransactionId xid; - TransactionId subxid; - - if (in_streamed_transaction) - ereport(ERROR, - (errcode(ERRCODE_PROTOCOL_VIOLATION), - errmsg_internal("STREAM ABORT message without STREAM STOP"))); - - logicalrep_read_stream_abort(s, &xid, &subxid); - /* * If the two XIDs are the same, it's in fact abort of toplevel xact, so * just delete the files with serialized info. */ if (xid == subxid) - { - set_apply_error_context_xact(xid, InvalidXLogRecPtr); stream_cleanup_files(MyLogicalRepWorker->subid, xid); - } else { /* @@ -1290,8 +1456,6 @@ apply_handle_stream_abort(StringInfo s) bool found = false; char path[MAXPGPATH]; - set_apply_error_context_xact(subxid, InvalidXLogRecPtr); - subidx = -1; begin_replication_step(); subxact_info_read(MyLogicalRepWorker->subid, xid); @@ -1316,7 +1480,6 @@ apply_handle_stream_abort(StringInfo s) cleanup_subxact_info(); end_replication_step(); CommitTransactionCommand(); - reset_apply_error_context_info(); return; } @@ -1339,6 +1502,143 @@ apply_handle_stream_abort(StringInfo s) end_replication_step(); CommitTransactionCommand(); } +} + +/* + * Handle STREAM ABORT message. + */ +static void +apply_handle_stream_abort(StringInfo s) +{ + TransactionId xid; + TransactionId subxid; + LogicalRepStreamAbortData abort_data; + bool read_abort_lsn = false; + + if (in_streamed_transaction) + ereport(ERROR, + (errcode(ERRCODE_PROTOCOL_VIOLATION), + errmsg_internal("STREAM ABORT message without STREAM STOP"))); + + /* Check whether the publisher sends abort_lsn and abort_time. */ + if (am_apply_bgworker()) + read_abort_lsn = MyParallelShared->server_version >= + LOGICALREP_PROTO_STREAM_PARALLEL_VERSION_NUM; + + logicalrep_read_stream_abort(s, &abort_data, read_abort_lsn); + + xid = abort_data.xid; + subxid = abort_data.subxid; + + set_apply_error_context_xact(subxid, abort_data.abort_lsn); + + if (am_apply_bgworker()) + { + elog(DEBUG1, "[Apply BGW #%u] aborting current transaction xid=%u, subxid=%u", + MyParallelShared->n, GetCurrentTransactionIdIfAny(), + GetCurrentSubTransactionId()); + + /* + * Update origin state so we can restart streaming from correct + * position in case of crash. + */ + if (read_abort_lsn) + { + replorigin_session_origin_lsn = abort_data.abort_lsn; + replorigin_session_origin_timestamp = abort_data.abort_time; + } + + /* + * If the two XIDs are the same, it's in fact abort of toplevel xact, + * so just free the subxactlist. + */ + if (subxid == xid) + { + AbortCurrentTransaction(); + + EndTransactionBlock(false); + CommitTransactionCommand(); + + in_remote_transaction = false; + pgstat_report_activity(STATE_IDLE, NULL); + + list_free(subxactlist); + subxactlist = NIL; + + apply_bgworker_set_status(APPLY_BGWORKER_FINISHED); + } + else + { + /* + * OK, so it's a subxact. Rollback to the savepoint. + * + * We also need to read the subxactlist, determine the offset + * tracked for the subxact, and truncate the list. + */ + int i; + bool found = false; + char spname[MAXPGPATH]; + + snprintf(spname, MAXPGPATH, "savepoint_for_xid_%u", subxid); + + elog(DEBUG1, "[Apply BGW #%u] rolling back to savepoint %s", + MyParallelShared->n, spname); + + for (i = list_length(subxactlist) - 1; i >= 0; i--) + { + xid = (TransactionId) list_nth_int(subxactlist, i); + if (xid == subxid) + { + found = true; + break; + } + } + + if (found) + { + RollbackToSavepoint(spname); + CommitTransactionCommand(); + subxactlist = list_truncate(subxactlist, i + 1); + } + + pgstat_report_activity(STATE_IDLEINTRANSACTION, NULL); + } + } + else + { + /* This is the main apply worker. */ + ApplyBgworkerState *wstate = apply_bgworker_find(xid); + + /* + * Check if we are processing this transaction in an apply background + * worker and if so, send the changes to that worker. + */ + if (wstate) + { + /* Send STREAM ABORT message to the apply background worker. */ + apply_bgworker_send_data(wstate, s->len, s->data); + + /* + * After sending the data to the apply background worker, wait for + * that worker to finish. This is necessary to maintain commit + * order which avoids failures due to transaction dependencies and + * deadlocks. + */ + if (subxid == xid) + { + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_FINISHED); + apply_bgworker_free(wstate); + } + } + else + { + /* + * We are in main apply worker and the transaction has been + * serialized to file. + */ + serialize_stream_abort(xid, subxid); + } + } reset_apply_error_context_info(); } @@ -1468,8 +1768,8 @@ apply_spooled_messages(TransactionId xid, XLogRecPtr lsn) static void apply_handle_stream_commit(StringInfo s) { - TransactionId xid; LogicalRepCommitData commit_data; + TransactionId xid; if (in_streamed_transaction) ereport(ERROR, @@ -1479,14 +1779,81 @@ apply_handle_stream_commit(StringInfo s) xid = logicalrep_read_stream_commit(s, &commit_data); set_apply_error_context_xact(xid, commit_data.commit_lsn); - elog(DEBUG1, "received commit for streamed transaction %u", xid); + if (am_apply_bgworker()) + { + /* + * Update origin state so we can restart streaming from correct + * position in case of crash. + */ + replorigin_session_origin_lsn = commit_data.end_lsn; + replorigin_session_origin_timestamp = commit_data.committime; - apply_spooled_messages(xid, commit_data.commit_lsn); + CommitTransactionCommand(); + EndTransactionBlock(false); + CommitTransactionCommand(); - apply_handle_commit_internal(&commit_data); + in_remote_transaction = false; + + pgstat_report_stat(false); + + list_free(subxactlist); + subxactlist = NIL; + + apply_bgworker_set_status(APPLY_BGWORKER_FINISHED); + } + else + { + /* This is the main apply worker. */ + ApplyBgworkerState *wstate = apply_bgworker_find(xid); + + elog(DEBUG1, "received commit for streamed transaction %u", xid); + + /* + * Check if we are processing this transaction in an apply background + * worker and if so, send the changes to that worker. + */ + if (wstate) + { + /* Send STREAM COMMIT message to the apply background worker. */ + apply_bgworker_send_data(wstate, s->len, s->data); + + /* + * After sending the data to the apply background worker, wait for + * that worker to finish. This is necessary to maintain commit + * order which avoids failures due to transaction dependencies and + * deadlocks. + */ + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_FINISHED); - /* unlink the files with serialized changes and subxact info */ - stream_cleanup_files(MyLogicalRepWorker->subid, xid); + pgstat_report_stat(false); + store_flush_position(commit_data.end_lsn); + stop_skipping_changes(); + + apply_bgworker_free(wstate); + + /* + * The transaction is either non-empty or skipped, so we clear the + * subskiplsn. + */ + clear_subscription_skip_lsn(commit_data.commit_lsn); + } + else + { + /* + * The transaction has been serialized to file, so replay all the + * spooled operations. + */ + apply_spooled_messages(xid, commit_data.commit_lsn); + + apply_handle_commit_internal(&commit_data); + + /* Unlink the files with serialized changes and subxact info. */ + stream_cleanup_files(MyLogicalRepWorker->subid, xid); + } + } + + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(commit_data.end_lsn); @@ -2467,7 +2834,7 @@ apply_handle_truncate(StringInfo s) /* * Logical replication protocol message dispatcher. */ -static void +void apply_dispatch(StringInfo s) { LogicalRepMsgType action = pq_getmsgbyte(s); @@ -2636,6 +3003,10 @@ store_flush_position(XLogRecPtr remote_lsn) { FlushPosition *flushpos; + /* Skip if not the main apply worker */ + if (am_apply_bgworker()) + return; + /* Need to do this in permanent context */ MemoryContextSwitchTo(ApplyContext); @@ -2650,7 +3021,7 @@ store_flush_position(XLogRecPtr remote_lsn) /* Update statistics of the worker. */ -static void +void UpdateWorkerStats(XLogRecPtr last_lsn, TimestampTz send_time, bool reply) { MyLogicalRepWorker->last_lsn = last_lsn; @@ -2812,6 +3183,9 @@ LogicalRepApplyLoop(XLogRecPtr last_received) AcceptInvalidationMessages(); maybe_reread_subscription(); + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); + /* Process any table synchronization changes. */ process_syncing_tables(last_received); } @@ -3113,7 +3487,7 @@ maybe_reread_subscription(void) /* * Callback from subscription syscache invalidation. */ -static void +void subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue) { MySubscriptionValid = false; @@ -3709,7 +4083,7 @@ ApplyWorkerMain(Datum main_arg) originid = replorigin_by_name(originname, true); if (!OidIsValid(originid)) originid = replorigin_create(originname); - replorigin_session_setup(originid); + replorigin_session_setup(originid, true); replorigin_session_origin = originid; origin_startpos = replorigin_session_get_progress(false); CommitTransactionCommand(); @@ -3750,13 +4124,14 @@ ApplyWorkerMain(Datum main_arg) server_version = walrcv_server_version(LogRepWorkerWalRcvConn); options.proto.logical.proto_version = + server_version >= 160000 ? LOGICALREP_PROTO_STREAM_PARALLEL_VERSION_NUM : server_version >= 150000 ? LOGICALREP_PROTO_TWOPHASE_VERSION_NUM : server_version >= 140000 ? LOGICALREP_PROTO_STREAM_VERSION_NUM : LOGICALREP_PROTO_VERSION_NUM; options.proto.logical.publication_names = MySubscription->publications; options.proto.logical.binary = MySubscription->binary; - options.proto.logical.streaming = MySubscription->stream; + options.proto.logical.streaming = (MySubscription->stream != SUBSTREAM_OFF); options.proto.logical.twophase = false; if (!am_tablesync_worker()) @@ -3914,7 +4289,8 @@ clear_subscription_skip_lsn(XLogRecPtr finish_lsn) XLogRecPtr myskiplsn = MySubscription->skiplsn; bool started_tx = false; - if (likely(XLogRecPtrIsInvalid(myskiplsn))) + if (likely(XLogRecPtrIsInvalid(myskiplsn)) || + am_apply_bgworker()) return; if (!IsTransactionState()) @@ -3986,7 +4362,7 @@ clear_subscription_skip_lsn(XLogRecPtr finish_lsn) } /* Error callback to give more context info about the change being applied */ -static void +void apply_error_callback(void *arg) { ApplyErrorCallbackArg *errarg = &apply_error_callback_arg; @@ -4014,23 +4390,47 @@ apply_error_callback(void *arg) errarg->remote_xid, LSN_FORMAT_ARGS(errarg->finish_lsn)); } - else if (errarg->remote_attnum < 0) - errcontext("processing remote data for replication origin \"%s\" during \"%s\" for replication target relation \"%s.%s\" in transaction %u finished at %X/%X", - errarg->origin_name, - logicalrep_message_type(errarg->command), - errarg->rel->remoterel.nspname, - errarg->rel->remoterel.relname, - errarg->remote_xid, - LSN_FORMAT_ARGS(errarg->finish_lsn)); else - errcontext("processing remote data for replication origin \"%s\" during \"%s\" for replication target relation \"%s.%s\" column \"%s\" in transaction %u finished at %X/%X", - errarg->origin_name, - logicalrep_message_type(errarg->command), - errarg->rel->remoterel.nspname, - errarg->rel->remoterel.relname, - errarg->rel->remoterel.attnames[errarg->remote_attnum], - errarg->remote_xid, - LSN_FORMAT_ARGS(errarg->finish_lsn)); + { + if (errarg->remote_attnum < 0) + { + if (XLogRecPtrIsInvalid(errarg->finish_lsn)) + errcontext("processing remote data for replication origin \"%s\" during \"%s\" for replication target relation \"%s.%s\" in transaction %u", + errarg->origin_name, + logicalrep_message_type(errarg->command), + errarg->rel->remoterel.nspname, + errarg->rel->remoterel.relname, + errarg->remote_xid); + else + errcontext("processing remote data for replication origin \"%s\" during \"%s\" for replication target relation \"%s.%s\" in transaction %u finished at %X/%X", + errarg->origin_name, + logicalrep_message_type(errarg->command), + errarg->rel->remoterel.nspname, + errarg->rel->remoterel.relname, + errarg->remote_xid, + LSN_FORMAT_ARGS(errarg->finish_lsn)); + } + else + { + if (XLogRecPtrIsInvalid(errarg->finish_lsn)) + errcontext("processing remote data for replication origin \"%s\" during \"%s\" for replication target relation \"%s.%s\" column \"%s\" in transaction %u", + errarg->origin_name, + logicalrep_message_type(errarg->command), + errarg->rel->remoterel.nspname, + errarg->rel->remoterel.relname, + errarg->rel->remoterel.attnames[errarg->remote_attnum], + errarg->remote_xid); + else + errcontext("processing remote data for replication origin \"%s\" during \"%s\" for replication target relation \"%s.%s\" column \"%s\" in transaction %u finished at %X/%X", + errarg->origin_name, + logicalrep_message_type(errarg->command), + errarg->rel->remoterel.nspname, + errarg->rel->remoterel.relname, + errarg->rel->remoterel.attnames[errarg->remote_attnum], + errarg->remote_xid, + LSN_FORMAT_ARGS(errarg->finish_lsn)); + } + } } /* Set transaction information of apply error callback */ diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c index ba8a24d099..de29dc6da9 100644 --- a/src/backend/replication/pgoutput/pgoutput.c +++ b/src/backend/replication/pgoutput/pgoutput.c @@ -1818,6 +1818,9 @@ pgoutput_stream_abort(struct LogicalDecodingContext *ctx, XLogRecPtr abort_lsn) { ReorderBufferTXN *toptxn; + PGOutputData *data = (PGOutputData *) ctx->output_plugin_private; + bool write_abort_lsn = (data->protocol_version >= + LOGICALREP_PROTO_STREAM_PARALLEL_VERSION_NUM); /* * The abort should happen outside streaming block, even for streamed @@ -1831,7 +1834,8 @@ pgoutput_stream_abort(struct LogicalDecodingContext *ctx, Assert(rbtxn_is_streamed(toptxn)); OutputPluginPrepareWrite(ctx, true); - logicalrep_write_stream_abort(ctx->out, toptxn->xid, txn->xid); + logicalrep_write_stream_abort(ctx->out, toptxn->xid, txn, abort_lsn, + write_abort_lsn); OutputPluginWrite(ctx, true); cleanup_rel_sync_cache(toptxn->xid, false); diff --git a/src/backend/utils/activity/wait_event.c b/src/backend/utils/activity/wait_event.c index da57a93034..2e146fe087 100644 --- a/src/backend/utils/activity/wait_event.c +++ b/src/backend/utils/activity/wait_event.c @@ -388,6 +388,9 @@ pgstat_get_wait_ipc(WaitEventIPC w) case WAIT_EVENT_HASH_GROW_BUCKETS_REINSERT: event_name = "HashGrowBucketsReinsert"; break; + case WAIT_EVENT_LOGICAL_APPLY_WORKER_STATE_CHANGE: + event_name = "LogicalApplyWorkerStateChange"; + break; case WAIT_EVENT_LOGICAL_SYNC_DATA: event_name = "LogicalSyncData"; break; diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c index 0328029d43..4284bcbcd1 100644 --- a/src/backend/utils/misc/guc.c +++ b/src/backend/utils/misc/guc.c @@ -3220,6 +3220,18 @@ static struct config_int ConfigureNamesInt[] = NULL, NULL, NULL }, + { + {"max_apply_bgworkers_per_subscription", + PGC_SIGHUP, + REPLICATION_SUBSCRIBERS, + gettext_noop("Maximum number of apply background workers per subscription."), + NULL, + }, + &max_apply_bgworkers_per_subscription, + 2, 0, MAX_BACKENDS, + NULL, NULL, NULL + }, + { {"log_rotation_age", PGC_SIGHUP, LOGGING_WHERE, gettext_noop("Sets the amount of time to wait before forcing " diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample index b4bc06e5f5..ad18710af4 100644 --- a/src/backend/utils/misc/postgresql.conf.sample +++ b/src/backend/utils/misc/postgresql.conf.sample @@ -360,6 +360,7 @@ #max_logical_replication_workers = 4 # taken from max_worker_processes # (change requires restart) #max_sync_workers_per_subscription = 2 # taken from max_logical_replication_workers +#max_apply_bgworkers_per_subscription = 2 # taken from max_logical_replication_workers #------------------------------------------------------------------------------ diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index e4fdb6b75b..9099fe5f74 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -4456,7 +4456,7 @@ getSubscriptions(Archive *fout) if (fout->remoteVersion >= 140000) appendPQExpBufferStr(query, " s.substream,\n"); else - appendPQExpBufferStr(query, " false AS substream,\n"); + appendPQExpBufferStr(query, " 'f' AS substream,\n"); if (fout->remoteVersion >= 150000) appendPQExpBufferStr(query, @@ -4586,8 +4586,10 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo) if (strcmp(subinfo->subbinary, "t") == 0) appendPQExpBufferStr(query, ", binary = true"); - if (strcmp(subinfo->substream, "f") != 0) + if (strcmp(subinfo->substream, "t") == 0) appendPQExpBufferStr(query, ", streaming = on"); + else if (strcmp(subinfo->substream, "p") == 0) + appendPQExpBufferStr(query, ", streaming = parallel"); if (strcmp(subinfo->subtwophasestate, two_phase_disabled) != 0) appendPQExpBufferStr(query, ", two_phase = on"); diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h index d1260f590c..d54540f5f5 100644 --- a/src/include/catalog/pg_subscription.h +++ b/src/include/catalog/pg_subscription.h @@ -68,7 +68,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW bool subbinary; /* True if the subscription wants the * publisher to send data in binary */ - bool substream; /* Stream in-progress transactions. */ + char substream; /* Stream in-progress transactions. + * See SUBSTREAM_xxx constants. */ char subtwophasestate; /* Stream two-phase transactions */ @@ -109,7 +110,8 @@ typedef struct Subscription bool enabled; /* Indicates if the subscription is enabled */ bool binary; /* Indicates if the subscription wants data in * binary format */ - bool stream; /* Allow streaming in-progress transactions. */ + char stream; /* Allow streaming in-progress transactions. + * See SUBSTREAM_xxx constants. */ char twophasestate; /* Allow streaming two-phase transactions */ bool disableonerr; /* Indicates if the subscription should be * automatically disabled if a worker error @@ -120,6 +122,21 @@ typedef struct Subscription List *publications; /* List of publication names to subscribe to */ } Subscription; +/* Disallow streaming in-progress transactions */ +#define SUBSTREAM_OFF 'f' + +/* + * Streaming in-progress transactions are written to a temporary file and + * applied only after the transaction is committed on upstream. + */ +#define SUBSTREAM_ON 't' + +/* + * Streaming in-progress transactions are applied immediately via a background + * worker + */ +#define SUBSTREAM_PARALLEL 'p' + extern Subscription *GetSubscription(Oid subid, bool missing_ok); extern void FreeSubscription(Subscription *sub); extern void DisableSubscription(Oid subid); diff --git a/src/include/replication/logicallauncher.h b/src/include/replication/logicallauncher.h index f1e2821e25..ac8ef94381 100644 --- a/src/include/replication/logicallauncher.h +++ b/src/include/replication/logicallauncher.h @@ -14,6 +14,7 @@ extern PGDLLIMPORT int max_logical_replication_workers; extern PGDLLIMPORT int max_sync_workers_per_subscription; +extern PGDLLIMPORT int max_apply_bgworkers_per_subscription; extern void ApplyLauncherRegister(void); extern void ApplyLauncherMain(Datum main_arg); diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h index a771ab8ff3..eb0fd24fd8 100644 --- a/src/include/replication/logicalproto.h +++ b/src/include/replication/logicalproto.h @@ -32,12 +32,17 @@ * * LOGICALREP_PROTO_TWOPHASE_VERSION_NUM is the minimum protocol version with * support for two-phase commit decoding (at prepare time). Introduced in PG15. + * + * LOGICALREP_PROTO_STREAM_PARALLEL_VERSION_NUM is the minimum protocol version + * with support for streaming large transactions using apply background + * workers. Introduced in PG16. */ #define LOGICALREP_PROTO_MIN_VERSION_NUM 1 #define LOGICALREP_PROTO_VERSION_NUM 1 #define LOGICALREP_PROTO_STREAM_VERSION_NUM 2 #define LOGICALREP_PROTO_TWOPHASE_VERSION_NUM 3 -#define LOGICALREP_PROTO_MAX_VERSION_NUM LOGICALREP_PROTO_TWOPHASE_VERSION_NUM +#define LOGICALREP_PROTO_STREAM_PARALLEL_VERSION_NUM 4 +#define LOGICALREP_PROTO_MAX_VERSION_NUM LOGICALREP_PROTO_STREAM_PARALLEL_VERSION_NUM /* * Logical message types @@ -175,6 +180,17 @@ typedef struct LogicalRepRollbackPreparedTxnData char gid[GIDSIZE]; } LogicalRepRollbackPreparedTxnData; +/* + * Transaction protocol information for stream abort. + */ +typedef struct LogicalRepStreamAbortData +{ + TransactionId xid; + TransactionId subxid; + XLogRecPtr abort_lsn; + TimestampTz abort_time; +} LogicalRepStreamAbortData; + extern void logicalrep_write_begin(StringInfo out, ReorderBufferTXN *txn); extern void logicalrep_read_begin(StringInfo in, LogicalRepBeginData *begin_data); @@ -246,9 +262,12 @@ extern void logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn extern TransactionId logicalrep_read_stream_commit(StringInfo out, LogicalRepCommitData *commit_data); extern void logicalrep_write_stream_abort(StringInfo out, TransactionId xid, - TransactionId subxid); -extern void logicalrep_read_stream_abort(StringInfo in, TransactionId *xid, - TransactionId *subxid); + ReorderBufferTXN *txn, + XLogRecPtr abort_lsn, + bool write_abort_lsn); +extern void logicalrep_read_stream_abort(StringInfo in, + LogicalRepStreamAbortData *abort_data, + bool read_abort_lsn); extern char *logicalrep_message_type(LogicalRepMsgType action); #endif /* LOGICAL_PROTO_H */ diff --git a/src/include/replication/logicalworker.h b/src/include/replication/logicalworker.h index cd1b6e8afc..6a1af7f13c 100644 --- a/src/include/replication/logicalworker.h +++ b/src/include/replication/logicalworker.h @@ -13,6 +13,7 @@ #define LOGICALWORKER_H extern void ApplyWorkerMain(Datum main_arg); +extern void ApplyBgworkerMain(Datum main_arg); extern bool IsLogicalWorker(void); diff --git a/src/include/replication/origin.h b/src/include/replication/origin.h index 2d1b5e5c28..c7389b40a7 100644 --- a/src/include/replication/origin.h +++ b/src/include/replication/origin.h @@ -53,7 +53,7 @@ extern XLogRecPtr replorigin_get_progress(RepOriginId node, bool flush); extern void replorigin_session_advance(XLogRecPtr remote_commit, XLogRecPtr local_commit); -extern void replorigin_session_setup(RepOriginId node); +extern void replorigin_session_setup(RepOriginId node, bool must_acquire); extern void replorigin_session_reset(void); extern XLogRecPtr replorigin_session_get_progress(bool flush); diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h index d109d0baed..d2a80d79e5 100644 --- a/src/include/replication/reorderbuffer.h +++ b/src/include/replication/reorderbuffer.h @@ -301,6 +301,7 @@ typedef struct ReorderBufferTXN { TimestampTz commit_time; TimestampTz prepare_time; + TimestampTz abort_time; } xact_time; /* @@ -647,9 +648,11 @@ extern void ReorderBufferFinishPrepared(ReorderBuffer *rb, TransactionId xid, extern void ReorderBufferAssignChild(ReorderBuffer *, TransactionId, TransactionId, XLogRecPtr commit_lsn); extern void ReorderBufferCommitChild(ReorderBuffer *, TransactionId, TransactionId, XLogRecPtr commit_lsn, XLogRecPtr end_lsn); -extern void ReorderBufferAbort(ReorderBuffer *, TransactionId, XLogRecPtr lsn); +extern void ReorderBufferAbort(ReorderBuffer *, TransactionId, XLogRecPtr lsn, + TimestampTz abort_time); extern void ReorderBufferAbortOld(ReorderBuffer *, TransactionId xid); -extern void ReorderBufferForget(ReorderBuffer *, TransactionId, XLogRecPtr lsn); +extern void ReorderBufferForget(ReorderBuffer *, TransactionId, XLogRecPtr lsn, + TimestampTz abort_time); extern void ReorderBufferInvalidate(ReorderBuffer *, TransactionId, XLogRecPtr lsn); extern void ReorderBufferSetBaseSnapshot(ReorderBuffer *, TransactionId, XLogRecPtr lsn, struct SnapshotData *snap); diff --git a/src/include/replication/worker_internal.h b/src/include/replication/worker_internal.h index 901845abc2..a3560d4904 100644 --- a/src/include/replication/worker_internal.h +++ b/src/include/replication/worker_internal.h @@ -17,8 +17,11 @@ #include "access/xlogdefs.h" #include "catalog/pg_subscription.h" #include "datatype/timestamp.h" +#include "replication/logicalrelation.h" #include "storage/fileset.h" #include "storage/lock.h" +#include "storage/shm_mq.h" +#include "storage/shm_toc.h" #include "storage/spin.h" @@ -60,6 +63,9 @@ typedef struct LogicalRepWorker */ FileSet *stream_fileset; + /* Indicates if this slot is used for an apply background worker. */ + bool subworker; + /* Stats. */ XLogRecPtr last_lsn; TimestampTz last_send_time; @@ -68,8 +74,68 @@ typedef struct LogicalRepWorker TimestampTz reply_time; } LogicalRepWorker; +/* Struct for saving and restoring apply errcontext information */ +typedef struct ApplyErrorCallbackArg +{ + LogicalRepMsgType command; /* 0 if invalid */ + LogicalRepRelMapEntry *rel; + + /* Remote node information */ + int remote_attnum; /* -1 if invalid */ + TransactionId remote_xid; + XLogRecPtr finish_lsn; + char *origin_name; +} ApplyErrorCallbackArg; + +/* + * Status of apply background worker. + */ +typedef enum ApplyBgworkerStatus +{ + APPLY_BGWORKER_BUSY = 0, /* assigned to a transaction */ + APPLY_BGWORKER_FINISHED, /* transaction is completed */ + APPLY_BGWORKER_EXIT /* exit */ +} ApplyBgworkerStatus; + +/* + * Struct for sharing information between apply main and apply background + * workers. + */ +typedef struct ApplyBgworkerShared +{ + slock_t mutex; + + /* Status of apply background worker. */ + ApplyBgworkerStatus status; + + /* server version of publisher. */ + uint32 server_version; + + TransactionId stream_xid; + uint32 n; /* id of apply background worker */ +} ApplyBgworkerShared; + +/* + * Struct for maintaining an apply background worker. + */ +typedef struct ApplyBgworkerState +{ + shm_mq_handle *mq_handle; + dsm_segment *dsm_seg; + ApplyBgworkerShared volatile *shared; +} ApplyBgworkerState; + /* Main memory context for apply worker. Permanent during worker lifetime. */ extern PGDLLIMPORT MemoryContext ApplyContext; +extern PGDLLIMPORT MemoryContext ApplyMessageContext; + +extern PGDLLIMPORT ApplyErrorCallbackArg apply_error_callback_arg; + +extern PGDLLIMPORT bool MySubscriptionValid; + +extern PGDLLIMPORT volatile ApplyBgworkerShared *MyParallelShared; + +extern PGDLLIMPORT List *subxactlist; /* libpqreceiver connection */ extern PGDLLIMPORT struct WalReceiverConn *LogRepWorkerWalRcvConn; @@ -79,18 +145,22 @@ extern PGDLLIMPORT Subscription *MySubscription; extern PGDLLIMPORT LogicalRepWorker *MyLogicalRepWorker; extern PGDLLIMPORT bool in_remote_transaction; +extern PGDLLIMPORT bool in_streamed_transaction; +extern PGDLLIMPORT TransactionId stream_xid; extern void logicalrep_worker_attach(int slot); extern LogicalRepWorker *logicalrep_worker_find(Oid subid, Oid relid, bool only_running); extern List *logicalrep_workers_find(Oid subid, bool only_running); -extern void logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, - Oid userid, Oid relid); +extern bool logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, + Oid userid, Oid relid, + dsm_handle subworker_dsm); extern void logicalrep_worker_stop(Oid subid, Oid relid); extern void logicalrep_worker_wakeup(Oid subid, Oid relid); extern void logicalrep_worker_wakeup_ptr(LogicalRepWorker *worker); extern int logicalrep_sync_worker_count(Oid subid); +extern int logicalrep_apply_bgworker_count(Oid subid); extern void ReplicationOriginNameForTablesync(Oid suboid, Oid relid, char *originname, int szorgname); @@ -103,10 +173,38 @@ extern void process_syncing_tables(XLogRecPtr current_lsn); extern void invalidate_syncing_table_states(Datum arg, int cacheid, uint32 hashvalue); +extern void UpdateWorkerStats(XLogRecPtr last_lsn, TimestampTz send_time, + bool reply); + +extern void apply_dispatch(StringInfo s); + +/* Function for apply error callback */ +extern void apply_error_callback(void *arg); + +extern void subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue); + +/* Apply background worker setup and interactions */ +extern ApplyBgworkerState *apply_bgworker_start(TransactionId xid); +extern ApplyBgworkerState *apply_bgworker_find(TransactionId xid); +extern void apply_bgworker_wait_for(ApplyBgworkerState *wstate, + ApplyBgworkerStatus wait_for_status); +extern void apply_bgworker_send_data(ApplyBgworkerState *wstate, Size nbytes, + const void *data); +extern void apply_bgworker_free(ApplyBgworkerState *wstate); +extern void apply_bgworker_check_status(void); +extern void apply_bgworker_set_status(ApplyBgworkerStatus status); +extern void apply_bgworker_subxact_info_add(TransactionId current_xid); + static inline bool am_tablesync_worker(void) { return OidIsValid(MyLogicalRepWorker->relid); } +static inline bool +am_apply_bgworker(void) +{ + return MyLogicalRepWorker->subworker; +} + #endif /* WORKER_INTERNAL_H */ diff --git a/src/include/utils/wait_event.h b/src/include/utils/wait_event.h index c3ade01120..e35f199fd4 100644 --- a/src/include/utils/wait_event.h +++ b/src/include/utils/wait_event.h @@ -105,6 +105,7 @@ typedef enum WAIT_EVENT_HASH_GROW_BUCKETS_ALLOCATE, WAIT_EVENT_HASH_GROW_BUCKETS_ELECT, WAIT_EVENT_HASH_GROW_BUCKETS_REINSERT, + WAIT_EVENT_LOGICAL_APPLY_WORKER_STATE_CHANGE, WAIT_EVENT_LOGICAL_SYNC_DATA, WAIT_EVENT_LOGICAL_SYNC_STATE_CHANGE, WAIT_EVENT_MQ_INTERNAL, diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out index 5db7146e06..919266ae06 100644 --- a/src/test/regress/expected/subscription.out +++ b/src/test/regress/expected/subscription.out @@ -197,7 +197,7 @@ ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); DROP SUBSCRIPTION regress_testsub; -- fail - streaming must be boolean CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = foo); -ERROR: streaming requires a Boolean value +ERROR: streaming requires a Boolean value or "parallel" -- now it works CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = true); WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 34a76ceb60..4137dc77b4 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -116,6 +116,10 @@ Append AppendPath AppendRelInfo AppendState +ApplyBgworkerEntry +ApplyBgworkerShared +ApplyBgworkerState +ApplyBgworkerStatus ApplyErrorCallbackArg ApplyExecutionData ApplySubXactData @@ -1485,6 +1489,7 @@ LogicalRepRelId LogicalRepRelMapEntry LogicalRepRelation LogicalRepRollbackPreparedTxnData +LogicalRepStreamAbortData LogicalRepTupleData LogicalRepTyp LogicalRepWorker -- 2.23.0.windows.1 [application/octet-stream] v18-0002-Test-streaming-parallel-option-in-tap-test.patch (69.1K, ../../OS3PR01MB62758A6AAED27B3A848CEB7A9E8F9@OS3PR01MB6275.jpnprd01.prod.outlook.com/3-v18-0002-Test-streaming-parallel-option-in-tap-test.patch) download | inline diff: From 3d9d950d2e4c4b6ad644c5baffa293b48d21cac1 Mon Sep 17 00:00:00 2001 From: "shiy.fnst" <[email protected]> Date: Fri, 13 May 2022 14:50:30 +0800 Subject: [PATCH v18 2/4] Test streaming parallel option in tap test Change all TAP tests using the SUBSCRIPTION "streaming" parameter, so they now test both 'on' and 'parallel' values. --- src/test/subscription/t/015_stream.pl | 199 ++++--- src/test/subscription/t/016_stream_subxact.pl | 119 +++-- src/test/subscription/t/017_stream_ddl.pl | 188 ++++--- .../t/018_stream_subxact_abort.pl | 195 ++++--- .../t/019_stream_subxact_ddl_abort.pl | 110 +++- .../subscription/t/022_twophase_cascade.pl | 363 +++++++------ .../subscription/t/023_twophase_stream.pl | 498 ++++++++++-------- 7 files changed, 1035 insertions(+), 637 deletions(-) diff --git a/src/test/subscription/t/015_stream.pl b/src/test/subscription/t/015_stream.pl index 6561b189de..0bdd234935 100644 --- a/src/test/subscription/t/015_stream.pl +++ b/src/test/subscription/t/015_stream.pl @@ -8,6 +8,116 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" +# parameter so the same code can be run both for the streaming=on and +# streaming=parallel cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_parallel) = @_; + + # Interleave a pair of transactions, each exceeding the 64kB limit. + my $in = ''; + my $out = ''; + + my $timer = IPC::Run::timeout($PostgreSQL::Test::Utils::timeout_default); + + my $h = $node_publisher->background_psql('postgres', \$in, \$out, $timer, + on_error_stop => 0); + + # Check that a background worker starts if "streaming" parameter is + # specified as "parallel". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_parallel) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + $in .= q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + }; + $h->pump_nb; + + if ($is_parallel) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001, 9999) s(i); + DELETE FROM test_tab WHERE a > 5000; + COMMIT; + }); + + $in .= q{ + COMMIT; + \q + }; + $h->finish; # errors make the next test fail, so ignore them here + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), + 'check extra columns contain local defaults'); + + # Test the streaming in binary mode + $node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET (binary = on)"); + + # Insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001, 10000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + COMMIT; + }); + + $node_publisher->wait_for_catchup($appname); + + $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(6667|6667|6667), + 'check extra columns contain local defaults'); + + # Change the local values of the extra columns on the subscriber, + # update publisher, and check that subscriber retains the expected + # values. This is to ensure that non-streaming transactions behave + # properly after a streaming transaction. + $node_subscriber->safe_psql('postgres', + "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'" + ); + $node_publisher->safe_psql('postgres', + "UPDATE test_tab SET b = md5(a::text)"); + + $node_publisher->wait_for_catchup($appname); + + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab" + ); + is($result, qq(6667|6667|6667), + 'check extra columns contain locally changed data'); + + # Cleanup the test data + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE (a > 2)"); + $node_publisher->wait_for_catchup($appname); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -37,6 +147,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -54,82 +168,25 @@ my $result = "SELECT count(*), count(c), count(d = 999) FROM test_tab"); is($result, qq(2|2|2), 'check initial data was copied to subscriber'); -# Interleave a pair of transactions, each exceeding the 64kB limit. -my $in = ''; -my $out = ''; - -my $timer = IPC::Run::timeout($PostgreSQL::Test::Utils::timeout_default); - -my $h = $node_publisher->background_psql('postgres', \$in, \$out, $timer, - on_error_stop => 0); - -$in .= q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -}; -$h->pump_nb; - -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001, 9999) s(i); -DELETE FROM test_tab WHERE a > 5000; -COMMIT; -}); - -$in .= q{ -COMMIT; -\q -}; -$h->finish; # errors make the next test fail, so ignore them here - -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), 'check extra columns contain local defaults'); - -# Test the streaming in binary mode -$node_subscriber->safe_psql('postgres', - "ALTER SUBSCRIPTION tap_sub SET (binary = on)"); - -# Insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001, 10000) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -COMMIT; -}); - -$node_publisher->wait_for_catchup($appname); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(6667|6667|6667), 'check extra columns contain local defaults'); +###################################### +# Test using streaming mode 'parallel' +###################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" +); -# Change the local values of the extra columns on the subscriber, -# update publisher, and check that subscriber retains the expected -# values. This is to ensure that non-streaming transactions behave -# properly after a streaming transaction. $node_subscriber->safe_psql('postgres', - "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'" -); -$node_publisher->safe_psql('postgres', - "UPDATE test_tab SET b = md5(a::text)"); + "ALTER SUBSCRIPTION tap_sub SET(streaming = parallel, binary = off)"); -$node_publisher->wait_for_catchup($appname); +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab" -); -is($result, qq(6667|6667|6667), - 'check extra columns contain locally changed data'); +test_streaming($node_publisher, $node_subscriber, $appname, 1); $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/016_stream_subxact.pl b/src/test/subscription/t/016_stream_subxact.pl index f27f1694f2..45429dddba 100644 --- a/src/test/subscription/t/016_stream_subxact.pl +++ b/src/test/subscription/t/016_stream_subxact.pl @@ -8,6 +8,72 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" +# parameter so the same code can be run both for the streaming=on and +# streaming=parallel cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_parallel) = @_; + + # Check that a background worker starts if "streaming" parameter is + # specified as "parallel". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_parallel) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + # Insert, update and delete enough rows to exceed 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series( 3, 500) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501, 1000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001, 1500) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + SAVEPOINT s3; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501, 2000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + SAVEPOINT s4; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001, 2500) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + COMMIT; + }); + + if ($is_parallel) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(1667|1667|1667), + 'check data was copied to subscriber in streaming mode and extra columns contain local defaults' + ); + + # Cleanup the test data + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE (a > 2)"); + $node_publisher->wait_for_catchup($appname); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -37,6 +103,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -54,41 +124,26 @@ my $result = "SELECT count(*), count(c), count(d = 999) FROM test_tab"); is($result, qq(2|2|2), 'check initial data was copied to subscriber'); -# Insert, update and delete enough rows to exceed 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series( 3, 500) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501, 1000) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001, 1500) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -SAVEPOINT s3; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501, 2000) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -SAVEPOINT s4; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001, 2500) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -COMMIT; -}); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(1667|1667|1667), - 'check data was copied to subscriber in streaming mode and extra columns contain local defaults' +###################################### +# Test using streaming mode 'parallel' +###################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" ); +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = parallel)"); + +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; + +test_streaming($node_publisher, $node_subscriber, $appname, 1); + $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/017_stream_ddl.pl b/src/test/subscription/t/017_stream_ddl.pl index 0bce63b716..52dfef4780 100644 --- a/src/test/subscription/t/017_stream_ddl.pl +++ b/src/test/subscription/t/017_stream_ddl.pl @@ -8,6 +8,111 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" +# parameter so the same code can be run both for the streaming=on and +# streaming=parallel cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_parallel) = @_; + + # a small (non-streamed) transaction with DDL and DML + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab VALUES (3, md5(3::text)); + ALTER TABLE test_tab ADD COLUMN c INT; + SAVEPOINT s1; + INSERT INTO test_tab VALUES (4, md5(4::text), -4); + COMMIT; + }); + + # Check that a background worker starts if "streaming" parameter is + # specified as "parallel". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_parallel) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + # large (streamed) transaction with DDL and DML + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(5, 1000) s(i); + ALTER TABLE test_tab ADD COLUMN d INT; + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001, 2000) s(i); + COMMIT; + }); + + if ($is_parallel) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + # a small (non-streamed) transaction with DDL and DML + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab VALUES (2001, md5(2001::text), -2001, 2*2001); + ALTER TABLE test_tab ADD COLUMN e INT; + SAVEPOINT s1; + INSERT INTO test_tab VALUES (2002, md5(2002::text), -2002, 2*2002, -3*2002); + COMMIT; + }); + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d), count(e) FROM test_tab"); + is($result, qq(2002|1999|1002|1), + 'check data was copied to subscriber in streaming mode and extra columns contain local defaults' + ); + + # A large (streamed) transaction with DDL and DML. One of the DDL is performed + # after DML to ensure that we invalidate the schema sent for test_tab so that + # the next transaction has to send the schema again. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(2003,5000) s(i); + ALTER TABLE test_tab ADD COLUMN f INT; + COMMIT; + }); + + # A small transaction that won't get streamed. This is just to ensure that we + # send the schema again to reflect the last column added in the previous test. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i, 4*i FROM generate_series(5001,5005) s(i); + COMMIT; + }); + + $node_publisher->wait_for_catchup($appname); + + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d), count(e), count(f) FROM test_tab" + ); + is($result, qq(5005|5002|4005|3004|5), + 'check data was copied to subscriber for both streaming and non-streaming transactions' + ); + + # Cleanup the test data + $node_publisher->safe_psql( + 'postgres', q{ + DELETE FROM test_tab WHERE (a > 2); + ALTER TABLE test_tab DROP COLUMN c, DROP COLUMN d, DROP COLUMN e, DROP COLUMN f; + }); + $node_publisher->wait_for_catchup($appname); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -37,6 +142,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -54,76 +163,25 @@ my $result = "SELECT count(*), count(c), count(d = 999) FROM test_tab"); is($result, qq(2|0|0), 'check initial data was copied to subscriber'); -# a small (non-streamed) transaction with DDL and DML -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab VALUES (3, md5(3::text)); -ALTER TABLE test_tab ADD COLUMN c INT; -SAVEPOINT s1; -INSERT INTO test_tab VALUES (4, md5(4::text), -4); -COMMIT; -}); - -# large (streamed) transaction with DDL and DML -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(5, 1000) s(i); -ALTER TABLE test_tab ADD COLUMN d INT; -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001, 2000) s(i); -COMMIT; -}); - -# a small (non-streamed) transaction with DDL and DML -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab VALUES (2001, md5(2001::text), -2001, 2*2001); -ALTER TABLE test_tab ADD COLUMN e INT; -SAVEPOINT s1; -INSERT INTO test_tab VALUES (2002, md5(2002::text), -2002, 2*2002, -3*2002); -COMMIT; -}); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d), count(e) FROM test_tab"); -is($result, qq(2002|1999|1002|1), - 'check data was copied to subscriber in streaming mode and extra columns contain local defaults' +###################################### +# Test using streaming mode 'parallel' +###################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" ); -# A large (streamed) transaction with DDL and DML. One of the DDL is performed -# after DML to ensure that we invalidate the schema sent for test_tab so that -# the next transaction has to send the schema again. -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(2003,5000) s(i); -ALTER TABLE test_tab ADD COLUMN f INT; -COMMIT; -}); - -# A small transaction that won't get streamed. This is just to ensure that we -# send the schema again to reflect the last column added in the previous test. -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i, 4*i FROM generate_series(5001,5005) s(i); -COMMIT; -}); +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = parallel)"); -$node_publisher->wait_for_catchup($appname); +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d), count(e), count(f) FROM test_tab"); -is($result, qq(5005|5002|4005|3004|5), - 'check data was copied to subscriber for both streaming and non-streaming transactions' -); +test_streaming($node_publisher, $node_subscriber, $appname, 1); $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/018_stream_subxact_abort.pl b/src/test/subscription/t/018_stream_subxact_abort.pl index 7155442e76..68f0e4b0d1 100644 --- a/src/test/subscription/t/018_stream_subxact_abort.pl +++ b/src/test/subscription/t/018_stream_subxact_abort.pl @@ -8,6 +8,113 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" +# parameter so the same code can be run both for the streaming=on and +# streaming=parallel cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_parallel) = @_; + + # Check that a background worker starts if "streaming" parameter is + # specified as "parallel". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_parallel) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + # large (streamed) transaction with DDL, DML and ROLLBACKs + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i); + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501,1000) s(i); + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001,1500) s(i); + SAVEPOINT s3; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501,2000) s(i); + ROLLBACK TO s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001,2500) s(i); + ROLLBACK TO s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2501,3000) s(i); + SAVEPOINT s4; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3001,3500) s(i); + SAVEPOINT s5; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3501,4000) s(i); + COMMIT; + }); + + if ($is_parallel) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c) FROM test_tab"); + is($result, qq(2000|0), + 'check rollback to savepoint was reflected on subscriber and extra columns contain local defaults' + ); + + # large (streamed) transaction with subscriber receiving out of order + # subtransaction ROLLBACKs + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(4001,4500) s(i); + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001,5500) s(i); + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(6001,6500) s(i); + SAVEPOINT s3; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(7001,7500) s(i); + RELEASE s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(8001,8500) s(i); + ROLLBACK TO s1; + COMMIT; + }); + + $node_publisher->wait_for_catchup($appname); + + $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c) FROM test_tab"); + is($result, qq(2500|0), + 'check rollback to savepoint was reflected on subscriber'); + + # large (streamed) transaction with subscriber receiving rollback + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(8501,9000) s(i); + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(9001,9500) s(i); + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(9501,10000) s(i); + ROLLBACK; + }); + + $node_publisher->wait_for_catchup($appname); + + $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c) FROM test_tab"); + is($result, qq(2500|0), 'check rollback was reflected on subscriber'); + + # Cleanup the test data + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE (a > 2)"); + $node_publisher->wait_for_catchup($appname); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -36,6 +143,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -53,81 +164,25 @@ my $result = "SELECT count(*), count(c) FROM test_tab"); is($result, qq(2|0), 'check initial data was copied to subscriber'); -# large (streamed) transaction with DDL, DML and ROLLBACKs -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i); -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501,1000) s(i); -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001,1500) s(i); -SAVEPOINT s3; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501,2000) s(i); -ROLLBACK TO s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001,2500) s(i); -ROLLBACK TO s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2501,3000) s(i); -SAVEPOINT s4; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3001,3500) s(i); -SAVEPOINT s5; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3501,4000) s(i); -COMMIT; -}); - -$node_publisher->wait_for_catchup($appname); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c) FROM test_tab"); -is($result, qq(2000|0), - 'check rollback to savepoint was reflected on subscriber and extra columns contain local defaults' +###################################### +# Test using streaming mode 'parallel' +###################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" ); -# large (streamed) transaction with subscriber receiving out of order -# subtransaction ROLLBACKs -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(4001,4500) s(i); -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001,5500) s(i); -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(6001,6500) s(i); -SAVEPOINT s3; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(7001,7500) s(i); -RELEASE s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(8001,8500) s(i); -ROLLBACK TO s1; -COMMIT; -}); - -$node_publisher->wait_for_catchup($appname); +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = parallel)"); -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c) FROM test_tab"); -is($result, qq(2500|0), - 'check rollback to savepoint was reflected on subscriber'); - -# large (streamed) transaction with subscriber receiving rollback -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(8501,9000) s(i); -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(9001,9500) s(i); -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(9501,10000) s(i); -ROLLBACK; -}); +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c) FROM test_tab"); -is($result, qq(2500|0), 'check rollback was reflected on subscriber'); +test_streaming($node_publisher, $node_subscriber, $appname, 1); $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/019_stream_subxact_ddl_abort.pl b/src/test/subscription/t/019_stream_subxact_ddl_abort.pl index dbd0fca4d1..b276063721 100644 --- a/src/test/subscription/t/019_stream_subxact_ddl_abort.pl +++ b/src/test/subscription/t/019_stream_subxact_ddl_abort.pl @@ -9,6 +9,69 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" +# parameter so the same code can be run both for the streaming=on and +# streaming=parallel cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_parallel) = @_; + + # Check that a background worker starts if "streaming" parameter is + # specified as "parallel". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_parallel) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + # large (streamed) transaction with DDL, DML and ROLLBACKs + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i); + ALTER TABLE test_tab ADD COLUMN c INT; + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(501,1000) s(i); + ALTER TABLE test_tab ADD COLUMN d INT; + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001,1500) s(i); + ALTER TABLE test_tab ADD COLUMN e INT; + SAVEPOINT s3; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(1501,2000) s(i); + ALTER TABLE test_tab DROP COLUMN c; + ROLLBACK TO s1; + INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(501,1000) s(i); + COMMIT; + }); + + if ($is_parallel) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c) FROM test_tab"); + is($result, qq(1000|500), + 'check rollback to savepoint was reflected on subscriber and extra columns contain local defaults' + ); + + # Cleanup the test data + $node_publisher->safe_psql( + 'postgres', q{ + DELETE FROM test_tab WHERE (a > 2); + ALTER TABLE test_tab DROP COLUMN c; + }); + $node_publisher->wait_for_catchup($appname); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -37,6 +100,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -54,35 +121,26 @@ my $result = "SELECT count(*), count(c) FROM test_tab"); is($result, qq(2|0), 'check initial data was copied to subscriber'); -# large (streamed) transaction with DDL, DML and ROLLBACKs -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i); -ALTER TABLE test_tab ADD COLUMN c INT; -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(501,1000) s(i); -ALTER TABLE test_tab ADD COLUMN d INT; -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001,1500) s(i); -ALTER TABLE test_tab ADD COLUMN e INT; -SAVEPOINT s3; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(1501,2000) s(i); -ALTER TABLE test_tab DROP COLUMN c; -ROLLBACK TO s1; -INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(501,1000) s(i); -COMMIT; -}); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c) FROM test_tab"); -is($result, qq(1000|500), - 'check rollback to savepoint was reflected on subscriber and extra columns contain local defaults' +###################################### +# Test using streaming mode 'parallel' +###################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" ); +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = parallel)"); + +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; + +test_streaming($node_publisher, $node_subscriber, $appname, 1); + $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/022_twophase_cascade.pl b/src/test/subscription/t/022_twophase_cascade.pl index 7a797f37ba..0a4152d3be 100644 --- a/src/test/subscription/t/022_twophase_cascade.pl +++ b/src/test/subscription/t/022_twophase_cascade.pl @@ -11,6 +11,208 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" parameter +# so the same code can be run both for the streaming=on and streaming=parallel +# cases. +sub test_streaming +{ + my ($node_A, $node_B, $node_C, $appname_B, $appname_C, $streaming_mode) = + @_; + + my $oldpid_B = $node_A->safe_psql( + 'postgres', " + SELECT pid FROM pg_stat_replication + WHERE application_name = '$appname_B' AND state = 'streaming';"); + my $oldpid_C = $node_B->safe_psql( + 'postgres', " + SELECT pid FROM pg_stat_replication + WHERE application_name = '$appname_C' AND state = 'streaming';"); + + # Setup logical replication streaming mode + + $node_B->safe_psql( + 'postgres', " + ALTER SUBSCRIPTION tap_sub_B + SET (streaming = $streaming_mode);"); + $node_C->safe_psql( + 'postgres', " + ALTER SUBSCRIPTION tap_sub_C + SET (streaming = $streaming_mode)"); + + # Wait for subscribers to finish initialization + + $node_A->poll_query_until( + 'postgres', " + SELECT pid != $oldpid_B FROM pg_stat_replication + WHERE application_name = '$appname_B' AND state = 'streaming';" + ) or die "Timed out while waiting for apply to restart"; + $node_B->poll_query_until( + 'postgres', " + SELECT pid != $oldpid_C FROM pg_stat_replication + WHERE application_name = '$appname_C' AND state = 'streaming';" + ) or die "Timed out while waiting for apply to restart"; + + ############################### + # Test 2PC PREPARE / COMMIT PREPARED. + # 1. Data is streamed as a 2PC transaction. + # 2. Then do commit prepared. + # + # Expect all data is replicated on subscriber(s) after the commit. + ############################### + + # Check that a background worker starts if "streaming" optparameterion is + # specified as "parallel". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($streaming_mode eq 'parallel') + { + $node_B->append_conf('postgresql.conf', "log_min_messages = debug1"); + $node_B->reload; + + $node_C->append_conf('postgresql.conf', "log_min_messages = debug1"); + $node_C->reload; + } + + # Insert, update and delete enough rows to exceed the 64kB limit. + # Then 2PC PREPARE + $node_A->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + if ($streaming_mode eq 'parallel') + { + $node_B->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_B->append_conf('postgresql.conf', "log_min_messages = warning"); + $node_B->reload; + + $node_C->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_C->append_conf('postgresql.conf', "log_min_messages = warning"); + $node_C->reload; + } + + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); + + # check the transaction state is prepared on subscriber(s) + my $result = + $node_B->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber B'); + $result = + $node_C->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber C'); + + # 2PC COMMIT + $node_A->safe_psql('postgres', "COMMIT PREPARED 'test_prepared_tab';"); + + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); + + # check that transaction was committed on subscriber(s) + $result = $node_B->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), + 'Rows inserted by 2PC have committed on subscriber B, and extra columns have local defaults' + ); + $result = $node_C->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), + 'Rows inserted by 2PC have committed on subscriber C, and extra columns have local defaults' + ); + + # check the transaction state is ended on subscriber(s) + $result = + $node_B->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is committed on subscriber B'); + $result = + $node_C->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is committed on subscriber C'); + + ############################### + # Test 2PC PREPARE with a nested ROLLBACK TO SAVEPOINT. + # 0. Cleanup from previous test leaving only 2 rows. + # 1. Insert one more row. + # 2. Record a SAVEPOINT. + # 3. Data is streamed using 2PC. + # 4. Do rollback to SAVEPOINT prior to the streamed inserts. + # 5. Then COMMIT PREPARED. + # + # Expect data after the SAVEPOINT is aborted leaving only 3 rows (= 2 original + 1 from step 1). + ############################### + + # First, delete the data except for 2 rows (delete will be replicated) + $node_A->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); + + # 2PC PREPARE with a nested ROLLBACK TO SAVEPOINT + $node_A->safe_psql( + 'postgres', " + BEGIN; + INSERT INTO test_tab VALUES (9999, 'foobar'); + SAVEPOINT sp_inner; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + ROLLBACK TO SAVEPOINT sp_inner; + PREPARE TRANSACTION 'outer'; + "); + + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); + + # check the transaction state prepared on subscriber(s) + $result = + $node_B->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber B'); + $result = + $node_C->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber C'); + + # 2PC COMMIT + $node_A->safe_psql('postgres', "COMMIT PREPARED 'outer';"); + + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); + + # check the transaction state is ended on subscriber + $result = + $node_B->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is ended on subscriber B'); + $result = + $node_C->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is ended on subscriber C'); + + # check inserts are visible at subscriber(s). + # All the streamed data (prior to the SAVEPOINT) should be rolled back. + # (9999, 'foobar') should be committed. + $result = $node_B->safe_psql('postgres', + "SELECT count(*) FROM test_tab where b = 'foobar';"); + is($result, qq(1), 'Rows committed are present on subscriber B'); + $result = + $node_B->safe_psql('postgres', "SELECT count(*) FROM test_tab;"); + is($result, qq(3), 'Rows committed are present on subscriber B'); + $result = $node_C->safe_psql('postgres', + "SELECT count(*) FROM test_tab where b = 'foobar';"); + is($result, qq(1), 'Rows committed are present on subscriber C'); + $result = + $node_C->safe_psql('postgres', "SELECT count(*) FROM test_tab;"); + is($result, qq(3), 'Rows committed are present on subscriber C'); + + # Cleanup the test data + $node_A->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); +} + ############################### # Setup a cascade of pub/sub nodes. # node_A -> node_B -> node_C @@ -260,160 +462,15 @@ is($result, qq(21), 'Rows committed are present on subscriber C'); # 2PC + STREAMING TESTS # --------------------- -my $oldpid_B = $node_A->safe_psql( - 'postgres', " - SELECT pid FROM pg_stat_replication - WHERE application_name = '$appname_B' AND state = 'streaming';"); -my $oldpid_C = $node_B->safe_psql( - 'postgres', " - SELECT pid FROM pg_stat_replication - WHERE application_name = '$appname_C' AND state = 'streaming';"); - -# Setup logical replication (streaming = on) - -$node_B->safe_psql( - 'postgres', " - ALTER SUBSCRIPTION tap_sub_B - SET (streaming = on);"); -$node_C->safe_psql( - 'postgres', " - ALTER SUBSCRIPTION tap_sub_C - SET (streaming = on)"); - -# Wait for subscribers to finish initialization - -$node_A->poll_query_until( - 'postgres', " - SELECT pid != $oldpid_B FROM pg_stat_replication - WHERE application_name = '$appname_B' AND state = 'streaming';" -) or die "Timed out while waiting for apply to restart"; -$node_B->poll_query_until( - 'postgres', " - SELECT pid != $oldpid_C FROM pg_stat_replication - WHERE application_name = '$appname_C' AND state = 'streaming';" -) or die "Timed out while waiting for apply to restart"; - -############################### -# Test 2PC PREPARE / COMMIT PREPARED. -# 1. Data is streamed as a 2PC transaction. -# 2. Then do commit prepared. -# -# Expect all data is replicated on subscriber(s) after the commit. -############################### - -# Insert, update and delete enough rows to exceed the 64kB limit. -# Then 2PC PREPARE -$node_A->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_A->wait_for_catchup($appname_B); -$node_B->wait_for_catchup($appname_C); +################################ +# Test using streaming mode 'on' +################################ +test_streaming($node_A, $node_B, $node_C, $appname_B, $appname_C, 'on'); -# check the transaction state is prepared on subscriber(s) -$result = - $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber B'); -$result = - $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber C'); - -# 2PC COMMIT -$node_A->safe_psql('postgres', "COMMIT PREPARED 'test_prepared_tab';"); - -$node_A->wait_for_catchup($appname_B); -$node_B->wait_for_catchup($appname_C); - -# check that transaction was committed on subscriber(s) -$result = $node_B->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), - 'Rows inserted by 2PC have committed on subscriber B, and extra columns have local defaults' -); -$result = $node_C->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), - 'Rows inserted by 2PC have committed on subscriber C, and extra columns have local defaults' -); - -# check the transaction state is ended on subscriber(s) -$result = - $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is committed on subscriber B'); -$result = - $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is committed on subscriber C'); - -############################### -# Test 2PC PREPARE with a nested ROLLBACK TO SAVEPOINT. -# 0. Cleanup from previous test leaving only 2 rows. -# 1. Insert one more row. -# 2. Record a SAVEPOINT. -# 3. Data is streamed using 2PC. -# 4. Do rollback to SAVEPOINT prior to the streamed inserts. -# 5. Then COMMIT PREPARED. -# -# Expect data after the SAVEPOINT is aborted leaving only 3 rows (= 2 original + 1 from step 1). -############################### - -# First, delete the data except for 2 rows (delete will be replicated) -$node_A->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); - -# 2PC PREPARE with a nested ROLLBACK TO SAVEPOINT -$node_A->safe_psql( - 'postgres', " - BEGIN; - INSERT INTO test_tab VALUES (9999, 'foobar'); - SAVEPOINT sp_inner; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - ROLLBACK TO SAVEPOINT sp_inner; - PREPARE TRANSACTION 'outer'; - "); - -$node_A->wait_for_catchup($appname_B); -$node_B->wait_for_catchup($appname_C); - -# check the transaction state prepared on subscriber(s) -$result = - $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber B'); -$result = - $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber C'); - -# 2PC COMMIT -$node_A->safe_psql('postgres', "COMMIT PREPARED 'outer';"); - -$node_A->wait_for_catchup($appname_B); -$node_B->wait_for_catchup($appname_C); - -# check the transaction state is ended on subscriber -$result = - $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is ended on subscriber B'); -$result = - $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is ended on subscriber C'); - -# check inserts are visible at subscriber(s). -# All the streamed data (prior to the SAVEPOINT) should be rolled back. -# (9999, 'foobar') should be committed. -$result = $node_B->safe_psql('postgres', - "SELECT count(*) FROM test_tab where b = 'foobar';"); -is($result, qq(1), 'Rows committed are present on subscriber B'); -$result = $node_B->safe_psql('postgres', "SELECT count(*) FROM test_tab;"); -is($result, qq(3), 'Rows committed are present on subscriber B'); -$result = $node_C->safe_psql('postgres', - "SELECT count(*) FROM test_tab where b = 'foobar';"); -is($result, qq(1), 'Rows committed are present on subscriber C'); -$result = $node_C->safe_psql('postgres', "SELECT count(*) FROM test_tab;"); -is($result, qq(3), 'Rows committed are present on subscriber C'); +###################################### +# Test using streaming mode 'parallel' +###################################### +test_streaming($node_A, $node_B, $node_C, $appname_B, $appname_C, 'parallel'); ############################### # check all the cleanup diff --git a/src/test/subscription/t/023_twophase_stream.pl b/src/test/subscription/t/023_twophase_stream.pl index d8475d25a4..b89414ab74 100644 --- a/src/test/subscription/t/023_twophase_stream.pl +++ b/src/test/subscription/t/023_twophase_stream.pl @@ -8,6 +8,266 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" +# parameter so the same code can be run both for the streaming=on and +# streaming=parallel cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_parallel) = @_; + + ############################### + # Test 2PC PREPARE / COMMIT PREPARED. + # 1. Data is streamed as a 2PC transaction. + # 2. Then do commit prepared. + # + # Expect all data is replicated on subscriber side after the commit. + ############################### + + # Check that a background worker starts if "streaming" parameter is + # specified as "parallel". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_parallel) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + # check that 2PC gets replicated to subscriber + # Insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + if ($is_parallel) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is in prepared state on subscriber + my $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber'); + + # 2PC transaction gets committed + $node_publisher->safe_psql('postgres', + "COMMIT PREPARED 'test_prepared_tab';"); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is committed on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), + 'Rows inserted by 2PC have committed on subscriber, and extra columns contain local defaults' + ); + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is committed on subscriber'); + + ############################### + # Test 2PC PREPARE / ROLLBACK PREPARED. + # 1. Table is deleted back to 2 rows which are replicated on subscriber. + # 2. Data is streamed using 2PC. + # 3. Do rollback prepared. + # + # Expect data rolls back leaving only the original 2 rows. + ############################### + + # First, delete the data except for 2 rows (will be replicated) + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE a > 2;"); + + # Then insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is in prepared state on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber'); + + # 2PC transaction gets aborted + $node_publisher->safe_psql('postgres', + "ROLLBACK PREPARED 'test_prepared_tab';"); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is aborted on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(2|2|2), + 'Rows inserted by 2PC are rolled back, leaving only the original 2 rows' + ); + + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is aborted on subscriber'); + + ############################### + # Check that 2PC COMMIT PREPARED is decoded properly on crash restart. + # 1. insert, update and delete enough rows to exceed the 64kB limit. + # 2. Then server crashes before the 2PC transaction is committed. + # 3. After servers are restarted the pending transaction is committed. + # + # Expect all data is replicated on subscriber side after the commit. + # Note: both publisher and subscriber do crash/restart. + ############################### + + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + $node_subscriber->stop('immediate'); + $node_publisher->stop('immediate'); + + $node_publisher->start; + $node_subscriber->start; + + # commit post the restart + $node_publisher->safe_psql('postgres', + "COMMIT PREPARED 'test_prepared_tab';"); + $node_publisher->wait_for_catchup($appname); + + # check inserts are visible + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), + 'Rows inserted by 2PC have committed on subscriber, and extra columns contain local defaults' + ); + + ############################### + # Do INSERT after the PREPARE but before ROLLBACK PREPARED. + # 1. Table is deleted back to 2 rows which are replicated on subscriber. + # 2. Data is streamed using 2PC. + # 3. A single row INSERT is done which is after the PREPARE. + # 4. Then do a ROLLBACK PREPARED. + # + # Expect the 2PC data rolls back leaving only 3 rows on the subscriber + # (the original 2 + inserted 1). + ############################### + + # First, delete the data except for 2 rows (will be replicated) + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE a > 2;"); + + # Then insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is in prepared state on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber'); + + # Insert a different record (now we are outside of the 2PC transaction) + # Note: the 2PC transaction still holds row locks so make sure this insert is for a separate primary key + $node_publisher->safe_psql('postgres', + "INSERT INTO test_tab VALUES (99999, 'foobar')"); + + # 2PC transaction gets aborted + $node_publisher->safe_psql('postgres', + "ROLLBACK PREPARED 'test_prepared_tab';"); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is aborted on subscriber, + # but the extra INSERT outside of the 2PC still was replicated + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3|3|3), + 'check the outside insert was copied to subscriber'); + + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is aborted on subscriber'); + + ############################### + # Do INSERT after the PREPARE but before COMMIT PREPARED. + # 1. Table is deleted back to 2 rows which are replicated on subscriber. + # 2. Data is streamed using 2PC. + # 3. A single row INSERT is done which is after the PREPARE. + # 4. Then do a COMMIT PREPARED. + # + # Expect 2PC data + the extra row are on the subscriber + # (the 3334 + inserted 1 = 3335). + ############################### + + # First, delete the data except for 2 rows (will be replicated) + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE a > 2;"); + + # Then insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is in prepared state on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber'); + + # Insert a different record (now we are outside of the 2PC transaction) + # Note: the 2PC transaction still holds row locks so make sure this insert is for a separare primary key + $node_publisher->safe_psql('postgres', + "INSERT INTO test_tab VALUES (99999, 'foobar')"); + + # 2PC transaction gets committed + $node_publisher->safe_psql('postgres', + "COMMIT PREPARED 'test_prepared_tab';"); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is committed on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3335|3335|3335), + 'Rows inserted by 2PC (as well as outside insert) have committed on subscriber, and extra columns contain local defaults' + ); + + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is committed on subscriber'); + + # Cleanup the test data + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE a > 2;"); + $node_publisher->wait_for_catchup($appname); +} + ############################### # Setup ############################### @@ -48,6 +308,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql( 'postgres', " CREATE SUBSCRIPTION tap_sub @@ -70,236 +334,30 @@ my $twophase_query = $node_subscriber->poll_query_until('postgres', $twophase_query) or die "Timed out while waiting for subscriber to enable twophase"; -############################### # Check initial data was copied to subscriber -############################### my $result = $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab"); is($result, qq(2|2|2), 'check initial data was copied to subscriber'); -############################### -# Test 2PC PREPARE / COMMIT PREPARED. -# 1. Data is streamed as a 2PC transaction. -# 2. Then do commit prepared. -# -# Expect all data is replicated on subscriber side after the commit. -############################### - -# check that 2PC gets replicated to subscriber -# Insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is in prepared state on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber'); - -# 2PC transaction gets committed -$node_publisher->safe_psql('postgres', - "COMMIT PREPARED 'test_prepared_tab';"); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is committed on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), - 'Rows inserted by 2PC have committed on subscriber, and extra columns contain local defaults' -); -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is committed on subscriber'); - -############################### -# Test 2PC PREPARE / ROLLBACK PREPARED. -# 1. Table is deleted back to 2 rows which are replicated on subscriber. -# 2. Data is streamed using 2PC. -# 3. Do rollback prepared. -# -# Expect data rolls back leaving only the original 2 rows. -############################### - -# First, delete the data except for 2 rows (will be replicated) -$node_publisher->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); - -# Then insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is in prepared state on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber'); - -# 2PC transaction gets aborted -$node_publisher->safe_psql('postgres', - "ROLLBACK PREPARED 'test_prepared_tab';"); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is aborted on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(2|2|2), - 'Rows inserted by 2PC are rolled back, leaving only the original 2 rows'); - -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is aborted on subscriber'); - -############################### -# Check that 2PC COMMIT PREPARED is decoded properly on crash restart. -# 1. insert, update and delete enough rows to exceed the 64kB limit. -# 2. Then server crashes before the 2PC transaction is committed. -# 3. After servers are restarted the pending transaction is committed. -# -# Expect all data is replicated on subscriber side after the commit. -# Note: both publisher and subscriber do crash/restart. -############################### - -$node_publisher->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_subscriber->stop('immediate'); -$node_publisher->stop('immediate'); - -$node_publisher->start; -$node_subscriber->start; - -# commit post the restart -$node_publisher->safe_psql('postgres', - "COMMIT PREPARED 'test_prepared_tab';"); -$node_publisher->wait_for_catchup($appname); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -# check inserts are visible -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), - 'Rows inserted by 2PC have committed on subscriber, and extra columns contain local defaults' +###################################### +# Test using streaming mode 'parallel' +###################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" ); -############################### -# Do INSERT after the PREPARE but before ROLLBACK PREPARED. -# 1. Table is deleted back to 2 rows which are replicated on subscriber. -# 2. Data is streamed using 2PC. -# 3. A single row INSERT is done which is after the PREPARE. -# 4. Then do a ROLLBACK PREPARED. -# -# Expect the 2PC data rolls back leaving only 3 rows on the subscriber -# (the original 2 + inserted 1). -############################### - -# First, delete the data except for 2 rows (will be replicated) -$node_publisher->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); - -# Then insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is in prepared state on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber'); - -# Insert a different record (now we are outside of the 2PC transaction) -# Note: the 2PC transaction still holds row locks so make sure this insert is for a separate primary key -$node_publisher->safe_psql('postgres', - "INSERT INTO test_tab VALUES (99999, 'foobar')"); - -# 2PC transaction gets aborted -$node_publisher->safe_psql('postgres', - "ROLLBACK PREPARED 'test_prepared_tab';"); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is aborted on subscriber, -# but the extra INSERT outside of the 2PC still was replicated -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3|3|3), 'check the outside insert was copied to subscriber'); - -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is aborted on subscriber'); - -############################### -# Do INSERT after the PREPARE but before COMMIT PREPARED. -# 1. Table is deleted back to 2 rows which are replicated on subscriber. -# 2. Data is streamed using 2PC. -# 3. A single row INSERT is done which is after the PREPARE. -# 4. Then do a COMMIT PREPARED. -# -# Expect 2PC data + the extra row are on the subscriber -# (the 3334 + inserted 1 = 3335). -############################### - -# First, delete the data except for 2 rows (will be replicated) -$node_publisher->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); - -# Then insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is in prepared state on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber'); - -# Insert a different record (now we are outside of the 2PC transaction) -# Note: the 2PC transaction still holds row locks so make sure this insert is for a separare primary key -$node_publisher->safe_psql('postgres', - "INSERT INTO test_tab VALUES (99999, 'foobar')"); - -# 2PC transaction gets committed -$node_publisher->safe_psql('postgres', - "COMMIT PREPARED 'test_prepared_tab';"); - -$node_publisher->wait_for_catchup($appname); +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = parallel)"); -# check that transaction is committed on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3335|3335|3335), - 'Rows inserted by 2PC (as well as outside insert) have committed on subscriber, and extra columns contain local defaults' -); +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is committed on subscriber'); +test_streaming($node_publisher, $node_subscriber, $appname, 1); ############################### # check all the cleanup -- 2.23.0.windows.1 [application/octet-stream] v18-0003-Add-some-checks-before-using-apply-background-wo.patch (37.8K, ../../OS3PR01MB62758A6AAED27B3A848CEB7A9E8F9@OS3PR01MB6275.jpnprd01.prod.outlook.com/4-v18-0003-Add-some-checks-before-using-apply-background-wo.patch) download | inline diff: From 1751692945eb72ec332e79ccf28c88380ac8e2ae Mon Sep 17 00:00:00 2001 From: wangw <[email protected]> Date: Tue, 14 Jun 2022 11:23:52 +0800 Subject: [PATCH v18 3/4] Add some checks before using apply background worker to apply changes. streaming=parallel mode has two requirements: 1) The unique column in the relation on the subscriber-side should also be the unique column on the publisher-side; 2) There cannot be any non-immutable functions used by the subscriber-side replicated table. Look for functions in the following places: * a. Trigger functions * b. Column default value expressions and domain constraints * c. Constraint expressions * d. Foreign keys --- doc/src/sgml/ref/create_subscription.sgml | 4 + .../replication/logical/applybgworker.c | 44 ++ src/backend/replication/logical/proto.c | 88 +++- src/backend/replication/logical/relation.c | 201 +++++++++ src/backend/replication/logical/tablesync.c | 1 + src/backend/replication/logical/worker.c | 23 +- src/backend/utils/cache/typcache.c | 17 + src/include/replication/logicalproto.h | 1 + src/include/replication/logicalrelation.h | 15 + src/include/replication/worker_internal.h | 1 + src/include/utils/typcache.h | 2 + .../subscription/t/022_twophase_cascade.pl | 6 + .../subscription/t/032_streaming_apply.pl | 380 ++++++++++++++++++ src/tools/pgindent/typedefs.list | 1 + 14 files changed, 775 insertions(+), 9 deletions(-) create mode 100644 src/test/subscription/t/032_streaming_apply.pl diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml index 71dd4aca81..bfd1895087 100644 --- a/doc/src/sgml/ref/create_subscription.sgml +++ b/doc/src/sgml/ref/create_subscription.sgml @@ -240,6 +240,10 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl transaction is committed. Note that if an error happens when applying changes in a background worker, the finish LSN of the remote transaction might not be reported in the server log. + <literal>parallel</literal> mode has two requirements: 1) the unique + column in the relation on the subscriber-side should also be the + unique column on the publisher-side; 2) there cannot be any + non-immutable functions used by the subscriber-side replicated table. </para> </listitem> </varlistentry> diff --git a/src/backend/replication/logical/applybgworker.c b/src/backend/replication/logical/applybgworker.c index aa222490a0..89c712f785 100644 --- a/src/backend/replication/logical/applybgworker.c +++ b/src/backend/replication/logical/applybgworker.c @@ -800,3 +800,47 @@ apply_bgworker_subxact_info_add(TransactionId current_xid) MemoryContextSwitchTo(oldctx); } } + +/* + * Check if changes on this relation can be applied by an apply background + * worker. + * + * Although the commit order is maintained only allowing one process to commit + * at a time, the access order to the relation has changed. This could cause + * unexpected problems if the unique column on the replicated table is + * inconsistent with the publisher-side or contains non-immutable functions + * when applying transactions in the apply background worker. + */ +void +apply_bgworker_relation_check(LogicalRepRelMapEntry *rel) +{ + /* Skip check if not an apply background worker. */ + if (!am_apply_bgworker()) + return; + + /* + * Partition table checks are done later in function + * apply_handle_tuple_routing. + */ + if (rel->localrel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + return; + + /* + * Return if changes on this relation can be applied by an apply background + * worker. + */ + if (rel->parallel_apply == PARALLEL_APPLY_SAFE) + return; + + /* We are in error mode and should give user correct error. */ + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("cannot replicate target relation \"%s.%s\" using " + "subscription parameter streaming=parallel", + rel->remoterel.nspname, rel->remoterel.relname), + errdetail("The unique column on subscriber is not the unique " + "column on publisher or there is at least one " + "non-immutable function."), + errhint("Please change to use subscription parameter " + "streaming=on."))); +} diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c index 47bd811fb7..511bd9c052 100644 --- a/src/backend/replication/logical/proto.c +++ b/src/backend/replication/logical/proto.c @@ -23,7 +23,8 @@ /* * Protocol message flags. */ -#define LOGICALREP_IS_REPLICA_IDENTITY 1 +#define ATTR_IS_REPLICA_IDENTITY (1 << 0) +#define ATTR_IS_UNIQUE (1 << 1) #define MESSAGE_TRANSACTIONAL (1<<0) #define TRUNCATE_CASCADE (1<<0) @@ -40,6 +41,68 @@ static void logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple); static void logicalrep_write_namespace(StringInfo out, Oid nspid); static const char *logicalrep_read_namespace(StringInfo in); +static Bitmapset *RelationGetUniqueKeyBitmap(Relation rel); + +/* + * RelationGetUniqueKeyBitmap -- get a bitmap of unique attribute numbers + * + * This is similar to RelationGetIdentityKeyBitmap(), but returns a bitmap of + * index attribute numbers for all unique indexes. + */ +static Bitmapset * +RelationGetUniqueKeyBitmap(Relation rel) +{ + List *indexoidlist = NIL; + ListCell *indexoidscan; + Bitmapset *attunique = NULL; + + if (!rel->rd_rel->relhasindex) + return NULL; + + indexoidlist = RelationGetIndexList(rel); + + foreach(indexoidscan, indexoidlist) + { + Oid indexoid = lfirst_oid(indexoidscan); + Relation indexRel; + int i; + + /* Look up the description for index */ + indexRel = RelationIdGetRelation(indexoid); + + if (!RelationIsValid(indexRel)) + elog(ERROR, "could not open relation with OID %u", indexoid); + + if (!indexRel->rd_index->indisunique) + { + RelationClose(indexRel); + continue; + } + + /* Add referenced attributes to idindexattrs */ + for (i = 0; i < indexRel->rd_index->indnatts; i++) + { + int attrnum = indexRel->rd_index->indkey.values[i]; + + /* + * We don't include non-key columns into idindexattrs + * bitmaps. See RelationGetIndexAttrBitmap. + */ + if (attrnum != 0) + { + if (i < indexRel->rd_index->indnkeyatts && + !bms_is_member(attrnum - FirstLowInvalidHeapAttributeNumber, attunique)) + attunique = bms_add_member(attunique, + attrnum - FirstLowInvalidHeapAttributeNumber); + } + } + RelationClose(indexRel); + } + list_free(indexoidlist); + + return attunique; +} + /* * Check if a column is covered by a column list. * @@ -933,7 +996,8 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns) TupleDesc desc; int i; uint16 nliveatts = 0; - Bitmapset *idattrs = NULL; + Bitmapset *idattrs = NULL, + *attunique = NULL; bool replidentfull; desc = RelationGetDescr(rel); @@ -958,6 +1022,9 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns) if (!replidentfull) idattrs = RelationGetIdentityKeyBitmap(rel); + /* fetch bitmap of UNIQUE attributes */ + attunique = RelationGetUniqueKeyBitmap(rel); + /* send the attributes */ for (i = 0; i < desc->natts; i++) { @@ -974,7 +1041,11 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns) if (replidentfull || bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber, idattrs)) - flags |= LOGICALREP_IS_REPLICA_IDENTITY; + flags |= ATTR_IS_REPLICA_IDENTITY; + + if (bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber, + attunique)) + flags |= ATTR_IS_UNIQUE; pq_sendbyte(out, flags); @@ -989,6 +1060,7 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns) } bms_free(idattrs); + bms_free(attunique); } /* @@ -1001,7 +1073,8 @@ logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel) int natts; char **attnames; Oid *atttyps; - Bitmapset *attkeys = NULL; + Bitmapset *attkeys = NULL, + *attunique = NULL; natts = pq_getmsgint(in, 2); attnames = palloc(natts * sizeof(char *)); @@ -1014,9 +1087,13 @@ logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel) /* Check for replica identity column */ flags = pq_getmsgbyte(in); - if (flags & LOGICALREP_IS_REPLICA_IDENTITY) + if (flags & ATTR_IS_REPLICA_IDENTITY) attkeys = bms_add_member(attkeys, i); + /* Check for unique column */ + if (flags & ATTR_IS_UNIQUE) + attunique = bms_add_member(attunique, i); + /* attribute name */ attnames[i] = pstrdup(pq_getmsgstring(in)); @@ -1030,6 +1107,7 @@ logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel) rel->attnames = attnames; rel->atttyps = atttyps; rel->attkeys = attkeys; + rel->attunique = attunique; rel->natts = natts; } diff --git a/src/backend/replication/logical/relation.c b/src/backend/replication/logical/relation.c index e989047681..37e410b8d0 100644 --- a/src/backend/replication/logical/relation.c +++ b/src/backend/replication/logical/relation.c @@ -19,12 +19,19 @@ #include "access/table.h" #include "catalog/namespace.h" +#include "catalog/pg_proc.h" #include "catalog/pg_subscription_rel.h" +#include "commands/trigger.h" #include "executor/executor.h" #include "nodes/makefuncs.h" +#include "optimizer/optimizer.h" #include "replication/logicalrelation.h" #include "replication/worker_internal.h" +#include "rewrite/rewriteHandler.h" #include "utils/inval.h" +#include "utils/lsyscache.h" +#include "utils/syscache.h" +#include "utils/typcache.h" static MemoryContext LogicalRepRelMapContext = NULL; @@ -91,6 +98,26 @@ logicalrep_relmap_invalidate_cb(Datum arg, Oid reloid) } } +/* + * Relcache invalidation callback to reset parallel flag. + */ +static void +logicalrep_relmap_reset_parallel_cb(Datum arg, int cacheid, uint32 hashvalue) +{ + HASH_SEQ_STATUS hash_seq; + LogicalRepRelMapEntry *entry; + + if (LogicalRepRelMap == NULL) + return; + + hash_seq_init(&hash_seq, LogicalRepRelMap); + while ((entry = hash_seq_search(&hash_seq)) != NULL) + { + entry->parallel_apply = PARALLEL_APPLY_UNKNOWN; + entry->localrelvalid = false; + } +} + /* * Initialize the relation map cache. */ @@ -116,6 +143,9 @@ logicalrep_relmap_init(void) /* Watch for invalidation events. */ CacheRegisterRelcacheCallback(logicalrep_relmap_invalidate_cb, (Datum) 0); + CacheRegisterSyscacheCallback(PROCOID, + logicalrep_relmap_reset_parallel_cb, + (Datum) 0); } /* @@ -142,6 +172,7 @@ logicalrep_relmap_free_entry(LogicalRepRelMapEntry *entry) pfree(remoterel->atttyps); } bms_free(remoterel->attkeys); + bms_free(remoterel->attunique); if (entry->attrmap) free_attrmap(entry->attrmap); @@ -190,6 +221,7 @@ logicalrep_relmap_update(LogicalRepRelation *remoterel) } entry->remoterel.replident = remoterel->replident; entry->remoterel.attkeys = bms_copy(remoterel->attkeys); + entry->remoterel.attunique = bms_copy(remoterel->attunique); MemoryContextSwitchTo(oldctx); } @@ -310,6 +342,168 @@ logicalrep_rel_mark_updatable(LogicalRepRelMapEntry *entry) } } +/* + * Check if changes on one relation can be applied by an apply background + * worker and assign the 'parallel_apply' flag. + * + * There are two requirements for applying changes in an apply background + * worker: 1) The unique column in the relation on the subscriber-side should + * also be the unique column on the publisher-side; 2) There cannot be any + * non-immutable functions used by the subscriber-side. + * + * We just mark the relation entry as 'PARALLEL_APPLY_UNSAFE' here if changes + * on one relation can not be applied by an apply background worker and leave + * it to apply_bgworker_relation_check() to throw the actual error if needed. + */ +static void +logicalrep_rel_mark_parallel_apply(LogicalRepRelMapEntry *entry) +{ + Bitmapset *ukey; + int i; + TupleDesc tupdesc; + int attnum; + List *fkeys = NIL; + + /* Fast path if 'parallel_apply' flag is already known. */ + if (entry->parallel_apply != PARALLEL_APPLY_UNKNOWN) + return; + + /* Initialize the flag. */ + entry->parallel_apply = PARALLEL_APPLY_SAFE; + + /* + * First, check if the unique column in the relation on the subscriber-side + * is also the unique column on the publisher-side. + */ + ukey = RelationGetIndexAttrBitmap(entry->localrel, + INDEX_ATTR_BITMAP_KEY); + + if (ukey) + { + i = -1; + while ((i = bms_next_member(ukey, i)) >= 0) + { + attnum = AttrNumberGetAttrOffset(i + FirstLowInvalidHeapAttributeNumber); + + if (entry->attrmap->attnums[attnum] < 0 || + !bms_is_member(entry->attrmap->attnums[attnum], entry->remoterel.attunique)) + { + entry->parallel_apply = PARALLEL_APPLY_UNSAFE; + return; + } + } + + bms_free(ukey); + } + + /* + * Then, check if there is any non-immutable function used by the local + * table. Look for functions in the following places: + * a. trigger functions; + * b. Column default value expressions and domain constraints; + * c. Constraint expressions; + * d. Foreign keys. + */ + /* Check the trigger functions. */ + if (entry->localrel->trigdesc != NULL) + { + for (i = 0; i < entry->localrel->trigdesc->numtriggers; i++) + { + Trigger *trig = entry->localrel->trigdesc->triggers + i; + + if (trig->tgenabled != TRIGGER_FIRES_ALWAYS && + trig->tgenabled != TRIGGER_FIRES_ON_REPLICA) + continue; + + if (func_volatile(trig->tgfoid) != PROVOLATILE_IMMUTABLE) + { + entry->parallel_apply = PARALLEL_APPLY_UNSAFE; + return; + } + } + } + + /* Check the columns. */ + tupdesc = RelationGetDescr(entry->localrel); + for (attnum = 0; attnum < tupdesc->natts; attnum++) + { + Form_pg_attribute att = TupleDescAttr(tupdesc, attnum); + + /* We don't need info for dropped or generated attributes */ + if (att->attisdropped || att->attgenerated) + continue; + + /* + * We don't need to check columns that only exist on the + * subscriber + */ + if (entry->attrmap->attnums[attnum] < 0) + continue; + + if (att->atthasdef) + { + Node *defaultexpr; + + defaultexpr = build_column_default(entry->localrel, attnum + 1); + if (contain_mutable_functions(defaultexpr)) + { + entry->parallel_apply = PARALLEL_APPLY_UNSAFE; + return; + } + } + + /* + * If the column is of a DOMAIN type, determine whether + * that domain has any CHECK expressions that are not + * immutable. + */ + if (get_typtype(att->atttypid) == TYPTYPE_DOMAIN) + { + List *domain_constraints; + ListCell *lc; + + domain_constraints = GetDomainConstraints(att->atttypid); + + foreach(lc, domain_constraints) + { + DomainConstraintState *con = (DomainConstraintState *) lfirst(lc); + + if (con->check_expr && contain_mutable_functions((Node *) con->check_expr)) + { + entry->parallel_apply = PARALLEL_APPLY_UNSAFE; + return; + } + } + } + } + + /* Check the constraints. */ + if (tupdesc->constr) + { + ConstrCheck *check = tupdesc->constr->check; + + /* + * Determine if there are any CHECK constraints which + * contains non-immutable function. + */ + for (i = 0; i < tupdesc->constr->num_check; i++) + { + Expr *check_expr = stringToNode(check[i].ccbin); + + if (contain_mutable_functions((Node *) check_expr)) + { + entry->parallel_apply = PARALLEL_APPLY_UNSAFE; + return; + } + } + } + + /* Check the foreign keys. */ + fkeys = RelationGetFKeyList(entry->localrel); + if (fkeys) + entry->parallel_apply = PARALLEL_APPLY_UNSAFE; +} + /* * Open the local relation associated with the remote one. * @@ -438,6 +632,9 @@ logicalrep_rel_open(LogicalRepRelId remoteid, LOCKMODE lockmode) */ logicalrep_rel_mark_updatable(entry); + /* Set if changes could be applied in the apply background worker. */ + logicalrep_rel_mark_parallel_apply(entry); + entry->localrelvalid = true; } @@ -653,6 +850,7 @@ logicalrep_partition_open(LogicalRepRelMapEntry *root, } entry->remoterel.replident = remoterel->replident; entry->remoterel.attkeys = bms_copy(remoterel->attkeys); + entry->remoterel.attunique = bms_copy(remoterel->attunique); } entry->localrel = partrel; @@ -696,6 +894,9 @@ logicalrep_partition_open(LogicalRepRelMapEntry *root, /* Set if the table's replica identity is enough to apply update/delete. */ logicalrep_rel_mark_updatable(entry); + /* Set if changes could be applied in the apply background worker. */ + logicalrep_rel_mark_parallel_apply(entry); + entry->localrelvalid = true; /* state and statelsn are left set to 0. */ diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c index 8ffba7e2e5..3cdbf8b457 100644 --- a/src/backend/replication/logical/tablesync.c +++ b/src/backend/replication/logical/tablesync.c @@ -884,6 +884,7 @@ fetch_remote_table_info(char *nspname, char *relname, lrel->attnames = palloc0(MaxTupleAttributeNumber * sizeof(char *)); lrel->atttyps = palloc0(MaxTupleAttributeNumber * sizeof(Oid)); lrel->attkeys = NULL; + lrel->attunique = NULL; /* * Store the columns as a list of names. Ignore those that are not diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c index b5aae0e19a..2216f6f3a7 100644 --- a/src/backend/replication/logical/worker.c +++ b/src/backend/replication/logical/worker.c @@ -1391,6 +1391,14 @@ apply_handle_stream_stop(StringInfo s) { char action = LOGICAL_REP_MSG_STREAM_STOP; + /* + * Unlike stream_commit, we don't need to wait here for stream_stop to + * finish. Allowing the other transaction to be applied before + * stream_stop is finished can lead to failures if the unique + * index/constraint is different between publisher and subscriber. But + * for such cases, we don't allow streamed transactions to be applied + * in parallel. See apply_bgworker_relation_check. + */ apply_bgworker_send_data(stream_apply_worker, 1, &action); elog(DEBUG1, "stopped streaming of xid %u, %u changes streamed", stream_xid, nchanges); @@ -2044,6 +2052,8 @@ apply_handle_insert(StringInfo s) /* Set relation for error callback */ apply_error_callback_arg.rel = rel; + apply_bgworker_relation_check(rel); + /* Initialize the executor state. */ edata = create_edata_for_relation(rel); estate = edata->estate; @@ -2187,6 +2197,8 @@ apply_handle_update(StringInfo s) /* Check if we can do the update. */ check_relation_updatable(rel); + apply_bgworker_relation_check(rel); + /* Initialize the executor state. */ edata = create_edata_for_relation(rel); estate = edata->estate; @@ -2355,6 +2367,8 @@ apply_handle_delete(StringInfo s) /* Check if we can do the delete. */ check_relation_updatable(rel); + apply_bgworker_relation_check(rel); + /* Initialize the executor state. */ edata = create_edata_for_relation(rel); estate = edata->estate; @@ -2540,13 +2554,14 @@ apply_handle_tuple_routing(ApplyExecutionData *edata, } MemoryContextSwitchTo(oldctx); + part_entry = logicalrep_partition_open(relmapentry, partrel, + attrmap); + /* Check if we can do the update or delete on the leaf partition. */ if (operation == CMD_UPDATE || operation == CMD_DELETE) - { - part_entry = logicalrep_partition_open(relmapentry, partrel, - attrmap); check_relation_updatable(part_entry); - } + + apply_bgworker_relation_check(part_entry); switch (operation) { diff --git a/src/backend/utils/cache/typcache.c b/src/backend/utils/cache/typcache.c index 808f9ebd0d..b248899d82 100644 --- a/src/backend/utils/cache/typcache.c +++ b/src/backend/utils/cache/typcache.c @@ -2540,6 +2540,23 @@ compare_values_of_enum(TypeCacheEntry *tcache, Oid arg1, Oid arg2) return 0; } +/* + * GetDomainConstraints --- get DomainConstraintState list of specified domain type + */ +List * +GetDomainConstraints(Oid type_id) +{ + TypeCacheEntry *typentry; + List *constraints = NIL; + + typentry = lookup_type_cache(type_id, TYPECACHE_DOMAIN_CONSTR_INFO); + + if(typentry->domainData != NULL) + constraints = typentry->domainData->constraints; + + return constraints; +} + /* * Load (or re-load) the enumData member of the typcache entry. */ diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h index eb0fd24fd8..4395c11f75 100644 --- a/src/include/replication/logicalproto.h +++ b/src/include/replication/logicalproto.h @@ -113,6 +113,7 @@ typedef struct LogicalRepRelation char replident; /* replica identity */ char relkind; /* remote relation kind */ Bitmapset *attkeys; /* Bitmap of key columns */ + Bitmapset *attunique; /* Bitmap of unique columns */ } LogicalRepRelation; /* Type mapping info */ diff --git a/src/include/replication/logicalrelation.h b/src/include/replication/logicalrelation.h index 78cd7e77f5..8011e648d7 100644 --- a/src/include/replication/logicalrelation.h +++ b/src/include/replication/logicalrelation.h @@ -15,6 +15,19 @@ #include "access/attmap.h" #include "replication/logicalproto.h" +/* + * States to determine if changes on one relation can be applied using an + * apply background worker. + */ +typedef enum ParalleApplySafety +{ + PARALLEL_APPLY_UNKNOWN = 0, /* unknown */ + PARALLEL_APPLY_SAFE, /* Can apply changes in an apply background + worker */ + PARALLEL_APPLY_UNSAFE /* Can not apply changes in an apply background + worker */ +} ParalleApplySafety; + typedef struct LogicalRepRelMapEntry { LogicalRepRelation remoterel; /* key is remoterel.remoteid */ @@ -31,6 +44,8 @@ typedef struct LogicalRepRelMapEntry Relation localrel; /* relcache entry (NULL when closed) */ AttrMap *attrmap; /* map of local attributes to remote ones */ bool updatable; /* Can apply updates/deletes? */ + ParalleApplySafety parallel_apply; /* Can apply changes in an apply + background worker? */ /* Sync state. */ char state; diff --git a/src/include/replication/worker_internal.h b/src/include/replication/worker_internal.h index a3560d4904..1c0db05c8a 100644 --- a/src/include/replication/worker_internal.h +++ b/src/include/replication/worker_internal.h @@ -194,6 +194,7 @@ extern void apply_bgworker_free(ApplyBgworkerState *wstate); extern void apply_bgworker_check_status(void); extern void apply_bgworker_set_status(ApplyBgworkerStatus status); extern void apply_bgworker_subxact_info_add(TransactionId current_xid); +extern void apply_bgworker_relation_check(LogicalRepRelMapEntry *rel); static inline bool am_tablesync_worker(void) diff --git a/src/include/utils/typcache.h b/src/include/utils/typcache.h index 431ad7f1b3..ed7c2e7f48 100644 --- a/src/include/utils/typcache.h +++ b/src/include/utils/typcache.h @@ -199,6 +199,8 @@ extern uint64 assign_record_type_identifier(Oid type_id, int32 typmod); extern int compare_values_of_enum(TypeCacheEntry *tcache, Oid arg1, Oid arg2); +extern List *GetDomainConstraints(Oid type_id); + extern size_t SharedRecordTypmodRegistryEstimate(void); extern void SharedRecordTypmodRegistryInit(SharedRecordTypmodRegistry *, diff --git a/src/test/subscription/t/022_twophase_cascade.pl b/src/test/subscription/t/022_twophase_cascade.pl index 0a4152d3be..30a01f7305 100644 --- a/src/test/subscription/t/022_twophase_cascade.pl +++ b/src/test/subscription/t/022_twophase_cascade.pl @@ -39,6 +39,12 @@ sub test_streaming ALTER SUBSCRIPTION tap_sub_C SET (streaming = $streaming_mode)"); + if ($streaming_mode eq 'parallel') + { + $node_C->safe_psql( + 'postgres', "ALTER TABLE test_tab ALTER c DROP DEFAULT"); + } + # Wait for subscribers to finish initialization $node_A->poll_query_until( diff --git a/src/test/subscription/t/032_streaming_apply.pl b/src/test/subscription/t/032_streaming_apply.pl new file mode 100644 index 0000000000..7f8bfa6745 --- /dev/null +++ b/src/test/subscription/t/032_streaming_apply.pl @@ -0,0 +1,380 @@ +# Copyright (c) 2022, PostgreSQL Global Development Group + +# Test the restrictions of streaming mode "parallel" in logical replication + +use strict; +use warnings; +use PostgreSQL::Test::Cluster; +use PostgreSQL::Test::Utils; +use Test::More; + +my $offset = 0; + +# Create publisher node +my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); +$node_publisher->init(allows_streaming => 'logical'); +$node_publisher->append_conf('postgresql.conf', + 'logical_decoding_work_mem = 64kB'); +$node_publisher->start; + +# Create subscriber node +my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber'); +$node_subscriber->init; +$node_subscriber->start; + +# Setup structure on publisher +$node_publisher->safe_psql('postgres', + "CREATE TABLE test_tab (a int primary key, b varchar)"); +$node_publisher->safe_psql('postgres', + "CREATE TABLE test_tab_partitioned (a int primary key, b varchar)"); + +# Setup structure on subscriber +# We need to test normal table and partition table. +$node_subscriber->safe_psql('postgres', + "CREATE TABLE test_tab (a int primary key, b varchar)"); +$node_subscriber->safe_psql('postgres', + "CREATE TABLE test_tab_partitioned (a int primary key, b varchar) PARTITION BY RANGE(a)"); +$node_subscriber->safe_psql('postgres', + "CREATE TABLE test_tab_partition (LIKE test_tab_partitioned)"); +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partitioned ATTACH PARTITION test_tab_partition DEFAULT" +); + +# Setup logical replication +my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres'; +$node_publisher->safe_psql('postgres', + "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); +$node_publisher->safe_psql('postgres', + "CREATE PUBLICATION tap_pub_partitioned FOR TABLE test_tab_partitioned"); + +my $appname = 'tap_sub'; +$node_subscriber->safe_psql( + 'postgres', " + CREATE SUBSCRIPTION tap_sub + CONNECTION '$publisher_connstr application_name=$appname' + PUBLICATION tap_pub, tap_pub_partitioned + WITH (streaming = parallel, copy_data = false)"); + +$node_publisher->wait_for_catchup($appname); + +# It is not allowed that the unique index on the publisher and the subscriber +# is different. Check the error reported by background worker in this case. +# First we check the unique index on normal table. +$node_subscriber->safe_psql('postgres', + "CREATE UNIQUE INDEX test_tab_b_idx ON test_tab (b)"); + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log( + qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab" using subscription parameter streaming=parallel/, + $offset); + +# Drop the unique index on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', "DROP INDEX test_tab_b_idx"); + +$node_publisher->wait_for_catchup($appname); + +my $result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); +is($result, qq(5000), 'data replicated to subscriber after dropping index'); + +# Then we check the unique index on partition table. +$node_subscriber->safe_psql('postgres', + "CREATE UNIQUE INDEX test_tab_b_partition_idx ON test_tab_partition (b)"); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab_partitioned SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log( + qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab_partitioned" using subscription parameter streaming=parallel/, + $offset); + +# Drop the unique index on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', "DROP INDEX test_tab_b_partition_idx"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); +is($result, qq(5000), 'data replicated to subscriber after dropping index'); + +# Triggers which execute non-immutable function are not allowed on the +# subscriber side. Check the error reported by background worker in this case. +# First we check the trigger function on normal table. +$node_subscriber->safe_psql( + 'postgres', qq{ +CREATE FUNCTION trigger_func() RETURNS TRIGGER AS \$\$ + BEGIN + RETURN NULL; + END +\$\$ language plpgsql; +CREATE TRIGGER insert_trig +BEFORE INSERT ON test_tab +FOR EACH ROW EXECUTE PROCEDURE trigger_func(); +ALTER TABLE test_tab ENABLE REPLICA TRIGGER insert_trig; +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', "DELETE FROM test_tab"); + +$node_subscriber->wait_for_log( + qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab" using subscription parameter streaming=parallel/, + $offset); + +# Drop the trigger on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "DROP TRIGGER insert_trig ON test_tab"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); +is($result, qq(0), 'data replicated to subscriber after dropping trigger'); + +# Then we check the trigger function on partition table. +$node_subscriber->safe_psql( + 'postgres', qq{ +CREATE TRIGGER insert_trig +BEFORE INSERT ON test_tab_partition +FOR EACH ROW EXECUTE PROCEDURE trigger_func(); +ALTER TABLE test_tab_partition ENABLE REPLICA TRIGGER insert_trig; +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', "DELETE FROM test_tab_partitioned"); + +$node_subscriber->wait_for_log( + qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab_partitioned" using subscription parameter streaming=parallel/, + $offset); + +# Drop the trigger on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "DROP TRIGGER insert_trig ON test_tab_partition"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); +is($result, qq(0), 'data replicated to subscriber after dropping trigger'); + +# It is not allowed that column default value expression contains a +# non-immutable function on the subscriber side. Check the error reported by +# background worker in this case. +# First we check the column default value expression on normal table. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab ALTER COLUMN b SET DEFAULT random()"); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log( + qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab" using subscription parameter streaming=parallel/, + $offset); + +# Drop default value on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab ALTER COLUMN b DROP DEFAULT"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); +is($result, qq(5000), + 'data replicated to subscriber after dropping default value expression'); + +# Then we check the column default value expression on partition table. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partition ALTER COLUMN b SET DEFAULT random()"); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab_partitioned SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log( + qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab_partitioned" using subscription parameter streaming=parallel/, + $offset); + +# Drop default value on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partition ALTER COLUMN b DROP DEFAULT"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); +is($result, qq(5000), + 'data replicated to subscriber after dropping default value expression'); + +# It is not allowed that domain constraint expression contains a non-immutable +# function on the subscriber side. Check the error reported by background +# worker in this case. +# Because the column type of the partition table must be the same as its parent +# table, only test normal table here. +$node_subscriber->safe_psql( + 'postgres', qq{ +CREATE DOMAIN test_domain AS int CHECK(VALUE > random()); +ALTER TABLE test_tab ALTER COLUMN a TYPE test_domain; +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', "DELETE FROM test_tab"); + +$node_subscriber->wait_for_log( + qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab" using subscription parameter streaming=parallel/, + $offset); + +# Drop domain constraint expression on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab ALTER COLUMN a TYPE int"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); +is($result, qq(0), + 'data replicated to subscriber after dropping domain constraint expression' +); + +# It is not allowed that constraint expression contains a non-immutable function +# on the subscriber side. Check the error reported by background worker in this +# case. +# First we check the constraint expression on normal table. +$node_subscriber->safe_psql( + 'postgres', qq{ +ALTER TABLE test_tab ADD CONSTRAINT test_tab_con check (a > random()); +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log( + qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab" using subscription parameter streaming=parallel/, + $offset); + +# Drop constraint on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab DROP CONSTRAINT test_tab_con"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); +is($result, qq(5000), + 'data replicated to subscriber after dropping constraint expression'); + +# Then we check the constraint expression on partition table. +$node_subscriber->safe_psql( + 'postgres', qq{ +ALTER TABLE test_tab_partition ADD CONSTRAINT test_tab_con check (a > random()); +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', "DELETE FROM test_tab_partitioned"); + +$node_subscriber->wait_for_log( + qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab_partitioned" using subscription parameter streaming=parallel/, + $offset); + +# Drop constraint on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partition DROP CONSTRAINT test_tab_con"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); +is($result, qq(0), + 'data replicated to subscriber after dropping constraint expression'); + +# It is not allowed that foreign key on the subscriber side. Check the error +# reported by background worker in this case. +# First we check the foreign key on normal table. +$node_publisher->safe_psql('postgres', "DELETE FROM test_tab"); +$node_publisher->wait_for_catchup($appname); +$node_subscriber->safe_psql( + 'postgres', qq{ +CREATE TABLE test_tab_f (a int primary key); +ALTER TABLE test_tab ADD CONSTRAINT test_tabfk FOREIGN KEY(a) REFERENCES test_tab_f(a); +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log( + qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab" using subscription parameter streaming=parallel/, + $offset); + +# Drop the foreign key constraint on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab DROP CONSTRAINT test_tabfk"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); +is($result, qq(5000), + 'data replicated to subscriber after dropping the foreign key'); + +# Then we check the foreign key on partition table. +$node_publisher->wait_for_catchup($appname); +$node_subscriber->safe_psql( + 'postgres', qq{ +CREATE TABLE test_tab_partition_f (a int primary key); +ALTER TABLE test_tab_partition ADD CONSTRAINT test_tab_patition_fk FOREIGN KEY(a) REFERENCES test_tab_partition_f(a); +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab_partitioned SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log( + qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab_partitioned" using subscription parameter streaming=parallel/, + $offset); + +# Drop the foreign key constraint on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partition DROP CONSTRAINT test_tab_patition_fk"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); +is($result, qq(5000), + 'data replicated to subscriber after dropping the foreign key'); + +$node_subscriber->stop; +$node_publisher->stop; + +done_testing(); diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 4137dc77b4..697e6a7ba3 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -1891,6 +1891,7 @@ PageXLogRecPtr PagetableEntry Pairs ParallelAppendState +ParallelApplySafety ParallelBitmapHeapState ParallelBlockTableScanDesc ParallelBlockTableScanWorker -- 2.23.0.windows.1 [application/octet-stream] v18-0004-Retry-to-apply-streaming-xact-only-in-apply-work.patch (29.2K, ../../OS3PR01MB62758A6AAED27B3A848CEB7A9E8F9@OS3PR01MB6275.jpnprd01.prod.outlook.com/5-v18-0004-Retry-to-apply-streaming-xact-only-in-apply-work.patch) download | inline diff: From 20f9f5a8ecad44f8ab761a9078b4e00d52de5bc1 Mon Sep 17 00:00:00 2001 From: wangw <[email protected]> Date: Wed, 15 Jun 2022 11:02:08 +0800 Subject: [PATCH v18 4/4] Retry to apply streaming xact only in apply worker When the subscription parameter is set streaming=parallel, the logic tries to apply the streaming transaction using an apply background worker. If this fails the background worker exits with an error. In this case, retry applying the streaming transaction using the normal streaming=on mode. This is done to avoid getting caught in a loop of the same retry errors. A new flag field "subretry" has been introduced to catalog "pg_subscription". If the subscriber exits with an error, this flag will be set true, and whenever the transaction is applied successfully, this flag is reset false. Now, when deciding how to apply a streaming transaction, the logic can know if this transaction has previously failed or not (by checking the "subretry" field). --- doc/src/sgml/catalogs.sgml | 9 + doc/src/sgml/ref/create_subscription.sgml | 5 + src/backend/catalog/pg_subscription.c | 1 + src/backend/catalog/system_views.sql | 4 +- src/backend/commands/subscriptioncmds.c | 1 + .../replication/logical/applybgworker.c | 16 +- src/backend/replication/logical/worker.c | 165 +++++++++++++----- src/bin/pg_dump/pg_dump.c | 5 +- src/include/catalog/pg_subscription.h | 4 + .../subscription/t/032_streaming_apply.pl | 154 +++++++++------- 10 files changed, 254 insertions(+), 110 deletions(-) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index 13c9e8eca1..816bcfe089 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -7907,6 +7907,15 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>subretry</structfield> <type>bool</type> + </para> + <para> + True if the previous apply change failed, necessitating a retry + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>subconninfo</structfield> <type>text</type> diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml index bfd1895087..240a8331a0 100644 --- a/doc/src/sgml/ref/create_subscription.sgml +++ b/doc/src/sgml/ref/create_subscription.sgml @@ -244,6 +244,11 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl column in the relation on the subscriber-side should also be the unique column on the publisher-side; 2) there cannot be any non-immutable functions used by the subscriber-side replicated table. + When applying a streaming transaction, if either requirement is not + met, the background worker will exit with an error. + <literal>parallel</literal> mode is disregarded when retrying; + instead the transaction will be applied using <literal>on</literal> + mode. </para> </listitem> </varlistentry> diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c index 8856ce3b50..9b7f09653d 100644 --- a/src/backend/catalog/pg_subscription.c +++ b/src/backend/catalog/pg_subscription.c @@ -71,6 +71,7 @@ GetSubscription(Oid subid, bool missing_ok) sub->stream = subform->substream; sub->twophasestate = subform->subtwophasestate; sub->disableonerr = subform->subdisableonerr; + sub->retry = subform->subretry; /* Get conninfo */ datum = SysCacheGetAttr(SUBSCRIPTIONOID, diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql index fedaed533b..10f4dd6785 100644 --- a/src/backend/catalog/system_views.sql +++ b/src/backend/catalog/system_views.sql @@ -1298,8 +1298,8 @@ REVOKE ALL ON pg_replication_origin_status FROM public; -- All columns of pg_subscription except subconninfo are publicly readable. REVOKE ALL ON pg_subscription FROM public; GRANT SELECT (oid, subdbid, subskiplsn, subname, subowner, subenabled, - subbinary, substream, subtwophasestate, subdisableonerr, subslotname, - subsynccommit, subpublications) + subbinary, substream, subtwophasestate, subdisableonerr, + subretry, subslotname, subsynccommit, subpublications) ON pg_subscription TO public; CREATE VIEW pg_stat_subscription_stats AS diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c index d4b2616ee6..612a83393a 100644 --- a/src/backend/commands/subscriptioncmds.c +++ b/src/backend/commands/subscriptioncmds.c @@ -662,6 +662,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt, LOGICALREP_TWOPHASE_STATE_PENDING : LOGICALREP_TWOPHASE_STATE_DISABLED); values[Anum_pg_subscription_subdisableonerr - 1] = BoolGetDatum(opts.disableonerr); + values[Anum_pg_subscription_subretry - 1] = BoolGetDatum(false); values[Anum_pg_subscription_subconninfo - 1] = CStringGetTextDatum(conninfo); if (opts.slot_name) diff --git a/src/backend/replication/logical/applybgworker.c b/src/backend/replication/logical/applybgworker.c index 89c712f785..5d7b55e5ab 100644 --- a/src/backend/replication/logical/applybgworker.c +++ b/src/backend/replication/logical/applybgworker.c @@ -106,6 +106,18 @@ apply_bgworker_can_start(TransactionId xid) if (!XLogRecPtrIsInvalid(MySubscription->skiplsn)) return false; + /* + * Don't use apply background workers for retries, because it is possible + * that the last time we tried to apply a transaction using an apply + * background worker the checks failed (see function + * apply_bgworker_relation_check). + */ + if (MySubscription->retry) + { + elog(DEBUG1, "apply background workers are not used for retries"); + return false; + } + /* * For streaming transactions that are being applied in apply background * worker, we cannot decide whether to apply the change for a relation @@ -840,7 +852,5 @@ apply_bgworker_relation_check(LogicalRepRelMapEntry *rel) rel->remoterel.nspname, rel->remoterel.relname), errdetail("The unique column on subscriber is not the unique " "column on publisher or there is at least one " - "non-immutable function."), - errhint("Please change to use subscription parameter " - "streaming=on."))); + "non-immutable function."))); } diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c index 2216f6f3a7..9635a12fea 100644 --- a/src/backend/replication/logical/worker.c +++ b/src/backend/replication/logical/worker.c @@ -379,6 +379,8 @@ static void clear_subscription_skip_lsn(XLogRecPtr finish_lsn); static inline void set_apply_error_context_xact(TransactionId xid, XLogRecPtr lsn); static inline void reset_apply_error_context_info(void); +static void set_subscription_retry(bool retry); + /* * Should this worker apply changes for given relation. * @@ -905,6 +907,9 @@ apply_handle_commit(StringInfo s) apply_handle_commit_internal(&commit_data); + /* Reset the retry flag. */ + set_subscription_retry(false); + /* Check the status of apply background worker if any. */ apply_bgworker_check_status(); @@ -1016,6 +1021,9 @@ apply_handle_prepare(StringInfo s) in_remote_transaction = false; + /* Reset the retry flag. */ + set_subscription_retry(false); + /* Check the status of apply background worker if any. */ apply_bgworker_check_status(); @@ -1069,6 +1077,9 @@ apply_handle_commit_prepared(StringInfo s) store_flush_position(prepare_data.end_lsn); in_remote_transaction = false; + /* Reset the retry flag. */ + set_subscription_retry(false); + /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(prepare_data.end_lsn); @@ -1124,6 +1135,9 @@ apply_handle_rollback_prepared(StringInfo s) store_flush_position(rollback_data.rollback_end_lsn); in_remote_transaction = false; + /* Reset the retry flag. */ + set_subscription_retry(false); + /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(rollback_data.rollback_end_lsn); @@ -1218,6 +1232,9 @@ apply_handle_stream_prepare(StringInfo s) /* Unlink the files with serialized changes and subxact info. */ stream_cleanup_files(MyLogicalRepWorker->subid, prepare_data.xid); } + + /* Reset the retry flag. */ + set_subscription_retry(false); } in_remote_transaction = false; @@ -1646,6 +1663,9 @@ apply_handle_stream_abort(StringInfo s) */ serialize_stream_abort(xid, subxid); } + + /* Reset the retry flag. */ + set_subscription_retry(false); } reset_apply_error_context_info(); @@ -1858,6 +1878,9 @@ apply_handle_stream_commit(StringInfo s) /* Unlink the files with serialized changes and subxact info. */ stream_cleanup_files(MyLogicalRepWorker->subid, xid); } + + /* Reset the retry flag. */ + set_subscription_retry(false); } /* Check the status of apply background worker if any. */ @@ -3901,20 +3924,28 @@ start_table_sync(XLogRecPtr *origin_startpos, char **myslotname) } PG_CATCH(); { + /* + * Emit the error message, and recover from the error state to an idle + * state + */ + HOLD_INTERRUPTS(); + + EmitErrorReport(); + AbortOutOfAnyTransaction(); + FlushErrorState(); + + RESUME_INTERRUPTS(); + + /* Report the worker failed during table synchronization */ + pgstat_report_subscription_error(MySubscription->oid, false); + + /* Set the retry flag. */ + set_subscription_retry(true); + if (MySubscription->disableonerr) DisableSubscriptionAndExit(); - else - { - /* - * Report the worker failed during table synchronization. Abort - * the current transaction so that the stats message is sent in an - * idle state. - */ - AbortOutOfAnyTransaction(); - pgstat_report_subscription_error(MySubscription->oid, false); - PG_RE_THROW(); - } + proc_exit(0); } PG_END_TRY(); @@ -3939,20 +3970,27 @@ start_apply(XLogRecPtr origin_startpos) } PG_CATCH(); { + /* + * Emit the error message, and recover from the error state to an idle + * state + */ + HOLD_INTERRUPTS(); + + EmitErrorReport(); + AbortOutOfAnyTransaction(); + FlushErrorState(); + + RESUME_INTERRUPTS(); + + /* Report the worker failed while applying changes */ + pgstat_report_subscription_error(MySubscription->oid, + !am_tablesync_worker()); + + /* Set the retry flag. */ + set_subscription_retry(true); + if (MySubscription->disableonerr) DisableSubscriptionAndExit(); - else - { - /* - * Report the worker failed while applying changes. Abort the - * current transaction so that the stats message is sent in an - * idle state. - */ - AbortOutOfAnyTransaction(); - pgstat_report_subscription_error(MySubscription->oid, !am_tablesync_worker()); - - PG_RE_THROW(); - } } PG_END_TRY(); } @@ -4198,28 +4236,11 @@ ApplyWorkerMain(Datum main_arg) } /* - * After error recovery, disable the subscription in a new transaction - * and exit cleanly. + * Disable the subscription in a new transaction. */ static void DisableSubscriptionAndExit(void) { - /* - * Emit the error message, and recover from the error state to an idle - * state - */ - HOLD_INTERRUPTS(); - - EmitErrorReport(); - AbortOutOfAnyTransaction(); - FlushErrorState(); - - RESUME_INTERRUPTS(); - - /* Report the worker failed during either table synchronization or apply */ - pgstat_report_subscription_error(MyLogicalRepWorker->subid, - !am_tablesync_worker()); - /* Disable the subscription */ StartTransactionCommand(); DisableSubscription(MySubscription->oid); @@ -4229,8 +4250,6 @@ DisableSubscriptionAndExit(void) ereport(LOG, errmsg("logical replication subscription \"%s\" has been disabled due to an error", MySubscription->name)); - - proc_exit(0); } /* @@ -4465,3 +4484,63 @@ reset_apply_error_context_info(void) apply_error_callback_arg.remote_attnum = -1; set_apply_error_context_xact(InvalidTransactionId, InvalidXLogRecPtr); } + +/* + * Set subretry of pg_subscription catalog. + * + * If retry is true, subscriber is about to exit with an error. Otherwise, it + * means that the transaction was applied successfully. + */ +static void +set_subscription_retry(bool retry) +{ + Relation rel; + HeapTuple tup; + bool started_tx = false; + bool nulls[Natts_pg_subscription]; + bool replaces[Natts_pg_subscription]; + Datum values[Natts_pg_subscription]; + + if (MySubscription->retry == retry || + am_apply_bgworker()) + return; + + if (!IsTransactionState()) + { + StartTransactionCommand(); + started_tx = true; + } + + /* Look up the subscription in the catalog */ + rel = table_open(SubscriptionRelationId, RowExclusiveLock); + tup = SearchSysCacheCopy1(SUBSCRIPTIONOID, + ObjectIdGetDatum(MySubscription->oid)); + + if (!HeapTupleIsValid(tup)) + elog(ERROR, "subscription \"%s\" does not exist", MySubscription->name); + + LockSharedObject(SubscriptionRelationId, MySubscription->oid, 0, + AccessShareLock); + + /* Form a new tuple. */ + memset(values, 0, sizeof(values)); + memset(nulls, false, sizeof(nulls)); + memset(replaces, false, sizeof(replaces)); + + /* Reset subretry */ + values[Anum_pg_subscription_subretry - 1] = BoolGetDatum(retry); + replaces[Anum_pg_subscription_subretry - 1] = true; + + tup = heap_modify_tuple(tup, RelationGetDescr(rel), values, nulls, + replaces); + + /* Update the catalog. */ + CatalogTupleUpdate(rel, &tup->t_self, tup); + + /* Cleanup. */ + heap_freetuple(tup); + table_close(rel, NoLock); + + if (started_tx) + CommitTransactionCommand(); +} diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 9099fe5f74..f6e257d06d 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -4478,8 +4478,9 @@ getSubscriptions(Archive *fout) ntups = PQntuples(res); /* - * Get subscription fields. We don't include subskiplsn in the dump as - * after restoring the dump this value may no longer be relevant. + * Get subscription fields. We don't include subskiplsn and subretry in + * the dump as after restoring the dump this value may no longer be + * relevant. */ i_tableoid = PQfnumber(res, "tableoid"); i_oid = PQfnumber(res, "oid"); diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h index d54540f5f5..5f4e058ec1 100644 --- a/src/include/catalog/pg_subscription.h +++ b/src/include/catalog/pg_subscription.h @@ -76,6 +76,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW bool subdisableonerr; /* True if a worker error should cause the * subscription to be disabled */ + bool subretry BKI_DEFAULT(f); /* True if the previous apply change failed. */ + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* Connection string to the publisher */ text subconninfo BKI_FORCE_NOT_NULL; @@ -116,6 +118,8 @@ typedef struct Subscription bool disableonerr; /* Indicates if the subscription should be * automatically disabled if a worker error * occurs */ + bool retry; /* Indicates if the previous apply change + * failed. */ char *conninfo; /* Connection string to the publisher */ char *slotname; /* Name of the replication slot */ char *synccommit; /* Synchronous commit setting for worker */ diff --git a/src/test/subscription/t/032_streaming_apply.pl b/src/test/subscription/t/032_streaming_apply.pl index 7f8bfa6745..24c519a870 100644 --- a/src/test/subscription/t/032_streaming_apply.pl +++ b/src/test/subscription/t/032_streaming_apply.pl @@ -57,8 +57,13 @@ $node_subscriber->safe_psql( $node_publisher->wait_for_catchup($appname); +# ============================================================================ # It is not allowed that the unique index on the publisher and the subscriber -# is different. Check the error reported by background worker in this case. +# is different. Check the error reported by background worker in this case. And +# after retrying in apply worker, we check if the data is replicated +# successfully. +# ============================================================================ + # First we check the unique index on normal table. $node_subscriber->safe_psql('postgres', "CREATE UNIQUE INDEX test_tab_b_idx ON test_tab (b)"); @@ -71,14 +76,15 @@ $node_subscriber->wait_for_log( qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab" using subscription parameter streaming=parallel/, $offset); -# Drop the unique index on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', "DROP INDEX test_tab_b_idx"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); my $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); -is($result, qq(5000), 'data replicated to subscriber after dropping index'); +is($result, qq(5000), 'data replicated to subscribers after retrying because of unique index'); + +# Drop the unique index on the subscriber. +$node_subscriber->safe_psql('postgres', "DROP INDEX test_tab_b_idx"); # Then we check the unique index on partition table. $node_subscriber->safe_psql('postgres', @@ -95,17 +101,20 @@ $node_subscriber->wait_for_log( qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab_partitioned" using subscription parameter streaming=parallel/, $offset); -# Drop the unique index on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', "DROP INDEX test_tab_b_partition_idx"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); -is($result, qq(5000), 'data replicated to subscriber after dropping index'); +is($result, qq(5000), 'data replicated to subscribers after retrying because of unique index'); + +# Drop the unique index on the subscriber. +$node_subscriber->safe_psql('postgres', "DROP INDEX test_tab_b_partition_idx"); # Triggers which execute non-immutable function are not allowed on the # subscriber side. Check the error reported by background worker in this case. +# And after retrying in apply worker, we check if the data is replicated +# successfully. # First we check the trigger function on normal table. $node_subscriber->safe_psql( 'postgres', qq{ @@ -129,15 +138,16 @@ $node_subscriber->wait_for_log( qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab" using subscription parameter streaming=parallel/, $offset); -# Drop the trigger on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "DROP TRIGGER insert_trig ON test_tab"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); -is($result, qq(0), 'data replicated to subscriber after dropping trigger'); +is($result, qq(0), 'data replicated to subscribers after retrying because of trigger'); + +# Drop the trigger on the subscriber. +$node_subscriber->safe_psql('postgres', + "DROP TRIGGER insert_trig ON test_tab"); # Then we check the trigger function on partition table. $node_subscriber->safe_psql( @@ -157,19 +167,24 @@ $node_subscriber->wait_for_log( qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab_partitioned" using subscription parameter streaming=parallel/, $offset); -# Drop the trigger on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "DROP TRIGGER insert_trig ON test_tab_partition"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); -is($result, qq(0), 'data replicated to subscriber after dropping trigger'); +is($result, qq(0), 'data replicated to subscribers after retrying because of trigger'); +# Drop the trigger on the subscriber. +$node_subscriber->safe_psql('postgres', + "DROP TRIGGER insert_trig ON test_tab_partition"); + +# ============================================================================ # It is not allowed that column default value expression contains a # non-immutable function on the subscriber side. Check the error reported by -# background worker in this case. +# background worker in this case. And after retrying in apply worker, we check +# if the data is replicated successfully. +# ============================================================================ + # First we check the column default value expression on normal table. $node_subscriber->safe_psql('postgres', "ALTER TABLE test_tab ALTER COLUMN b SET DEFAULT random()"); @@ -185,16 +200,17 @@ $node_subscriber->wait_for_log( qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab" using subscription parameter streaming=parallel/, $offset); -# Drop default value on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "ALTER TABLE test_tab ALTER COLUMN b DROP DEFAULT"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); is($result, qq(5000), - 'data replicated to subscriber after dropping default value expression'); + 'data replicated to subscribers after retrying because of column default value'); + +# Drop default value on the subscriber. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab ALTER COLUMN b DROP DEFAULT"); # Then we check the column default value expression on partition table. $node_subscriber->safe_psql('postgres', @@ -211,20 +227,25 @@ $node_subscriber->wait_for_log( qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab_partitioned" using subscription parameter streaming=parallel/, $offset); -# Drop default value on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "ALTER TABLE test_tab_partition ALTER COLUMN b DROP DEFAULT"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); is($result, qq(5000), - 'data replicated to subscriber after dropping default value expression'); + 'data replicated to subscribers after retrying because of column default value'); + +# Drop default value on the subscriber. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partition ALTER COLUMN b DROP DEFAULT"); +# ============================================================================ # It is not allowed that domain constraint expression contains a non-immutable # function on the subscriber side. Check the error reported by background -# worker in this case. +# worker in this case. And after retrying in apply worker, we check if the data +# is replicated successfully. +# ============================================================================ + # Because the column type of the partition table must be the same as its parent # table, only test normal table here. $node_subscriber->safe_psql( @@ -242,21 +263,26 @@ $node_subscriber->wait_for_log( qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab" using subscription parameter streaming=parallel/, $offset); -# Drop domain constraint expression on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "ALTER TABLE test_tab ALTER COLUMN a TYPE int"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); is($result, qq(0), - 'data replicated to subscriber after dropping domain constraint expression' + 'data replicated to subscribers after retrying because of domain' ); -# It is not allowed that constraint expression contains a non-immutable function -# on the subscriber side. Check the error reported by background worker in this -# case. +# Drop domain constraint expression on the subscriber. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab ALTER COLUMN a TYPE int"); + +# ============================================================================ +# It is not allowed that constraint expression contains a non-immutable +# function on the subscriber side. Check the error reported by background +# worker in this case. And after retrying in apply worker, we check if the data +# is replicated successfully. +# ============================================================================ + # First we check the constraint expression on normal table. $node_subscriber->safe_psql( 'postgres', qq{ @@ -274,16 +300,17 @@ $node_subscriber->wait_for_log( qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab" using subscription parameter streaming=parallel/, $offset); -# Drop constraint on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "ALTER TABLE test_tab DROP CONSTRAINT test_tab_con"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); is($result, qq(5000), - 'data replicated to subscriber after dropping constraint expression'); + 'data replicated to subscribers after retrying because of constraint'); + +# Drop constraint on the subscriber. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab DROP CONSTRAINT test_tab_con"); # Then we check the constraint expression on partition table. $node_subscriber->safe_psql( @@ -300,19 +327,24 @@ $node_subscriber->wait_for_log( qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab_partitioned" using subscription parameter streaming=parallel/, $offset); -# Drop constraint on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "ALTER TABLE test_tab_partition DROP CONSTRAINT test_tab_con"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); is($result, qq(0), - 'data replicated to subscriber after dropping constraint expression'); + 'data replicated to subscribers after retrying because of constraint'); +# Drop constraint on the subscriber. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partition DROP CONSTRAINT test_tab_con"); + +# ============================================================================ # It is not allowed that foreign key on the subscriber side. Check the error -# reported by background worker in this case. +# reported by background worker in this case. And after retrying in apply +# worker, we check if the data is replicated successfully. +# ============================================================================ + # First we check the foreign key on normal table. $node_publisher->safe_psql('postgres', "DELETE FROM test_tab"); $node_publisher->wait_for_catchup($appname); @@ -333,16 +365,17 @@ $node_subscriber->wait_for_log( qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab" using subscription parameter streaming=parallel/, $offset); -# Drop the foreign key constraint on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "ALTER TABLE test_tab DROP CONSTRAINT test_tabfk"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); is($result, qq(5000), - 'data replicated to subscriber after dropping the foreign key'); + 'data replicated to subscribers after retrying because of foreign key'); + +# Drop the foreign key constraint on the subscriber. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab DROP CONSTRAINT test_tabfk"); # Then we check the foreign key on partition table. $node_publisher->wait_for_catchup($appname); @@ -363,16 +396,17 @@ $node_subscriber->wait_for_log( qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab_partitioned" using subscription parameter streaming=parallel/, $offset); -# Drop the foreign key constraint on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "ALTER TABLE test_tab_partition DROP CONSTRAINT test_tab_patition_fk"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); is($result, qq(5000), - 'data replicated to subscriber after dropping the foreign key'); + 'data replicated to subscribers after retrying because of foreign key'); + +# Drop the foreign key constraint on the subscriber. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partition DROP CONSTRAINT test_tab_patition_fk"); $node_subscriber->stop; $node_publisher->stop; -- 2.23.0.windows.1 ^ permalink raw reply [nested|flat] 66+ messages in thread
* RE: Perform streaming logical transactions by background workers and parallel apply @ 2022-07-22 02:56 [email protected] <[email protected]> parent: [email protected] <[email protected]> 0 siblings, 6 replies; 66+ messages in thread From: [email protected] @ 2022-07-22 02:56 UTC (permalink / raw) To: Peter Smith <[email protected]>; +Cc: Amit Kapila <[email protected]>; Masahiko Sawada <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; PostgreSQL Hackers <[email protected]> On Tues, Jul 19, 2022 at 10:29 AM I wrote: > Attach the news patches. Not able to apply patches cleanly because the change in HEAD (366283961a). Therefore, I rebased the patch based on the changes in HEAD. Attach the new patches. Regards, Wang wei Attachments: [application/octet-stream] v19-0001-Perform-streaming-logical-transactions-by-backgr.patch (106.6K, ../../OS3PR01MB62752B29CEC2592E18236A9A9E909@OS3PR01MB6275.jpnprd01.prod.outlook.com/2-v19-0001-Perform-streaming-logical-transactions-by-backgr.patch) download | inline diff: From 88fe01450d105be3b03c0a3e7bbefcda23df83b0 Mon Sep 17 00:00:00 2001 From: "houzj.fnst" <[email protected]> Date: Wed, 20 Apr 2022 16:45:07 +0800 Subject: [PATCH v19 1/4] Perform streaming logical transactions by background workers Currently, for large transactions, the publisher sends the data in multiple streams (changes divided into chunks depending upon logical_decoding_work_mem), and then on the subscriber-side, the apply worker writes the changes into temporary files and once it receives the commit, it reads from the file and applies the entire transaction. To improve the performance of such transactions, we can instead allow them to be applied via background workers. In this approach, we assign a new apply background worker (if available) as soon as the xact's first stream is received and the main apply worker will send changes to this new worker via shared memory. The apply background worker will directly apply the change instead of writing it to temporary files. We keep this worker assigned till the transaction commit is received and also wait for the worker to finish at commit. This preserves commit ordering and avoids writing to and reading from file in most cases. We still need to spill if there is no worker available. This patch also extends the SUBSCRIPTION 'streaming' parameter so that the user can control whether to apply the streaming transaction in an apply background worker or spill the change to disk. The user can set the streaming parameter to 'on/off', 'parallel'. The parameter value 'parallel' means the streaming will be applied via an apply background worker, if available. The parameter value 'on' means the streaming transaction will be spilled to disk. The default value is 'off' (same as current behaviour). --- doc/src/sgml/catalogs.sgml | 10 +- doc/src/sgml/config.sgml | 25 + doc/src/sgml/logical-replication.sgml | 10 + doc/src/sgml/protocol.sgml | 19 + doc/src/sgml/ref/create_subscription.sgml | 24 +- src/backend/access/transam/xact.c | 13 + src/backend/commands/subscriptioncmds.c | 66 +- src/backend/postmaster/bgworker.c | 3 + src/backend/replication/logical/Makefile | 1 + .../replication/logical/applybgworker.c | 802 ++++++++++++++++++ src/backend/replication/logical/decode.c | 10 +- src/backend/replication/logical/launcher.c | 130 ++- src/backend/replication/logical/origin.c | 26 +- src/backend/replication/logical/proto.c | 41 +- .../replication/logical/reorderbuffer.c | 10 +- src/backend/replication/logical/tablesync.c | 10 +- src/backend/replication/logical/worker.c | 692 +++++++++++---- src/backend/replication/pgoutput/pgoutput.c | 6 +- src/backend/utils/activity/wait_event.c | 3 + src/backend/utils/misc/guc.c | 12 + src/backend/utils/misc/postgresql.conf.sample | 1 + src/bin/pg_dump/pg_dump.c | 6 +- src/include/catalog/pg_subscription.h | 21 +- src/include/replication/logicallauncher.h | 1 + src/include/replication/logicalproto.h | 27 +- src/include/replication/logicalworker.h | 1 + src/include/replication/origin.h | 2 +- src/include/replication/reorderbuffer.h | 7 +- src/include/replication/worker_internal.h | 102 ++- src/include/utils/wait_event.h | 1 + src/test/regress/expected/subscription.out | 2 +- src/tools/pgindent/typedefs.list | 5 + 32 files changed, 1869 insertions(+), 220 deletions(-) create mode 100644 src/backend/replication/logical/applybgworker.c diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index a186e35f00..099b3c9661 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -7873,11 +7873,15 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l <row> <entry role="catalog_table_entry"><para role="column_definition"> - <structfield>substream</structfield> <type>bool</type> + <structfield>substream</structfield> <type>char</type> </para> <para> - If true, the subscription will allow streaming of in-progress - transactions + Controls how to handle the streaming of in-progress transactions: + <literal>f</literal> = disallow streaming of in-progress transactions, + <literal>t</literal> = spill the changes of in-progress transactions to + disk and apply at once after the transaction is committed on the + publisher, + <literal>p</literal> = apply changes directly using a background worker </para></entry> </row> diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml index e2d728e0c4..a926b530ea 100644 --- a/doc/src/sgml/config.sgml +++ b/doc/src/sgml/config.sgml @@ -4970,6 +4970,31 @@ ANY <replaceable class="parameter">num_sync</replaceable> ( <replaceable class=" </listitem> </varlistentry> + <varlistentry id="guc-max-apply-bgworkers-per-subscription" xreflabel="max_apply_bgworkers_per_subscription"> + <term><varname>max_apply_bgworkers_per_subscription</varname> (<type>integer</type>) + <indexterm> + <primary><varname>max_apply_bgworkers_per_subscription</varname> configuration parameter</primary> + </indexterm> + </term> + <listitem> + <para> + Maximum number of apply background workers per subscription. This + parameter controls the amount of parallelism of the streaming of + in-progress transactions when subscription parameter + <literal>streaming = parallel</literal>. + </para> + <para> + The apply background workers are taken from the pool defined by + <varname>max_logical_replication_workers</varname>. + </para> + <para> + The default value is 2. This parameter can only be set in the + <filename>postgresql.conf</filename> file or on the server command + line. + </para> + </listitem> + </varlistentry> + </variablelist> </sect2> diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml index bdf1e7b727..92997f9299 100644 --- a/doc/src/sgml/logical-replication.sgml +++ b/doc/src/sgml/logical-replication.sgml @@ -1153,6 +1153,16 @@ CONTEXT: processing remote data for replication origin "pg_16395" during "INSER might not violate any constraint. This can easily make the subscriber inconsistent. </para> + + <para> + When the streaming mode is <literal>parallel</literal>, the finish LSN of + failed transactions may not be logged. In that case, it may be necessary to + change the streaming mode to <literal>on</literal> and cause the same + conflicts again so the finish LSN of the failed transaction will be written + to the server log. For the usage of finish LSN, please refer to <link + linkend="sql-altersubscription"><command>ALTER SUBSCRIPTION ... + SKIP</command></link>. + </para> </sect1> <sect1 id="logical-replication-restrictions"> diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml index c0b89a3c01..7e88ba9631 100644 --- a/doc/src/sgml/protocol.sgml +++ b/doc/src/sgml/protocol.sgml @@ -6809,6 +6809,25 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;" </listitem> </varlistentry> + <varlistentry> + <term>Int64 (XLogRecPtr)</term> + <listitem> + <para> + The LSN of the abort. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Int64 (TimestampTz)</term> + <listitem> + <para> + Abort timestamp of the transaction. The value is in number + of microseconds since PostgreSQL epoch (2000-01-01). + </para> + </listitem> + </varlistentry> + <varlistentry> <term>Int32 (TransactionId)</term> <listitem> diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml index 7390c715bc..b08e4b5580 100644 --- a/doc/src/sgml/ref/create_subscription.sgml +++ b/doc/src/sgml/ref/create_subscription.sgml @@ -217,13 +217,29 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl </varlistentry> <varlistentry> - <term><literal>streaming</literal> (<type>boolean</type>)</term> + <term><literal>streaming</literal> (<type>enum</type>)</term> <listitem> <para> Specifies whether to enable streaming of in-progress transactions - for this subscription. By default, all transactions - are fully decoded on the publisher and only then sent to the - subscriber as a whole. + for this subscription. The default value is <literal>off</literal>, + meaning all transactions are fully decoded on the publisher and only + then sent to the subscriber as a whole. + </para> + + <para> + If set to <literal>on</literal>, the incoming changes are written to + temporary files and then applied only after the transaction is + committed on the publisher. + </para> + + <para> + If set to <literal>parallel</literal>, incoming changes are directly + applied via one of the apply background workers, if available. If no + background worker is free to handle streaming transaction then the + changes are written to temporary files and applied after the + transaction is committed. Note that if an error happens when + applying changes in a background worker, the finish LSN of the + remote transaction might not be reported in the server log. </para> </listitem> </varlistentry> diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c index 116de1175b..3e61a57b50 100644 --- a/src/backend/access/transam/xact.c +++ b/src/backend/access/transam/xact.c @@ -1711,6 +1711,7 @@ RecordTransactionAbort(bool isSubXact) int nchildren; TransactionId *children; TimestampTz xact_time; + bool replorigin; /* * If we haven't been assigned an XID, nobody will care whether we aborted @@ -1741,6 +1742,13 @@ RecordTransactionAbort(bool isSubXact) elog(PANIC, "cannot abort transaction %u, it was already committed", xid); + /* + * Are we using the replication origins feature? Or, in other words, + * are we replaying remote actions? + */ + replorigin = (replorigin_session_origin != InvalidRepOriginId && + replorigin_session_origin != DoNotReplicateId); + /* Fetch the data we need for the abort record */ nrels = smgrGetPendingDeletes(false, &rels); nchildren = xactGetCommittedChildren(&children); @@ -1765,6 +1773,11 @@ RecordTransactionAbort(bool isSubXact) MyXactFlags, InvalidTransactionId, NULL); + if (replorigin) + /* Move LSNs forward for this replication origin */ + replorigin_session_advance(replorigin_session_origin_lsn, + XactLastRecEnd); + /* * Report the latest async abort LSN, so that the WAL writer knows to * flush this abort. There's nothing to be gained by delaying this, since diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c index bd0cc0848d..9697128414 100644 --- a/src/backend/commands/subscriptioncmds.c +++ b/src/backend/commands/subscriptioncmds.c @@ -84,7 +84,7 @@ typedef struct SubOpts bool copy_data; bool refresh; bool binary; - bool streaming; + char streaming; bool twophase; bool disableonerr; char *origin; @@ -97,6 +97,62 @@ static List *merge_publications(List *oldpublist, List *newpublist, bool addpub, static void ReportSlotConnectionError(List *rstates, Oid subid, char *slotname, char *err); +/* + * Extract the streaming mode value from a DefElem. This is like + * defGetBoolean() but also accepts the special value of "parallel". + */ +static char +defGetStreamingMode(DefElem *def) +{ + /* + * If no parameter value given, assume "true" is meant. + */ + if (def->arg == NULL) + return SUBSTREAM_ON; + + /* + * Allow 0, 1, "false", "true", "off", "on" or "parallel". + */ + switch (nodeTag(def->arg)) + { + case T_Integer: + switch (intVal(def->arg)) + { + case 0: + return SUBSTREAM_OFF; + case 1: + return SUBSTREAM_ON; + default: + /* otherwise, error out below */ + break; + } + break; + default: + { + char *sval = defGetString(def); + + /* + * The set of strings accepted here should match up with the + * grammar's opt_boolean_or_string production. + */ + if (pg_strcasecmp(sval, "false") == 0 || + pg_strcasecmp(sval, "off") == 0) + return SUBSTREAM_OFF; + if (pg_strcasecmp(sval, "true") == 0 || + pg_strcasecmp(sval, "on") == 0) + return SUBSTREAM_ON; + if (pg_strcasecmp(sval, "parallel") == 0) + return SUBSTREAM_PARALLEL; + } + break; + } + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s requires a Boolean value or \"parallel\"", + def->defname))); + return SUBSTREAM_OFF; /* keep compiler quiet */ +} + /* * Common option parsing function for CREATE and ALTER SUBSCRIPTION commands. * @@ -134,7 +190,7 @@ parse_subscription_options(ParseState *pstate, List *stmt_options, if (IsSet(supported_opts, SUBOPT_BINARY)) opts->binary = false; if (IsSet(supported_opts, SUBOPT_STREAMING)) - opts->streaming = false; + opts->streaming = SUBSTREAM_OFF; if (IsSet(supported_opts, SUBOPT_TWOPHASE_COMMIT)) opts->twophase = false; if (IsSet(supported_opts, SUBOPT_DISABLE_ON_ERR)) @@ -237,7 +293,7 @@ parse_subscription_options(ParseState *pstate, List *stmt_options, errorConflictingDefElem(defel, pstate); opts->specified_opts |= SUBOPT_STREAMING; - opts->streaming = defGetBoolean(defel); + opts->streaming = defGetStreamingMode(defel); } else if (strcmp(defel->defname, "two_phase") == 0) { @@ -627,7 +683,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt, values[Anum_pg_subscription_subowner - 1] = ObjectIdGetDatum(owner); values[Anum_pg_subscription_subenabled - 1] = BoolGetDatum(opts.enabled); values[Anum_pg_subscription_subbinary - 1] = BoolGetDatum(opts.binary); - values[Anum_pg_subscription_substream - 1] = BoolGetDatum(opts.streaming); + values[Anum_pg_subscription_substream - 1] = CharGetDatum(opts.streaming); values[Anum_pg_subscription_subtwophasestate - 1] = CharGetDatum(opts.twophase ? LOGICALREP_TWOPHASE_STATE_PENDING : @@ -1089,7 +1145,7 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt, if (IsSet(opts.specified_opts, SUBOPT_STREAMING)) { values[Anum_pg_subscription_substream - 1] = - BoolGetDatum(opts.streaming); + CharGetDatum(opts.streaming); replaces[Anum_pg_subscription_substream - 1] = true; } diff --git a/src/backend/postmaster/bgworker.c b/src/backend/postmaster/bgworker.c index 40601aefd9..40ccb8993c 100644 --- a/src/backend/postmaster/bgworker.c +++ b/src/backend/postmaster/bgworker.c @@ -128,6 +128,9 @@ static const struct }, { "ApplyWorkerMain", ApplyWorkerMain + }, + { + "ApplyBgworkerMain", ApplyBgworkerMain } }; diff --git a/src/backend/replication/logical/Makefile b/src/backend/replication/logical/Makefile index c4e2fdeb71..cbfb5d794e 100644 --- a/src/backend/replication/logical/Makefile +++ b/src/backend/replication/logical/Makefile @@ -15,6 +15,7 @@ include $(top_builddir)/src/Makefile.global override CPPFLAGS := -I$(srcdir) $(CPPFLAGS) OBJS = \ + applybgworker.o \ decode.o \ launcher.o \ logical.o \ diff --git a/src/backend/replication/logical/applybgworker.c b/src/backend/replication/logical/applybgworker.c new file mode 100644 index 0000000000..aa222490a0 --- /dev/null +++ b/src/backend/replication/logical/applybgworker.c @@ -0,0 +1,802 @@ +/*------------------------------------------------------------------------- + * applybgworker.c + * Support routines for applying xact by apply background worker + * + * Copyright (c) 2016-2022, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/replication/logical/applybgworker.c + * + * This file contains routines that are intended to support setting up, using, + * and tearing down a ApplyBgworkerState. + * + * Refer to the comments in file header of logical/worker.c to see more + * information about apply background worker. + * + *------------------------------------------------------------------------- + */ + +#include "postgres.h" + +#include "libpq/pqformat.h" +#include "mb/pg_wchar.h" +#include "pgstat.h" +#include "postmaster/interrupt.h" +#include "replication/logicallauncher.h" +#include "replication/logicalworker.h" +#include "replication/origin.h" +#include "replication/walreceiver.h" +#include "replication/worker_internal.h" +#include "storage/ipc.h" +#include "storage/procarray.h" +#include "tcop/tcopprot.h" +#include "utils/inval.h" +#include "utils/memutils.h" +#include "utils/resowner.h" +#include "utils/syscache.h" + +#define PG_LOGICAL_APPLY_SHM_MAGIC 0x79fb2447 // TODO Consider change + +/* + * DSM keys for apply background worker. Unlike other parallel execution code, + * since we don't need to worry about DSM keys conflicting with plan_node_id we + * can use small integers. + */ +#define APPLY_BGWORKER_KEY_SHARED 1 +#define APPLY_BGWORKER_KEY_MQ 2 + +/* Queue size of DSM, 16 MB for now. */ +#define DSM_QUEUE_SIZE 160000000 + +/* + * There are three fields in message: start_lsn, end_lsn and send_time. Because + * we have updated these statistics in apply worker, we could ignore these + * fields in apply background worker. (see function LogicalRepApplyLoop) + */ +#define IGNORE_SIZE_IN_MESSAGE (3 * sizeof(uint64)) + +/* + * Entry for a hash table we use to map from xid to our apply background worker + * state. + */ +typedef struct ApplyBgworkerEntry +{ + TransactionId xid; + ApplyBgworkerState *wstate; +} ApplyBgworkerEntry; + +/* Apply background workers hash table (initialized on first use) */ +static HTAB *ApplyWorkersHash = NULL; +static List *ApplyWorkersFreeList = NIL; +static List *ApplyWorkersList = NIL; + +/* + * Information shared between main apply worker and apply background worker. + */ +volatile ApplyBgworkerShared *MyParallelShared = NULL; + +List *subxactlist = NIL; + +static bool apply_bgworker_can_start(TransactionId xid); +static ApplyBgworkerState *apply_bgworker_setup(void); +static void apply_bgworker_setup_dsm(ApplyBgworkerState *wstate); + +/* + * Check if starting a new apply background worker is allowed. + */ +static bool +apply_bgworker_can_start(TransactionId xid) +{ + if (!TransactionIdIsValid(xid)) + return false; + + /* + * Don't start a new background worker if not in streaming parallel mode. + */ + if (MySubscription->stream != SUBSTREAM_PARALLEL) + return false; + + /* + * Don't start a new background worker if user has set skiplsn as it's + * possible that user want to skip the streaming transaction. For + * streaming transaction, we need to spill the transaction to disk so that + * we can get the last LSN of the transaction to judge whether to skip + * before starting to apply the change. + */ + if (!XLogRecPtrIsInvalid(MySubscription->skiplsn)) + return false; + + /* + * For streaming transactions that are being applied in apply background + * worker, we cannot decide whether to apply the change for a relation + * that is not in the READY state (see should_apply_changes_for_rel) as we + * won't know remote_final_lsn by that time. So, we don't start new apply + * background worker in this case. + */ + if (!AllTablesyncsReady()) + return false; + + return true; +} + +/* + * Try to start an apply background worker and, if successful, cache it in + * ApplyWorkersHash keyed by the specified xid. + */ +ApplyBgworkerState * +apply_bgworker_start(TransactionId xid) +{ + bool found; + int server_version; + ApplyBgworkerState *wstate; + ApplyBgworkerEntry *entry = NULL; + + if (!apply_bgworker_can_start(xid)) + return NULL; + + /* First time through, initialize apply workers hashtable */ + if (ApplyWorkersHash == NULL) + { + HASHCTL ctl; + + MemSet(&ctl, 0, sizeof(ctl)); + ctl.keysize = sizeof(TransactionId); + ctl.entrysize = sizeof(ApplyBgworkerEntry); + ctl.hcxt = ApplyContext; + + ApplyWorkersHash = hash_create("logical apply workers hash", 8, &ctl, + HASH_ELEM | HASH_BLOBS | HASH_CONTEXT); + } + + /* + * Now, we try to get an apply background worker. If there is at least one + * worker in the free list, then take one. Otherwise, we try to start a + * new apply background worker. + */ + if (list_length(ApplyWorkersFreeList) > 0) + { + wstate = (ApplyBgworkerState *) llast(ApplyWorkersFreeList); + ApplyWorkersFreeList = list_delete_last(ApplyWorkersFreeList); + Assert(wstate->shared->status == APPLY_BGWORKER_FINISHED); + } + else + { + wstate = apply_bgworker_setup(); + + if (wstate == NULL) + return NULL; + } + + /* + * Create entry for requested transaction. + */ + entry = hash_search(ApplyWorkersHash, &xid, HASH_ENTER, &found); + if (found) + elog(ERROR, "hash table corrupted"); + + /* Fill up the hash entry */ + wstate->shared->status = APPLY_BGWORKER_BUSY; + + server_version = walrcv_server_version(LogRepWorkerWalRcvConn); + wstate->shared->server_version = + server_version >= 160000 ? LOGICALREP_PROTO_STREAM_PARALLEL_VERSION_NUM : + server_version >= 150000 ? LOGICALREP_PROTO_TWOPHASE_VERSION_NUM : + server_version >= 140000 ? LOGICALREP_PROTO_STREAM_VERSION_NUM : + LOGICALREP_PROTO_VERSION_NUM; + + wstate->shared->stream_xid = xid; + entry->wstate = wstate; + entry->xid = xid; + + return wstate; +} + +/* + * Try to look up worker inside ApplyWorkersHash for requested xid. + */ +ApplyBgworkerState * +apply_bgworker_find(TransactionId xid) +{ + bool found; + ApplyBgworkerEntry *entry = NULL; + + if (!TransactionIdIsValid(xid)) + return NULL; + + if (ApplyWorkersHash == NULL) + return NULL; + + /* + * Find entry for requested transaction. + */ + entry = hash_search(ApplyWorkersHash, &xid, HASH_FIND, &found); + if (found) + { + char status = entry->wstate->shared->status; + + /* If any workers (or the postmaster) have died, we have failed. */ + if (status == APPLY_BGWORKER_EXIT) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("background worker %u failed to apply transaction %u", + entry->wstate->shared->n, + entry->wstate->shared->stream_xid))); + + Assert(status == APPLY_BGWORKER_BUSY); + + return entry->wstate; + } + else + return NULL; +} + +/* + * Add the worker to the free list and remove the entry from the hash table. + */ +void +apply_bgworker_free(ApplyBgworkerState *wstate) +{ + MemoryContext oldctx; + TransactionId xid = wstate->shared->stream_xid; + + Assert(wstate->shared->status == APPLY_BGWORKER_FINISHED); + + oldctx = MemoryContextSwitchTo(ApplyContext); + + hash_search(ApplyWorkersHash, &xid, HASH_REMOVE, NULL); + + elog(DEBUG1, "adding finished apply worker #%u for xid %u to the free list", + wstate->shared->n, wstate->shared->stream_xid); + + ApplyWorkersFreeList = lappend(ApplyWorkersFreeList, wstate); + + MemoryContextSwitchTo(oldctx); +} + +/* Apply Background Worker main loop */ +static void +LogicalApplyBgwLoop(shm_mq_handle *mqh, volatile ApplyBgworkerShared *shared) +{ + shm_mq_result shmq_res; + PGPROC *registrant; + ErrorContextCallback errcallback; + + registrant = BackendPidGetProc(MyBgworkerEntry->bgw_notify_pid); + SetLatch(®istrant->procLatch); + + /* + * Push apply error context callback. Fields will be filled applying a + * change. + */ + errcallback.callback = apply_error_callback; + errcallback.previous = error_context_stack; + error_context_stack = &errcallback; + + for (;;) + { + void *data; + Size len; + int c; + StringInfoData s; + MemoryContext oldctx; + + CHECK_FOR_INTERRUPTS(); + + /* Ensure we are reading the data into our memory context. */ + oldctx = MemoryContextSwitchTo(ApplyMessageContext); + + shmq_res = shm_mq_receive(mqh, &len, &data, false); + + if (shmq_res != SHM_MQ_SUCCESS) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("lost connection to the main apply worker"))); + + if (len == 0) + break; + + s.cursor = 0; + s.maxlen = -1; + s.data = (char *) data; + s.len = len; + + /* + * We use first byte of message for additional communication between + * main Logical replication worker and apply background workers, so if + * it differs from 'w', then process it first. + */ + c = pq_getmsgbyte(&s); + switch (c) + { + /* End message of streaming chunk */ + case LOGICAL_REP_MSG_STREAM_STOP: + elog(DEBUG1, "[Apply BGW #%u] ended processing streaming chunk," + "waiting on shm_mq_receive", shared->n); + + in_streamed_transaction = false; + pgstat_report_activity(STATE_IDLEINTRANSACTION, NULL); + continue; + case 'w': + break; + default: + elog(ERROR, "[Apply BGW #%u] unexpected message \"%c\"", + shared->n, c); + break; + } + + /* Ignore statistics fields that have been updated. */ + s.cursor += IGNORE_SIZE_IN_MESSAGE; + + apply_dispatch(&s); + + if (ConfigReloadPending) + { + ConfigReloadPending = false; + ProcessConfigFile(PGC_SIGHUP); + } + + MemoryContextSwitchTo(oldctx); + MemoryContextReset(ApplyMessageContext); + } + + MemoryContextSwitchTo(TopMemoryContext); + MemoryContextReset(ApplyContext); + + /* Pop the error context stack */ + error_context_stack = errcallback.previous; + + elog(DEBUG1, "[Apply BGW #%u] exiting", shared->n); + + /* Signal main process that we are done. */ + SetLatch(®istrant->procLatch); +} + +/* + * Set the exit status so that the main apply worker can realize we have + * shutdown. + */ +static void +apply_bgworker_shutdown(int code, Datum arg) +{ + SpinLockAcquire(&MyParallelShared->mutex); + MyParallelShared->status = APPLY_BGWORKER_EXIT; + SpinLockRelease(&MyParallelShared->mutex); + + dsm_detach((dsm_segment *) DatumGetPointer(arg)); +} + +/* + * Apply Background Worker entry point + */ +void +ApplyBgworkerMain(Datum main_arg) +{ + volatile ApplyBgworkerShared *shared; + + dsm_handle handle; + dsm_segment *seg; + shm_toc *toc; + shm_mq *mq; + shm_mq_handle *mqh; + MemoryContext oldcontext; + RepOriginId originid; + int worker_slot = DatumGetInt32(main_arg); + char originname[NAMEDATALEN]; + + MemoryContextSwitchTo(TopMemoryContext); + + /* Init the memory context for the apply background worker to work in. */ + ApplyContext = AllocSetContextCreate(TopMemoryContext, + "ApplyContext", + ALLOCSET_DEFAULT_SIZES); + + /* + * Init the ApplyMessageContext which we clean up after each replication + * protocol message. + */ + ApplyMessageContext = AllocSetContextCreate(ApplyContext, + "ApplyMessageContext", + ALLOCSET_DEFAULT_SIZES); + + /* Setup signal handling */ + pqsignal(SIGHUP, SignalHandlerForConfigReload); + pqsignal(SIGTERM, die); + BackgroundWorkerUnblockSignals(); + + /* + * Connect to the dynamic shared memory segment. + * + * The backend that registered this worker passed us the ID of a shared + * memory segment to which we must attach for further instructions. In + * order to attach to dynamic shared memory, we need a resource owner. + * Once we've mapped the segment in our address space, attach to the table + * of contents so we can locate the various data structures we'll need to + * find within the segment. + */ + CurrentResourceOwner = ResourceOwnerCreate(NULL, "Logical apply worker"); + memcpy(&handle, MyBgworkerEntry->bgw_extra, sizeof(dsm_handle)); + seg = dsm_attach(handle); + if (seg == NULL) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("unable to map dynamic shared memory segment"))); + toc = shm_toc_attach(PG_LOGICAL_APPLY_SHM_MAGIC, dsm_segment_address(seg)); + if (toc == NULL) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("bad magic number in dynamic shared memory segment"))); + + before_shmem_exit(apply_bgworker_shutdown, PointerGetDatum(seg)); + + /* Look up the shared information. */ + shared = shm_toc_lookup(toc, APPLY_BGWORKER_KEY_SHARED, false); + MyParallelShared = shared; + + /* + * Attach to the message queue. + */ + mq = shm_toc_lookup(toc, APPLY_BGWORKER_KEY_MQ, false); + shm_mq_set_receiver(mq, MyProc); + mqh = shm_mq_attach(mq, seg, NULL); + + /* Run as replica session replication role. */ + SetConfigOption("session_replication_role", "replica", + PGC_SUSET, PGC_S_OVERRIDE); + + /* + * Now, we have initialized DSM. Attach to slot. + */ + logicalrep_worker_attach(worker_slot); + + /* Connect to our database. */ + BackgroundWorkerInitializeConnectionByOid(MyLogicalRepWorker->dbid, + MyLogicalRepWorker->userid, + 0); + + /* + * Set always-secure search path, so malicious users can't redirect user + * code (e.g. pg_index.indexprs). + */ + SetConfigOption("search_path", "", PGC_SUSET, PGC_S_OVERRIDE); + + /* + * Set the client encoding to the database encoding, since that is what + * the leader will expect. + */ + SetClientEncoding(GetDatabaseEncoding()); + + stream_xid = shared->stream_xid; + + StartTransactionCommand(); + oldcontext = MemoryContextSwitchTo(ApplyContext); + + MySubscription = GetSubscription(MyLogicalRepWorker->subid, true); + if (!MySubscription) + { + ereport(LOG, + (errmsg("logical replication apply worker for subscription %u will not " + "start because the subscription was removed during startup", + MyLogicalRepWorker->subid))); + proc_exit(0); + } + + MySubscriptionValid = true; + MemoryContextSwitchTo(oldcontext); + + /* Setup synchronous commit according to the user's wishes */ + SetConfigOption("synchronous_commit", MySubscription->synccommit, + PGC_BACKEND, PGC_S_OVERRIDE); + + /* Keep us informed about subscription changes. */ + CacheRegisterSyscacheCallback(SUBSCRIPTIONOID, + subscription_change_cb, + (Datum) 0); + + CommitTransactionCommand(); + + /* Setup replication origin tracking. */ + StartTransactionCommand(); + snprintf(originname, sizeof(originname), "pg_%u", MySubscription->oid); + originid = replorigin_by_name(originname, true); + if (!OidIsValid(originid)) + originid = replorigin_create(originname); + + /* + * The apply background worker doesn't need to monopolize this replication + * origin which was already acquired by its leader process. + */ + replorigin_session_setup(originid, false); + replorigin_session_origin = originid; + CommitTransactionCommand(); + + /* + * Allocate the origin name in long-lived context for error context + * message. + */ + apply_error_callback_arg.origin_name = MemoryContextStrdup(ApplyContext, + originname); + + elog(DEBUG1, "[Apply BGW #%u] started", shared->n); + + LogicalApplyBgwLoop(mqh, shared); + + /* + * We're done. Explicitly detach the shared memory segment so that we + * don't get a resource leak warning at commit time. This will fire any + * on_dsm_detach callbacks we've registered, as well. Once that's done, + * we can go ahead and exit. + */ + dsm_detach(seg); + proc_exit(0); +} + +/* + * Set up a dynamic shared memory segment. + * + * We set up a control region that contains a ApplyBgworkerShared, + * plus one region per message queue. There are as many message queues as + * the number of workers. + */ +static void +apply_bgworker_setup_dsm(ApplyBgworkerState *wstate) +{ + shm_toc_estimator e; + Size segsize; + dsm_segment *seg; + shm_toc *toc; + ApplyBgworkerShared *shared; + shm_mq *mq; + int64 queue_size = DSM_QUEUE_SIZE; + int server_version; + + /* + * Estimate how much shared memory we need. + * + * Because the TOC machinery may choose to insert padding of oddly-sized + * requests, we must estimate each chunk separately. + * + * We need one key to register the location of the header, and we need + * another key to track the location of the message queue. + */ + shm_toc_initialize_estimator(&e); + shm_toc_estimate_chunk(&e, sizeof(ApplyBgworkerShared)); + shm_toc_estimate_chunk(&e, (Size) queue_size); + + shm_toc_estimate_keys(&e, 1 + 1); + segsize = shm_toc_estimate(&e); + + /* Create the shared memory segment and establish a table of contents. */ + seg = dsm_create(shm_toc_estimate(&e), 0); + toc = shm_toc_create(PG_LOGICAL_APPLY_SHM_MAGIC, dsm_segment_address(seg), + segsize); + + /* Set up the header region. */ + shared = shm_toc_allocate(toc, sizeof(ApplyBgworkerShared)); + SpinLockInit(&shared->mutex); + shared->status = APPLY_BGWORKER_BUSY; + + server_version = walrcv_server_version(LogRepWorkerWalRcvConn); + shared->server_version = + server_version >= 160000 ? LOGICALREP_PROTO_STREAM_PARALLEL_VERSION_NUM : + server_version >= 150000 ? LOGICALREP_PROTO_TWOPHASE_VERSION_NUM : + server_version >= 140000 ? LOGICALREP_PROTO_STREAM_VERSION_NUM : + LOGICALREP_PROTO_VERSION_NUM; + + shared->stream_xid = stream_xid; + shared->n = list_length(ApplyWorkersList) + 1; + + shm_toc_insert(toc, APPLY_BGWORKER_KEY_SHARED, shared); + + /* Set up message queue for the worker. */ + mq = shm_mq_create(shm_toc_allocate(toc, (Size) queue_size), + (Size) queue_size); + shm_toc_insert(toc, APPLY_BGWORKER_KEY_MQ, mq); + shm_mq_set_sender(mq, MyProc); + + /* Attach the queue. */ + wstate->mq_handle = shm_mq_attach(mq, seg, NULL); + + /* Return results to caller. */ + wstate->dsm_seg = seg; + wstate->shared = shared; +} + +/* + * Start apply background worker process and allocate shared memory for it. + */ +static ApplyBgworkerState * +apply_bgworker_setup(void) +{ + MemoryContext oldcontext; + bool launched; + ApplyBgworkerState *wstate; + int napplyworkers; + + elog(DEBUG1, "setting up apply worker #%u", list_length(ApplyWorkersList) + 1); + + /* Check if there are free worker slot(s) */ + LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); + napplyworkers = logicalrep_apply_bgworker_count(MyLogicalRepWorker->subid); + LWLockRelease(LogicalRepWorkerLock); + if (napplyworkers >= max_apply_bgworkers_per_subscription) + return NULL; + + oldcontext = MemoryContextSwitchTo(ApplyContext); + + wstate = (ApplyBgworkerState *) palloc0(sizeof(ApplyBgworkerState)); + + /* Setup shared memory */ + apply_bgworker_setup_dsm(wstate); + + launched = logicalrep_worker_launch(MyLogicalRepWorker->dbid, + MySubscription->oid, + MySubscription->name, + MyLogicalRepWorker->userid, + InvalidOid, + dsm_segment_handle(wstate->dsm_seg)); + + if (launched) + ApplyWorkersList = lappend(ApplyWorkersList, wstate); + else + { + dsm_detach(wstate->dsm_seg); + wstate->dsm_seg = NULL; + + pfree(wstate); + wstate = NULL; + } + + MemoryContextSwitchTo(oldcontext); + + return wstate; +} + +/* + * Send the data to the specified apply background worker via shared-memory queue. + */ +void +apply_bgworker_send_data(ApplyBgworkerState *wstate, Size nbytes, const void *data) +{ + shm_mq_result result; + + result = shm_mq_send(wstate->mq_handle, nbytes, data, false, true); + + if (result != SHM_MQ_SUCCESS) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("could not send tuples to shared-memory queue"))); +} + +/* + * Wait until the status of apply background worker reaches the + * 'wait_for_status' + */ +void +apply_bgworker_wait_for(ApplyBgworkerState *wstate, + ApplyBgworkerStatus wait_for_status) +{ + for (;;) + { + char status; + + SpinLockAcquire(&wstate->shared->mutex); + status = wstate->shared->status; + SpinLockRelease(&wstate->shared->mutex); + + /* Done if already in correct status. */ + if (status == wait_for_status) + break; + + /* If any workers (or the postmaster) have died, we have failed. */ + if (status == APPLY_BGWORKER_EXIT) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("background worker %u failed to apply transaction %u", + wstate->shared->n, wstate->shared->stream_xid))); + + /* Wait to be signalled. */ + WaitLatch(MyLatch, WL_LATCH_SET | WL_EXIT_ON_PM_DEATH, 0, + WAIT_EVENT_LOGICAL_APPLY_WORKER_STATE_CHANGE); + + /* Reset the latch so we don't spin. */ + ResetLatch(MyLatch); + + /* An interrupt may have occurred while we were waiting. */ + CHECK_FOR_INTERRUPTS(); + } +} + +/* + * Check the status of workers and report an error if any apply background + * worker has exited unexpectedly. + */ +void +apply_bgworker_check_status(void) +{ + ListCell *lc; + + if (am_apply_bgworker() || MySubscription->stream != SUBSTREAM_PARALLEL) + return; + + foreach(lc, ApplyWorkersList) + { + ApplyBgworkerState *wstate = (ApplyBgworkerState *) lfirst(lc); + + /* + * We don't lock here as in the worst case we will just detect the + * failure of worker a bit later. + */ + if (wstate->shared->status == APPLY_BGWORKER_EXIT) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("background worker %u exited unexpectedly", + wstate->shared->n))); + } + + /* + * Exit if any relation is not in the READY state and if any worker is + * handling the streaming transaction at the same time. Because for + * streaming transactions that is being applied in apply background + * worker, we cannot decide whether to apply the change for a relation + * that is not in the READY state (see should_apply_changes_for_rel) as we + * won't know remote_final_lsn by that time. + */ + if (list_length(ApplyWorkersFreeList) != list_length(ApplyWorkersList) && + !AllTablesyncsReady()) + { + ereport(LOG, + (errmsg("logical replication apply workers for subscription \"%s\" will restart", + MySubscription->name), + errdetail("Cannot handle streamed replication transaction by apply " + "background workers until all tables are synchronized"))); + + proc_exit(0); + } +} + +/* Set the apply background worker status */ +void +apply_bgworker_set_status(ApplyBgworkerStatus status) +{ + if (!am_apply_bgworker()) + return; + + elog(DEBUG1, "[Apply BGW #%u] set status to %d", MyParallelShared->n, status); + + SpinLockAcquire(&MyParallelShared->mutex); + MyParallelShared->status = status; + SpinLockRelease(&MyParallelShared->mutex); +} + +/* + * Define a savepoint for a subxact in apply background worker if needed. + * + * Inside apply background worker we can figure out that new subtransaction was + * started if new change arrived with different xid. In that case we can define + * named savepoint, so that we were able to commit/rollback it separately + * later. + * Special case is if the first change comes from subtransaction, then + * we check that current_xid differs from stream_xid. + */ +void +apply_bgworker_subxact_info_add(TransactionId current_xid) +{ + if (current_xid != stream_xid && + !list_member_int(subxactlist, (int) current_xid)) + { + MemoryContext oldctx; + char spname[MAXPGPATH]; + + snprintf(spname, MAXPGPATH, "savepoint_for_xid_%u", current_xid); + + elog(DEBUG1, "[Apply BGW #%u] defining savepoint %s", + MyParallelShared->n, spname); + + DefineSavepoint(spname); + CommitTransactionCommand(); + + oldctx = MemoryContextSwitchTo(ApplyContext); + subxactlist = lappend_int(subxactlist, (int) current_xid); + MemoryContextSwitchTo(oldctx); + } +} diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c index c5c6a2ba68..d4d5093a0b 100644 --- a/src/backend/replication/logical/decode.c +++ b/src/backend/replication/logical/decode.c @@ -651,9 +651,10 @@ DecodeCommit(LogicalDecodingContext *ctx, XLogRecordBuffer *buf, { for (i = 0; i < parsed->nsubxacts; i++) { - ReorderBufferForget(ctx->reorder, parsed->subxacts[i], buf->origptr); + ReorderBufferForget(ctx->reorder, parsed->subxacts[i], buf->origptr, + commit_time); } - ReorderBufferForget(ctx->reorder, xid, buf->origptr); + ReorderBufferForget(ctx->reorder, xid, buf->origptr, commit_time); return; } @@ -821,10 +822,11 @@ DecodeAbort(LogicalDecodingContext *ctx, XLogRecordBuffer *buf, for (i = 0; i < parsed->nsubxacts; i++) { ReorderBufferAbort(ctx->reorder, parsed->subxacts[i], - buf->record->EndRecPtr); + buf->record->EndRecPtr, abort_time); } - ReorderBufferAbort(ctx->reorder, xid, buf->record->EndRecPtr); + ReorderBufferAbort(ctx->reorder, xid, buf->record->EndRecPtr, + abort_time); } /* update the decoding stats */ diff --git a/src/backend/replication/logical/launcher.c b/src/backend/replication/logical/launcher.c index 3bbd522724..d92bfaf6d6 100644 --- a/src/backend/replication/logical/launcher.c +++ b/src/backend/replication/logical/launcher.c @@ -54,6 +54,7 @@ int max_logical_replication_workers = 4; int max_sync_workers_per_subscription = 2; +int max_apply_bgworkers_per_subscription = 2; LogicalRepWorker *MyLogicalRepWorker = NULL; @@ -73,6 +74,7 @@ static void logicalrep_launcher_onexit(int code, Datum arg); static void logicalrep_worker_onexit(int code, Datum arg); static void logicalrep_worker_detach(void); static void logicalrep_worker_cleanup(LogicalRepWorker *worker); +static void logicalrep_worker_stop_internal(LogicalRepWorker *worker); static bool on_commit_launcher_wakeup = false; @@ -151,8 +153,10 @@ get_subscription_list(void) * * This is only needed for cleaning up the shared memory in case the worker * fails to attach. + * + * Return false if the attach fails. Otherwise return true. */ -static void +static bool WaitForReplicationWorkerAttach(LogicalRepWorker *worker, uint16 generation, BackgroundWorkerHandle *handle) @@ -168,11 +172,11 @@ WaitForReplicationWorkerAttach(LogicalRepWorker *worker, LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); - /* Worker either died or has started; no need to do anything. */ + /* Worker either died or has started. Return false if died. */ if (!worker->in_use || worker->proc) { LWLockRelease(LogicalRepWorkerLock); - return; + return worker->in_use; } LWLockRelease(LogicalRepWorkerLock); @@ -187,7 +191,7 @@ WaitForReplicationWorkerAttach(LogicalRepWorker *worker, if (generation == worker->generation) logicalrep_worker_cleanup(worker); LWLockRelease(LogicalRepWorkerLock); - return; + return false; } /* @@ -223,6 +227,13 @@ logicalrep_worker_find(Oid subid, Oid relid, bool only_running) { LogicalRepWorker *w = &LogicalRepCtx->workers[i]; + /* + * We are only interested in the main apply worker or table sync worker + * here. + */ + if (w->subworker) + continue; + if (w->in_use && w->subid == subid && w->relid == relid && (!only_running || w->proc)) { @@ -259,11 +270,11 @@ logicalrep_workers_find(Oid subid, bool only_running) } /* - * Start new apply background worker, if possible. + * Start new background worker, if possible. */ -void +bool logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, Oid userid, - Oid relid) + Oid relid, dsm_handle subworker_dsm) { BackgroundWorker bgw; BackgroundWorkerHandle *bgw_handle; @@ -273,6 +284,10 @@ logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, Oid userid, LogicalRepWorker *worker = NULL; int nsyncworkers; TimestampTz now; + bool is_subworker = (subworker_dsm != DSM_HANDLE_INVALID); + + /* Sanity check : we don't support table sync in subworker. */ + Assert(!(is_subworker && OidIsValid(relid))); ereport(DEBUG1, (errmsg_internal("starting logical replication worker for subscription \"%s\"", @@ -350,7 +365,7 @@ retry: if (OidIsValid(relid) && nsyncworkers >= max_sync_workers_per_subscription) { LWLockRelease(LogicalRepWorkerLock); - return; + return false; } /* @@ -364,7 +379,7 @@ retry: (errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED), errmsg("out of logical replication worker slots"), errhint("You might need to increase max_logical_replication_workers."))); - return; + return false; } /* Prepare the worker slot. */ @@ -379,6 +394,7 @@ retry: worker->relstate = SUBREL_STATE_UNKNOWN; worker->relstate_lsn = InvalidXLogRecPtr; worker->stream_fileset = NULL; + worker->subworker = is_subworker; worker->last_lsn = InvalidXLogRecPtr; TIMESTAMP_NOBEGIN(worker->last_send_time); TIMESTAMP_NOBEGIN(worker->last_recv_time); @@ -396,19 +412,31 @@ retry: BGWORKER_BACKEND_DATABASE_CONNECTION; bgw.bgw_start_time = BgWorkerStart_RecoveryFinished; snprintf(bgw.bgw_library_name, BGW_MAXLEN, "postgres"); - snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyWorkerMain"); + + if (is_subworker) + snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyBgworkerMain"); + else + snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyWorkerMain"); + if (OidIsValid(relid)) snprintf(bgw.bgw_name, BGW_MAXLEN, "logical replication worker for subscription %u sync %u", subid, relid); + else if (is_subworker) + snprintf(bgw.bgw_name, BGW_MAXLEN, + "logical replication apply background worker for subscription %u", subid); else snprintf(bgw.bgw_name, BGW_MAXLEN, "logical replication worker for subscription %u", subid); + snprintf(bgw.bgw_type, BGW_MAXLEN, "logical replication worker"); bgw.bgw_restart_time = BGW_NEVER_RESTART; bgw.bgw_notify_pid = MyProcPid; bgw.bgw_main_arg = Int32GetDatum(slot); + if (is_subworker) + memcpy(bgw.bgw_extra, &subworker_dsm, sizeof(dsm_handle)); + if (!RegisterDynamicBackgroundWorker(&bgw, &bgw_handle)) { /* Failed to start worker, so clean up the worker slot. */ @@ -421,11 +449,11 @@ retry: (errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED), errmsg("out of background worker slots"), errhint("You might need to increase max_worker_processes."))); - return; + return false; } /* Now wait until it attaches. */ - WaitForReplicationWorkerAttach(worker, generation, bgw_handle); + return WaitForReplicationWorkerAttach(worker, generation, bgw_handle); } /* @@ -436,18 +464,27 @@ void logicalrep_worker_stop(Oid subid, Oid relid) { LogicalRepWorker *worker; - uint16 generation; LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); worker = logicalrep_worker_find(subid, relid, false); - /* No worker, nothing to do. */ - if (!worker) - { - LWLockRelease(LogicalRepWorkerLock); - return; - } + if (worker) + logicalrep_worker_stop_internal(worker); + + LWLockRelease(LogicalRepWorkerLock); +} + +/* + * Workhorse for logicalrep_worker_stop() and logicalrep_worker_detach(). Stop + * the worker and wait for it to die. + */ +static void +logicalrep_worker_stop_internal(LogicalRepWorker *worker) +{ + uint16 generation; + + Assert(LWLockHeldByMe(LogicalRepWorkerLock)); /* * Remember which generation was our worker so we can check if what we see @@ -485,10 +522,7 @@ logicalrep_worker_stop(Oid subid, Oid relid) * different, meaning that a different worker has taken the slot. */ if (!worker->in_use || worker->generation != generation) - { - LWLockRelease(LogicalRepWorkerLock); return; - } /* Worker has assigned proc, so it has started. */ if (worker->proc) @@ -522,8 +556,6 @@ logicalrep_worker_stop(Oid subid, Oid relid) LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); } - - LWLockRelease(LogicalRepWorkerLock); } /* @@ -599,6 +631,29 @@ logicalrep_worker_attach(int slot) static void logicalrep_worker_detach(void) { + /* + * If we are the main apply worker, stop all the apply background workers + * we started before. + */ + if (!MyLogicalRepWorker->subworker) + { + List *workers; + ListCell *lc; + + LWLockAcquire(LogicalRepWorkerLock, LW_SHARED); + + workers = logicalrep_workers_find(MyLogicalRepWorker->subid, true); + foreach(lc, workers) + { + LogicalRepWorker *w = (LogicalRepWorker *) lfirst(lc); + + if (w->subworker) + logicalrep_worker_stop_internal(w); + } + + LWLockRelease(LogicalRepWorkerLock); + } + /* Block concurrent access. */ LWLockAcquire(LogicalRepWorkerLock, LW_EXCLUSIVE); @@ -621,6 +676,7 @@ logicalrep_worker_cleanup(LogicalRepWorker *worker) worker->userid = InvalidOid; worker->subid = InvalidOid; worker->relid = InvalidOid; + worker->subworker = false; } /* @@ -679,6 +735,30 @@ logicalrep_sync_worker_count(Oid subid) return res; } +/* + * Count the number of registered (not necessarily running) apply background + * workers for a subscription. + */ +int +logicalrep_apply_bgworker_count(Oid subid) +{ + int i; + int res = 0; + + Assert(LWLockHeldByMe(LogicalRepWorkerLock)); + + /* Search for attached worker for a given subscription id. */ + for (i = 0; i < max_logical_replication_workers; i++) + { + LogicalRepWorker *w = &LogicalRepCtx->workers[i]; + + if (w->subid == subid && w->subworker) + res++; + } + + return res; +} + /* * ApplyLauncherShmemSize * Compute space needed for replication launcher shared memory @@ -868,7 +948,7 @@ ApplyLauncherMain(Datum main_arg) wait_time = wal_retrieve_retry_interval; logicalrep_worker_launch(sub->dbid, sub->oid, sub->name, - sub->owner, InvalidOid); + sub->owner, InvalidOid, DSM_HANDLE_INVALID); } } diff --git a/src/backend/replication/logical/origin.c b/src/backend/replication/logical/origin.c index c72ad6b93d..2458e2fce9 100644 --- a/src/backend/replication/logical/origin.c +++ b/src/backend/replication/logical/origin.c @@ -1075,12 +1075,21 @@ ReplicationOriginExitCleanup(int code, Datum arg) * array doesn't have to be searched when calling * replorigin_session_advance(). * - * Obviously only one such cached origin can exist per process and the current + * Normally only one such cached origin can exist per process and the current * cached value can only be set again after the previous value is torn down * with replorigin_session_reset(). + * + * However, if the function parameter 'must_acquire' is false, we allow the + * process to use the same slot already acquired by another process. It's safe + * because 1) The only caller (apply background workers) will maintain the + * commit order by allowing only one process to commit at a time, so no two + * workers will be operating on the same origin at the same time (see comments + * in logical/worker.c). 2) Even though we try to advance the session's origin + * concurrently, it's safe to do so as we change/advance the session_origin + * LSNs under replicate_state LWLock. */ void -replorigin_session_setup(RepOriginId node) +replorigin_session_setup(RepOriginId node, bool must_acquire) { static bool registered_cleanup; int i; @@ -1122,7 +1131,7 @@ replorigin_session_setup(RepOriginId node) if (curstate->roident != node) continue; - else if (curstate->acquired_by != 0) + else if (curstate->acquired_by != 0 && must_acquire) { ereport(ERROR, (errcode(ERRCODE_OBJECT_IN_USE), @@ -1153,7 +1162,14 @@ replorigin_session_setup(RepOriginId node) Assert(session_replication_state->roident != InvalidRepOriginId); - session_replication_state->acquired_by = MyProcPid; + if (must_acquire) + session_replication_state->acquired_by = MyProcPid; + else if (session_replication_state->acquired_by == 0) + ereport(ERROR, + (errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED), + errmsg("apply background worker could not find replication state slot for replication origin with OID %u", + node), + errdetail("There is no replication state slot set by its main apply worker."))); LWLockRelease(ReplicationOriginLock); @@ -1337,7 +1353,7 @@ pg_replication_origin_session_setup(PG_FUNCTION_ARGS) name = text_to_cstring((text *) DatumGetPointer(PG_GETARG_DATUM(0))); origin = replorigin_by_name(name, false); - replorigin_session_setup(origin); + replorigin_session_setup(origin, true); replorigin_session_origin = origin; diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c index ff8513e2d2..47bd811fb7 100644 --- a/src/backend/replication/logical/proto.c +++ b/src/backend/replication/logical/proto.c @@ -1163,31 +1163,56 @@ logicalrep_read_stream_commit(StringInfo in, LogicalRepCommitData *commit_data) /* * Write STREAM ABORT to the output stream. Note that xid and subxid will be * same for the top-level transaction abort. + * + * If write_abort_lsn is true, send the abort_lsn and abort_time fields, + * otherwise don't. */ void logicalrep_write_stream_abort(StringInfo out, TransactionId xid, - TransactionId subxid) + ReorderBufferTXN *txn, XLogRecPtr abort_lsn, + bool write_abort_lsn) { pq_sendbyte(out, LOGICAL_REP_MSG_STREAM_ABORT); - Assert(TransactionIdIsValid(xid) && TransactionIdIsValid(subxid)); + Assert(TransactionIdIsValid(xid) && TransactionIdIsValid(txn->xid)); /* transaction ID */ pq_sendint32(out, xid); - pq_sendint32(out, subxid); + pq_sendint32(out, txn->xid); + + if (write_abort_lsn) + { + pq_sendint64(out, abort_lsn); + pq_sendint64(out, txn->xact_time.abort_time); + } } /* * Read STREAM ABORT from the output stream. + * + * If read_abort_lsn is true, try to read the abort_lsn and abort_time fields, + * otherwise don't. */ void -logicalrep_read_stream_abort(StringInfo in, TransactionId *xid, - TransactionId *subxid) +logicalrep_read_stream_abort(StringInfo in, + LogicalRepStreamAbortData *abort_data, + bool read_abort_lsn) { - Assert(xid && subxid); + Assert(abort_data); - *xid = pq_getmsgint(in, 4); - *subxid = pq_getmsgint(in, 4); + abort_data->xid = pq_getmsgint(in, 4); + abort_data->subxid = pq_getmsgint(in, 4); + + if (read_abort_lsn) + { + abort_data->abort_lsn = pq_getmsgint64(in); + abort_data->abort_time = pq_getmsgint64(in); + } + else + { + abort_data->abort_lsn = InvalidXLogRecPtr; + abort_data->abort_time = 0; + } } /* diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c index 88a37fde72..8989328046 100644 --- a/src/backend/replication/logical/reorderbuffer.c +++ b/src/backend/replication/logical/reorderbuffer.c @@ -2826,7 +2826,8 @@ ReorderBufferFinishPrepared(ReorderBuffer *rb, TransactionId xid, * disk. */ void -ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn) +ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn, + TimestampTz abort_time) { ReorderBufferTXN *txn; @@ -2837,6 +2838,8 @@ ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn) if (txn == NULL) return; + txn->xact_time.abort_time = abort_time; + /* For streamed transactions notify the remote node about the abort. */ if (rbtxn_is_streamed(txn)) { @@ -2911,7 +2914,8 @@ ReorderBufferAbortOld(ReorderBuffer *rb, TransactionId oldestRunningXid) * to this xid might re-create the transaction incompletely. */ void -ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn) +ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn, + TimestampTz abort_time) { ReorderBufferTXN *txn; @@ -2922,6 +2926,8 @@ ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn) if (txn == NULL) return; + txn->xact_time.abort_time = abort_time; + /* For streamed transactions notify the remote node about the abort. */ if (rbtxn_is_streamed(txn)) rb->stream_abort(rb, txn, lsn); diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c index 670c6fcada..8ffba7e2e5 100644 --- a/src/backend/replication/logical/tablesync.c +++ b/src/backend/replication/logical/tablesync.c @@ -568,7 +568,8 @@ process_syncing_tables_for_apply(XLogRecPtr current_lsn) MySubscription->oid, MySubscription->name, MyLogicalRepWorker->userid, - rstate->relid); + rstate->relid, + DSM_HANDLE_INVALID); hentry->last_start_time = now; } } @@ -589,6 +590,9 @@ process_syncing_tables_for_apply(XLogRecPtr current_lsn) void process_syncing_tables(XLogRecPtr current_lsn) { + if (MyLogicalRepWorker->subworker) + return; + if (am_tablesync_worker()) process_syncing_tables_for_sync(current_lsn); else @@ -1273,7 +1277,7 @@ LogicalRepSyncTableStart(XLogRecPtr *origin_startpos) * time this tablesync was launched. */ originid = replorigin_by_name(originname, false); - replorigin_session_setup(originid); + replorigin_session_setup(originid, true); replorigin_session_origin = originid; *origin_startpos = replorigin_session_get_progress(false); @@ -1384,7 +1388,7 @@ LogicalRepSyncTableStart(XLogRecPtr *origin_startpos) true /* go backward */ , true /* WAL log */ ); UnlockRelationOid(ReplicationOriginRelationId, RowExclusiveLock); - replorigin_session_setup(originid); + replorigin_session_setup(originid, true); replorigin_session_origin = originid; } else diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c index 5f8c541763..2aa7797628 100644 --- a/src/backend/replication/logical/worker.c +++ b/src/backend/replication/logical/worker.c @@ -22,8 +22,28 @@ * STREAMED TRANSACTIONS * --------------------- * Streamed transactions (large transactions exceeding a memory limit on the - * upstream) are not applied immediately, but instead, the data is written - * to temporary files and then applied at once when the final commit arrives. + * upstream) are applied using one of two approaches. + * + * 1) Separate background workers + * + * If streaming = parallel, we assign a new apply background worker (if + * available) as soon as the xact's first stream is received. The main apply + * worker will send changes to this new worker via shared memory. We keep this + * worker assigned till the transaction commit is received and also wait for + * the worker to finish at commit. This preserves commit ordering and avoids + * file I/O in most cases. We still need to spill to a file if there is no + * worker available. It is important to maintain commit order to avoid failures + * due to (a) transaction dependencies, say if we insert a row in the first + * transaction and update it in the second transaction then allowing to apply + * both in parallel can lead to failure in the update. (b) deadlocks, allowing + * transactions that update the same set of rows/tables in opposite order to be + * applied in parallel can lead to deadlocks. + * + * 2) Write to temporary files and apply when the final commit arrives + * + * If no worker is available to handle streamed transaction, the data is + * written to temporary files and then applied at once when the final commit + * arrives. * * Unlike the regular (non-streamed) case, handling streamed transactions has * to handle aborts of both the toplevel transaction and subtransactions. This @@ -219,20 +239,8 @@ typedef struct ApplyExecutionData PartitionTupleRouting *proute; /* partition routing info */ } ApplyExecutionData; -/* Struct for saving and restoring apply errcontext information */ -typedef struct ApplyErrorCallbackArg -{ - LogicalRepMsgType command; /* 0 if invalid */ - LogicalRepRelMapEntry *rel; - - /* Remote node information */ - int remote_attnum; /* -1 if invalid */ - TransactionId remote_xid; - XLogRecPtr finish_lsn; - char *origin_name; -} ApplyErrorCallbackArg; - -static ApplyErrorCallbackArg apply_error_callback_arg = +/* errcontext tracker */ +ApplyErrorCallbackArg apply_error_callback_arg = { .command = 0, .rel = NULL, @@ -242,7 +250,7 @@ static ApplyErrorCallbackArg apply_error_callback_arg = .origin_name = NULL, }; -static MemoryContext ApplyMessageContext = NULL; +MemoryContext ApplyMessageContext = NULL; MemoryContext ApplyContext = NULL; /* per stream context for streaming transactions */ @@ -251,27 +259,39 @@ static MemoryContext LogicalStreamingContext = NULL; WalReceiverConn *LogRepWorkerWalRcvConn = NULL; Subscription *MySubscription = NULL; -static bool MySubscriptionValid = false; +bool MySubscriptionValid = false; bool in_remote_transaction = false; static XLogRecPtr remote_final_lsn = InvalidXLogRecPtr; /* fields valid only when processing streamed transaction */ -static bool in_streamed_transaction = false; +bool in_streamed_transaction = false; + +TransactionId stream_xid = InvalidTransactionId; +static ApplyBgworkerState *stream_apply_worker = NULL; -static TransactionId stream_xid = InvalidTransactionId; +/* Check if we are applying the transaction in an apply background worker */ +#define apply_bgworker_active() (in_streamed_transaction && stream_apply_worker != NULL) + +/* + * The number of changes during one streaming block (only for apply background + * workers) + */ +static uint32 nchanges = 0; /* * We enable skipping all data modification changes (INSERT, UPDATE, etc.) for * the subscription if the remote transaction's finish LSN matches the subskiplsn. * Once we start skipping changes, we don't stop it until we skip all changes of * the transaction even if pg_subscription is updated and MySubscription->skiplsn - * gets changed or reset during that. Also, in streaming transaction cases, we - * don't skip receiving and spooling the changes since we decide whether or not + * gets changed or reset during that. Also, in streaming transaction cases (streaming = on), + * we don't skip receiving and spooling the changes since we decide whether or not * to skip applying the changes when starting to apply changes. The subskiplsn is * cleared after successfully skipping the transaction or applying non-empty * transaction. The latter prevents the mistakenly specified subskiplsn from - * being left. + * being left. Note that we cannot skip the streaming transactions when using + * apply background workers because we cannot get the finish LSN before + * applying the changes. */ static XLogRecPtr skip_xact_finish_lsn = InvalidXLogRecPtr; #define is_skipping_changes() (unlikely(!XLogRecPtrIsInvalid(skip_xact_finish_lsn))) @@ -324,9 +344,6 @@ static void maybe_reread_subscription(void); static void DisableSubscriptionAndExit(void); -/* prototype needed because of stream_commit */ -static void apply_dispatch(StringInfo s); - static void apply_handle_commit_internal(LogicalRepCommitData *commit_data); static void apply_handle_insert_internal(ApplyExecutionData *edata, ResultRelInfo *relinfo, @@ -359,7 +376,6 @@ static void stop_skipping_changes(void); static void clear_subscription_skip_lsn(XLogRecPtr finish_lsn); /* Functions for apply error callback */ -static void apply_error_callback(void *arg); static inline void set_apply_error_context_xact(TransactionId xid, XLogRecPtr lsn); static inline void reset_apply_error_context_info(void); @@ -426,40 +442,85 @@ end_replication_step(void) } /* - * Handle streamed transactions. + * Handle streamed transactions for both the main apply worker and the apply + * background workers. + * + * In streaming case (receiving a block of streamed transaction), for + * SUBSTREAM_ON mode, simply redirect it to a file for the proper toplevel + * transaction, and for SUBSTREAM_PARALLEL mode, send the changes to apply + * background workers (LOGICAL_REP_MSG_RELATION or LOGICAL_REP_MSG_TYPE changes + * will also be applied in main apply worker). * - * If in streaming mode (receiving a block of streamed transaction), we - * simply redirect it to a file for the proper toplevel transaction. + * For non-streamed transactions, returns false; + * For streamed transactions, returns true if in main apply worker, false + * otherwise. * - * Returns true for streamed transactions, false otherwise (regular mode). + * Exception: When the main apply worker is applying streaming transactions in + * parallel mode (e.g. when addressing LOGICAL_REP_MSG_RELATION or + * LOGICAL_REP_MSG_TYPE changes), then return false. */ static bool handle_streamed_transaction(LogicalRepMsgType action, StringInfo s) { - TransactionId xid; + TransactionId current_xid = InvalidTransactionId; - /* not in streaming mode */ - if (!in_streamed_transaction) + /* Not in streaming mode */ + if (!(in_streamed_transaction || am_apply_bgworker())) return false; - Assert(stream_fd != NULL); Assert(TransactionIdIsValid(stream_xid)); /* * We should have received XID of the subxact as the first part of the * message, so extract it. */ - xid = pq_getmsgint(s, 4); + current_xid = pq_getmsgint(s, 4); - if (!TransactionIdIsValid(xid)) + if (!TransactionIdIsValid(current_xid)) ereport(ERROR, (errcode(ERRCODE_PROTOCOL_VIOLATION), errmsg_internal("invalid transaction ID in streamed replication transaction"))); - /* Add the new subxact to the array (unless already there). */ - subxact_info_add(xid); + if (am_apply_bgworker()) + { + /* Define a savepoint for a subxact if needed. */ + apply_bgworker_subxact_info_add(current_xid); + + return false; + } + + if (apply_bgworker_active()) + { + /* + * This is the main apply worker, but there is an apply background + * worker, so apply the changes of this transaction in that background + * worker. Pass the data to the worker. + */ + apply_bgworker_send_data(stream_apply_worker, s->len, s->data); + nchanges += 1; + + /* + * XXX The publisher side doesn't always send relation/type update + * messages after the streaming transaction, so also update the + * relation/type in main apply worker here. See function + * cleanup_rel_sync_cache. + */ + if (action == LOGICAL_REP_MSG_RELATION || + action == LOGICAL_REP_MSG_TYPE) + return false; + + return true; + } + + /* + * This is the main apply worker, but there is no apply background worker, + * so write to temporary files and apply when the final commit arrives. + * + * Add the new subxact to the array (unless already there). + */ + subxact_info_add(current_xid); - /* write the change to the current file */ + /* Write the change to the current file */ stream_write_change(action, s); return true; @@ -844,6 +905,9 @@ apply_handle_commit(StringInfo s) apply_handle_commit_internal(&commit_data); + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); + /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(commit_data.end_lsn); @@ -898,7 +962,9 @@ apply_handle_prepare_internal(LogicalRepPreparedTxnData *prepare_data) * BeginTransactionBlock is necessary to balance the EndTransactionBlock * called within the PrepareTransactionBlock below. */ - BeginTransactionBlock(); + if (!IsTransactionBlock()) + BeginTransactionBlock(); + CommitTransactionCommand(); /* Completes the preceding Begin command. */ /* @@ -950,6 +1016,9 @@ apply_handle_prepare(StringInfo s) in_remote_transaction = false; + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); + /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(prepare_data.end_lsn); @@ -1064,10 +1133,6 @@ apply_handle_rollback_prepared(StringInfo s) /* * Handle STREAM PREPARE. - * - * Logic is in two parts: - * 1. Replay all the spooled operations - * 2. Mark the transaction as prepared */ static void apply_handle_stream_prepare(StringInfo s) @@ -1088,24 +1153,78 @@ apply_handle_stream_prepare(StringInfo s) logicalrep_read_stream_prepare(s, &prepare_data); set_apply_error_context_xact(prepare_data.xid, prepare_data.prepare_lsn); - elog(DEBUG1, "received prepare for streamed transaction %u", prepare_data.xid); + if (am_apply_bgworker()) + { + /* Mark the transaction as prepared. */ + apply_handle_prepare_internal(&prepare_data); - /* Replay all the spooled operations. */ - apply_spooled_messages(prepare_data.xid, prepare_data.prepare_lsn); + CommitTransactionCommand(); - /* Mark the transaction as prepared. */ - apply_handle_prepare_internal(&prepare_data); + pgstat_report_stat(false); - CommitTransactionCommand(); + list_free(subxactlist); + subxactlist = NIL; - pgstat_report_stat(false); + apply_bgworker_set_status(APPLY_BGWORKER_FINISHED); + } + else + { + /* This is the main apply worker. */ + ApplyBgworkerState *wstate = apply_bgworker_find(prepare_data.xid); - store_flush_position(prepare_data.end_lsn); + elog(DEBUG1, "received prepare for streamed transaction %u", + prepare_data.xid); + + /* + * Check if we are processing this transaction in an apply background + * worker and if so, send the changes to that worker. + */ + if (wstate) + { + /* Send STREAM PREPARE message to the apply background worker. */ + apply_bgworker_send_data(wstate, s->len, s->data); + + /* + * After sending the data to the apply background worker, wait for + * that worker to finish. This is necessary to maintain commit + * order which avoids failures due to transaction dependencies and + * deadlocks. + */ + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_FINISHED); + apply_bgworker_free(wstate); + + pgstat_report_stat(false); + store_flush_position(prepare_data.end_lsn); + } + else + { + /* + * The transaction has been serialized to file, so replay all the + * spooled operations. + */ + apply_spooled_messages(prepare_data.xid, prepare_data.prepare_lsn); + + /* Mark the transaction as prepared. */ + apply_handle_prepare_internal(&prepare_data); + + CommitTransactionCommand(); + + pgstat_report_stat(false); + + store_flush_position(prepare_data.end_lsn); + + in_remote_transaction = false; + + /* Unlink the files with serialized changes and subxact info. */ + stream_cleanup_files(MyLogicalRepWorker->subid, prepare_data.xid); + } + } in_remote_transaction = false; + stream_apply_worker = NULL; - /* unlink the files with serialized changes and subxact info. */ - stream_cleanup_files(MyLogicalRepWorker->subid, prepare_data.xid); + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(prepare_data.end_lsn); @@ -1155,15 +1274,6 @@ apply_handle_stream_start(StringInfo s) (errcode(ERRCODE_PROTOCOL_VIOLATION), errmsg_internal("duplicate STREAM START message"))); - /* - * Start a transaction on stream start, this transaction will be committed - * on the stream stop unless it is a tablesync worker in which case it - * will be committed after processing all the messages. We need the - * transaction for handling the buffile, used for serializing the - * streaming data and subxact info. - */ - begin_replication_step(); - /* notify handle methods we're processing a remote transaction */ in_streamed_transaction = true; @@ -1177,36 +1287,93 @@ apply_handle_stream_start(StringInfo s) set_apply_error_context_xact(stream_xid, InvalidXLogRecPtr); - /* - * Initialize the worker's stream_fileset if we haven't yet. This will be - * used for the entire duration of the worker so create it in a permanent - * context. We create this on the very first streaming message from any - * transaction and then use it for this and other streaming transactions. - * Now, we could create a fileset at the start of the worker as well but - * then we won't be sure that it will ever be used. - */ - if (MyLogicalRepWorker->stream_fileset == NULL) + if (am_apply_bgworker()) { - MemoryContext oldctx; - - oldctx = MemoryContextSwitchTo(ApplyContext); + /* + * Make sure the handle apply_dispatch methods are aware we're in a + * remote transaction. + */ + in_remote_transaction = true; - MyLogicalRepWorker->stream_fileset = palloc(sizeof(FileSet)); - FileSetInit(MyLogicalRepWorker->stream_fileset); + /* Begin the transaction. */ + AcceptInvalidationMessages(); + maybe_reread_subscription(); - MemoryContextSwitchTo(oldctx); + StartTransactionCommand(); + BeginTransactionBlock(); + CommitTransactionCommand(); } + else + { + /* + * This is the main apply worker. Check if there is any free apply + * background worker we can use to process this transaction. + */ + if (first_segment) + stream_apply_worker = apply_bgworker_start(stream_xid); + else + stream_apply_worker = apply_bgworker_find(stream_xid); - /* open the spool file for this transaction */ - stream_open_file(MyLogicalRepWorker->subid, stream_xid, first_segment); + if (stream_apply_worker) + { + /* + * If we have found a free worker or if we are already applying this + * transaction in an apply background worker, then we pass the data to + * that worker. + */ + if (first_segment) + apply_bgworker_send_data(stream_apply_worker, s->len, s->data); - /* if this is not the first segment, open existing subxact file */ - if (!first_segment) - subxact_info_read(MyLogicalRepWorker->subid, stream_xid); + nchanges = 0; + elog(DEBUG1, "starting streaming of xid %u", stream_xid); + } + else + { + /* + * Since no apply background worker is available for the first + * stream start, serialize all the changes of the transaction. + * + * Start a transaction on stream start, this transaction will be + * committed on the stream stop unless it is a tablesync worker in + * which case it will be committed after processing all the + * messages. We need the transaction for handling the buffile, + * used for serializing the streaming data and subxact info. + */ + begin_replication_step(); - pgstat_report_activity(STATE_RUNNING, NULL); + /* + * Initialize the worker's stream_fileset if we haven't yet. This will + * be used for the entire duration of the worker so create it in a + * permanent context. We create this on the very first streaming + * message from any transaction and then use it for this and other + * streaming transactions. Now, we could create a fileset at the start + * of the worker as well but then we won't be sure that it will ever + * be used. + */ + if (MyLogicalRepWorker->stream_fileset == NULL) + { + MemoryContext oldctx; - end_replication_step(); + oldctx = MemoryContextSwitchTo(ApplyContext); + + MyLogicalRepWorker->stream_fileset = palloc(sizeof(FileSet)); + FileSetInit(MyLogicalRepWorker->stream_fileset); + + MemoryContextSwitchTo(oldctx); + } + + /* Open the spool file for this transaction. */ + stream_open_file(MyLogicalRepWorker->subid, stream_xid, first_segment); + + /* If this is not the first segment, open existing subxact file. */ + if (!first_segment) + subxact_info_read(MyLogicalRepWorker->subid, stream_xid); + + end_replication_step(); + } + } + + pgstat_report_activity(STATE_RUNNING, NULL); } /* @@ -1220,53 +1387,52 @@ apply_handle_stream_stop(StringInfo s) (errcode(ERRCODE_PROTOCOL_VIOLATION), errmsg_internal("STREAM STOP message without STREAM START"))); - /* - * Close the file with serialized changes, and serialize information about - * subxacts for the toplevel transaction. - */ - subxact_info_write(MyLogicalRepWorker->subid, stream_xid); - stream_close_file(); + if (apply_bgworker_active()) + { + char action = LOGICAL_REP_MSG_STREAM_STOP; - /* We must be in a valid transaction state */ - Assert(IsTransactionState()); + apply_bgworker_send_data(stream_apply_worker, 1, &action); - /* Commit the per-stream transaction */ - CommitTransactionCommand(); + elog(DEBUG1, "stopped streaming of xid %u, %u changes streamed", stream_xid, nchanges); + } + else + { + /* + * Close the file with serialized changes, and serialize information + * about subxacts for the toplevel transaction. + */ + subxact_info_write(MyLogicalRepWorker->subid, stream_xid); + stream_close_file(); - in_streamed_transaction = false; + /* We must be in a valid transaction state */ + Assert(IsTransactionState()); - /* Reset per-stream context */ - MemoryContextReset(LogicalStreamingContext); + /* Commit the per-stream transaction */ + CommitTransactionCommand(); + + /* Reset per-stream context */ + MemoryContextReset(LogicalStreamingContext); + } + + in_streamed_transaction = false; + stream_apply_worker = NULL; pgstat_report_activity(STATE_IDLE, NULL); reset_apply_error_context_info(); } /* - * Handle STREAM abort message. + * Handle STREAM ABORT message when the transaction was spilled to disk. */ static void -apply_handle_stream_abort(StringInfo s) +serialize_stream_abort(TransactionId xid, TransactionId subxid) { - TransactionId xid; - TransactionId subxid; - - if (in_streamed_transaction) - ereport(ERROR, - (errcode(ERRCODE_PROTOCOL_VIOLATION), - errmsg_internal("STREAM ABORT message without STREAM STOP"))); - - logicalrep_read_stream_abort(s, &xid, &subxid); - /* * If the two XIDs are the same, it's in fact abort of toplevel xact, so * just delete the files with serialized info. */ if (xid == subxid) - { - set_apply_error_context_xact(xid, InvalidXLogRecPtr); stream_cleanup_files(MyLogicalRepWorker->subid, xid); - } else { /* @@ -1290,8 +1456,6 @@ apply_handle_stream_abort(StringInfo s) bool found = false; char path[MAXPGPATH]; - set_apply_error_context_xact(subxid, InvalidXLogRecPtr); - subidx = -1; begin_replication_step(); subxact_info_read(MyLogicalRepWorker->subid, xid); @@ -1316,7 +1480,6 @@ apply_handle_stream_abort(StringInfo s) cleanup_subxact_info(); end_replication_step(); CommitTransactionCommand(); - reset_apply_error_context_info(); return; } @@ -1339,6 +1502,143 @@ apply_handle_stream_abort(StringInfo s) end_replication_step(); CommitTransactionCommand(); } +} + +/* + * Handle STREAM ABORT message. + */ +static void +apply_handle_stream_abort(StringInfo s) +{ + TransactionId xid; + TransactionId subxid; + LogicalRepStreamAbortData abort_data; + bool read_abort_lsn = false; + + if (in_streamed_transaction) + ereport(ERROR, + (errcode(ERRCODE_PROTOCOL_VIOLATION), + errmsg_internal("STREAM ABORT message without STREAM STOP"))); + + /* Check whether the publisher sends abort_lsn and abort_time. */ + if (am_apply_bgworker()) + read_abort_lsn = MyParallelShared->server_version >= + LOGICALREP_PROTO_STREAM_PARALLEL_VERSION_NUM; + + logicalrep_read_stream_abort(s, &abort_data, read_abort_lsn); + + xid = abort_data.xid; + subxid = abort_data.subxid; + + set_apply_error_context_xact(subxid, abort_data.abort_lsn); + + if (am_apply_bgworker()) + { + elog(DEBUG1, "[Apply BGW #%u] aborting current transaction xid=%u, subxid=%u", + MyParallelShared->n, GetCurrentTransactionIdIfAny(), + GetCurrentSubTransactionId()); + + /* + * Update origin state so we can restart streaming from correct + * position in case of crash. + */ + if (read_abort_lsn) + { + replorigin_session_origin_lsn = abort_data.abort_lsn; + replorigin_session_origin_timestamp = abort_data.abort_time; + } + + /* + * If the two XIDs are the same, it's in fact abort of toplevel xact, + * so just free the subxactlist. + */ + if (subxid == xid) + { + AbortCurrentTransaction(); + + EndTransactionBlock(false); + CommitTransactionCommand(); + + in_remote_transaction = false; + pgstat_report_activity(STATE_IDLE, NULL); + + list_free(subxactlist); + subxactlist = NIL; + + apply_bgworker_set_status(APPLY_BGWORKER_FINISHED); + } + else + { + /* + * OK, so it's a subxact. Rollback to the savepoint. + * + * We also need to read the subxactlist, determine the offset + * tracked for the subxact, and truncate the list. + */ + int i; + bool found = false; + char spname[MAXPGPATH]; + + snprintf(spname, MAXPGPATH, "savepoint_for_xid_%u", subxid); + + elog(DEBUG1, "[Apply BGW #%u] rolling back to savepoint %s", + MyParallelShared->n, spname); + + for (i = list_length(subxactlist) - 1; i >= 0; i--) + { + xid = (TransactionId) list_nth_int(subxactlist, i); + if (xid == subxid) + { + found = true; + break; + } + } + + if (found) + { + RollbackToSavepoint(spname); + CommitTransactionCommand(); + subxactlist = list_truncate(subxactlist, i + 1); + } + + pgstat_report_activity(STATE_IDLEINTRANSACTION, NULL); + } + } + else + { + /* This is the main apply worker. */ + ApplyBgworkerState *wstate = apply_bgworker_find(xid); + + /* + * Check if we are processing this transaction in an apply background + * worker and if so, send the changes to that worker. + */ + if (wstate) + { + /* Send STREAM ABORT message to the apply background worker. */ + apply_bgworker_send_data(wstate, s->len, s->data); + + /* + * After sending the data to the apply background worker, wait for + * that worker to finish. This is necessary to maintain commit + * order which avoids failures due to transaction dependencies and + * deadlocks. + */ + if (subxid == xid) + { + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_FINISHED); + apply_bgworker_free(wstate); + } + } + else + { + /* + * We are in main apply worker and the transaction has been + * serialized to file. + */ + serialize_stream_abort(xid, subxid); + } + } reset_apply_error_context_info(); } @@ -1468,8 +1768,8 @@ apply_spooled_messages(TransactionId xid, XLogRecPtr lsn) static void apply_handle_stream_commit(StringInfo s) { - TransactionId xid; LogicalRepCommitData commit_data; + TransactionId xid; if (in_streamed_transaction) ereport(ERROR, @@ -1479,14 +1779,81 @@ apply_handle_stream_commit(StringInfo s) xid = logicalrep_read_stream_commit(s, &commit_data); set_apply_error_context_xact(xid, commit_data.commit_lsn); - elog(DEBUG1, "received commit for streamed transaction %u", xid); + if (am_apply_bgworker()) + { + /* + * Update origin state so we can restart streaming from correct + * position in case of crash. + */ + replorigin_session_origin_lsn = commit_data.end_lsn; + replorigin_session_origin_timestamp = commit_data.committime; - apply_spooled_messages(xid, commit_data.commit_lsn); + CommitTransactionCommand(); + EndTransactionBlock(false); + CommitTransactionCommand(); - apply_handle_commit_internal(&commit_data); + in_remote_transaction = false; + + pgstat_report_stat(false); + + list_free(subxactlist); + subxactlist = NIL; + + apply_bgworker_set_status(APPLY_BGWORKER_FINISHED); + } + else + { + /* This is the main apply worker. */ + ApplyBgworkerState *wstate = apply_bgworker_find(xid); + + elog(DEBUG1, "received commit for streamed transaction %u", xid); + + /* + * Check if we are processing this transaction in an apply background + * worker and if so, send the changes to that worker. + */ + if (wstate) + { + /* Send STREAM COMMIT message to the apply background worker. */ + apply_bgworker_send_data(wstate, s->len, s->data); + + /* + * After sending the data to the apply background worker, wait for + * that worker to finish. This is necessary to maintain commit + * order which avoids failures due to transaction dependencies and + * deadlocks. + */ + apply_bgworker_wait_for(wstate, APPLY_BGWORKER_FINISHED); - /* unlink the files with serialized changes and subxact info */ - stream_cleanup_files(MyLogicalRepWorker->subid, xid); + pgstat_report_stat(false); + store_flush_position(commit_data.end_lsn); + stop_skipping_changes(); + + apply_bgworker_free(wstate); + + /* + * The transaction is either non-empty or skipped, so we clear the + * subskiplsn. + */ + clear_subscription_skip_lsn(commit_data.commit_lsn); + } + else + { + /* + * The transaction has been serialized to file, so replay all the + * spooled operations. + */ + apply_spooled_messages(xid, commit_data.commit_lsn); + + apply_handle_commit_internal(&commit_data); + + /* Unlink the files with serialized changes and subxact info. */ + stream_cleanup_files(MyLogicalRepWorker->subid, xid); + } + } + + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(commit_data.end_lsn); @@ -2467,7 +2834,7 @@ apply_handle_truncate(StringInfo s) /* * Logical replication protocol message dispatcher. */ -static void +void apply_dispatch(StringInfo s) { LogicalRepMsgType action = pq_getmsgbyte(s); @@ -2636,6 +3003,10 @@ store_flush_position(XLogRecPtr remote_lsn) { FlushPosition *flushpos; + /* Skip if not the main apply worker */ + if (am_apply_bgworker()) + return; + /* Need to do this in permanent context */ MemoryContextSwitchTo(ApplyContext); @@ -2650,7 +3021,7 @@ store_flush_position(XLogRecPtr remote_lsn) /* Update statistics of the worker. */ -static void +void UpdateWorkerStats(XLogRecPtr last_lsn, TimestampTz send_time, bool reply) { MyLogicalRepWorker->last_lsn = last_lsn; @@ -2812,6 +3183,9 @@ LogicalRepApplyLoop(XLogRecPtr last_received) AcceptInvalidationMessages(); maybe_reread_subscription(); + /* Check the status of apply background worker if any. */ + apply_bgworker_check_status(); + /* Process any table synchronization changes. */ process_syncing_tables(last_received); } @@ -3114,7 +3488,7 @@ maybe_reread_subscription(void) /* * Callback from subscription syscache invalidation. */ -static void +void subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue) { MySubscriptionValid = false; @@ -3710,7 +4084,7 @@ ApplyWorkerMain(Datum main_arg) originid = replorigin_by_name(originname, true); if (!OidIsValid(originid)) originid = replorigin_create(originname); - replorigin_session_setup(originid); + replorigin_session_setup(originid, true); replorigin_session_origin = originid; origin_startpos = replorigin_session_get_progress(false); CommitTransactionCommand(); @@ -3751,13 +4125,14 @@ ApplyWorkerMain(Datum main_arg) server_version = walrcv_server_version(LogRepWorkerWalRcvConn); options.proto.logical.proto_version = + server_version >= 160000 ? LOGICALREP_PROTO_STREAM_PARALLEL_VERSION_NUM : server_version >= 150000 ? LOGICALREP_PROTO_TWOPHASE_VERSION_NUM : server_version >= 140000 ? LOGICALREP_PROTO_STREAM_VERSION_NUM : LOGICALREP_PROTO_VERSION_NUM; options.proto.logical.publication_names = MySubscription->publications; options.proto.logical.binary = MySubscription->binary; - options.proto.logical.streaming = MySubscription->stream; + options.proto.logical.streaming = (MySubscription->stream != SUBSTREAM_OFF); options.proto.logical.twophase = false; options.proto.logical.origin = pstrdup(MySubscription->origin); @@ -3916,7 +4291,8 @@ clear_subscription_skip_lsn(XLogRecPtr finish_lsn) XLogRecPtr myskiplsn = MySubscription->skiplsn; bool started_tx = false; - if (likely(XLogRecPtrIsInvalid(myskiplsn))) + if (likely(XLogRecPtrIsInvalid(myskiplsn)) || + am_apply_bgworker()) return; if (!IsTransactionState()) @@ -3988,7 +4364,7 @@ clear_subscription_skip_lsn(XLogRecPtr finish_lsn) } /* Error callback to give more context info about the change being applied */ -static void +void apply_error_callback(void *arg) { ApplyErrorCallbackArg *errarg = &apply_error_callback_arg; @@ -4016,23 +4392,47 @@ apply_error_callback(void *arg) errarg->remote_xid, LSN_FORMAT_ARGS(errarg->finish_lsn)); } - else if (errarg->remote_attnum < 0) - errcontext("processing remote data for replication origin \"%s\" during \"%s\" for replication target relation \"%s.%s\" in transaction %u finished at %X/%X", - errarg->origin_name, - logicalrep_message_type(errarg->command), - errarg->rel->remoterel.nspname, - errarg->rel->remoterel.relname, - errarg->remote_xid, - LSN_FORMAT_ARGS(errarg->finish_lsn)); else - errcontext("processing remote data for replication origin \"%s\" during \"%s\" for replication target relation \"%s.%s\" column \"%s\" in transaction %u finished at %X/%X", - errarg->origin_name, - logicalrep_message_type(errarg->command), - errarg->rel->remoterel.nspname, - errarg->rel->remoterel.relname, - errarg->rel->remoterel.attnames[errarg->remote_attnum], - errarg->remote_xid, - LSN_FORMAT_ARGS(errarg->finish_lsn)); + { + if (errarg->remote_attnum < 0) + { + if (XLogRecPtrIsInvalid(errarg->finish_lsn)) + errcontext("processing remote data for replication origin \"%s\" during \"%s\" for replication target relation \"%s.%s\" in transaction %u", + errarg->origin_name, + logicalrep_message_type(errarg->command), + errarg->rel->remoterel.nspname, + errarg->rel->remoterel.relname, + errarg->remote_xid); + else + errcontext("processing remote data for replication origin \"%s\" during \"%s\" for replication target relation \"%s.%s\" in transaction %u finished at %X/%X", + errarg->origin_name, + logicalrep_message_type(errarg->command), + errarg->rel->remoterel.nspname, + errarg->rel->remoterel.relname, + errarg->remote_xid, + LSN_FORMAT_ARGS(errarg->finish_lsn)); + } + else + { + if (XLogRecPtrIsInvalid(errarg->finish_lsn)) + errcontext("processing remote data for replication origin \"%s\" during \"%s\" for replication target relation \"%s.%s\" column \"%s\" in transaction %u", + errarg->origin_name, + logicalrep_message_type(errarg->command), + errarg->rel->remoterel.nspname, + errarg->rel->remoterel.relname, + errarg->rel->remoterel.attnames[errarg->remote_attnum], + errarg->remote_xid); + else + errcontext("processing remote data for replication origin \"%s\" during \"%s\" for replication target relation \"%s.%s\" column \"%s\" in transaction %u finished at %X/%X", + errarg->origin_name, + logicalrep_message_type(errarg->command), + errarg->rel->remoterel.nspname, + errarg->rel->remoterel.relname, + errarg->rel->remoterel.attnames[errarg->remote_attnum], + errarg->remote_xid, + LSN_FORMAT_ARGS(errarg->finish_lsn)); + } + } } /* Set transaction information of apply error callback */ diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c index a3c1ba8a40..f9e388ada1 100644 --- a/src/backend/replication/pgoutput/pgoutput.c +++ b/src/backend/replication/pgoutput/pgoutput.c @@ -1843,6 +1843,9 @@ pgoutput_stream_abort(struct LogicalDecodingContext *ctx, XLogRecPtr abort_lsn) { ReorderBufferTXN *toptxn; + PGOutputData *data = (PGOutputData *) ctx->output_plugin_private; + bool write_abort_lsn = (data->protocol_version >= + LOGICALREP_PROTO_STREAM_PARALLEL_VERSION_NUM); /* * The abort should happen outside streaming block, even for streamed @@ -1856,7 +1859,8 @@ pgoutput_stream_abort(struct LogicalDecodingContext *ctx, Assert(rbtxn_is_streamed(toptxn)); OutputPluginPrepareWrite(ctx, true); - logicalrep_write_stream_abort(ctx->out, toptxn->xid, txn->xid); + logicalrep_write_stream_abort(ctx->out, toptxn->xid, txn, abort_lsn, + write_abort_lsn); OutputPluginWrite(ctx, true); cleanup_rel_sync_cache(toptxn->xid, false); diff --git a/src/backend/utils/activity/wait_event.c b/src/backend/utils/activity/wait_event.c index da57a93034..2e146fe087 100644 --- a/src/backend/utils/activity/wait_event.c +++ b/src/backend/utils/activity/wait_event.c @@ -388,6 +388,9 @@ pgstat_get_wait_ipc(WaitEventIPC w) case WAIT_EVENT_HASH_GROW_BUCKETS_REINSERT: event_name = "HashGrowBucketsReinsert"; break; + case WAIT_EVENT_LOGICAL_APPLY_WORKER_STATE_CHANGE: + event_name = "LogicalApplyWorkerStateChange"; + break; case WAIT_EVENT_LOGICAL_SYNC_DATA: event_name = "LogicalSyncData"; break; diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c index af4a1c3068..a40ddf847a 100644 --- a/src/backend/utils/misc/guc.c +++ b/src/backend/utils/misc/guc.c @@ -3220,6 +3220,18 @@ static struct config_int ConfigureNamesInt[] = NULL, NULL, NULL }, + { + {"max_apply_bgworkers_per_subscription", + PGC_SIGHUP, + REPLICATION_SUBSCRIBERS, + gettext_noop("Maximum number of apply background workers per subscription."), + NULL, + }, + &max_apply_bgworkers_per_subscription, + 2, 0, MAX_BACKENDS, + NULL, NULL, NULL + }, + { {"log_rotation_age", PGC_SIGHUP, LOGGING_WHERE, gettext_noop("Sets the amount of time to wait before forcing " diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample index b4bc06e5f5..ad18710af4 100644 --- a/src/backend/utils/misc/postgresql.conf.sample +++ b/src/backend/utils/misc/postgresql.conf.sample @@ -360,6 +360,7 @@ #max_logical_replication_workers = 4 # taken from max_worker_processes # (change requires restart) #max_sync_workers_per_subscription = 2 # taken from max_logical_replication_workers +#max_apply_bgworkers_per_subscription = 2 # taken from max_logical_replication_workers #------------------------------------------------------------------------------ diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index f9c51d1e67..b894cca929 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -4457,7 +4457,7 @@ getSubscriptions(Archive *fout) if (fout->remoteVersion >= 140000) appendPQExpBufferStr(query, " s.substream,\n"); else - appendPQExpBufferStr(query, " false AS substream,\n"); + appendPQExpBufferStr(query, " 'f' AS substream,\n"); if (fout->remoteVersion >= 150000) appendPQExpBufferStr(query, @@ -4594,8 +4594,10 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo) if (strcmp(subinfo->subbinary, "t") == 0) appendPQExpBufferStr(query, ", binary = true"); - if (strcmp(subinfo->substream, "f") != 0) + if (strcmp(subinfo->substream, "t") == 0) appendPQExpBufferStr(query, ", streaming = on"); + else if (strcmp(subinfo->substream, "p") == 0) + appendPQExpBufferStr(query, ", streaming = parallel"); if (strcmp(subinfo->subtwophasestate, two_phase_disabled) != 0) appendPQExpBufferStr(query, ", two_phase = on"); diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h index c9a3026b28..71ad03a934 100644 --- a/src/include/catalog/pg_subscription.h +++ b/src/include/catalog/pg_subscription.h @@ -80,7 +80,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW bool subbinary; /* True if the subscription wants the * publisher to send data in binary */ - bool substream; /* Stream in-progress transactions. */ + char substream; /* Stream in-progress transactions. + * See SUBSTREAM_xxx constants. */ char subtwophasestate; /* Stream two-phase transactions */ @@ -124,7 +125,8 @@ typedef struct Subscription bool enabled; /* Indicates if the subscription is enabled */ bool binary; /* Indicates if the subscription wants data in * binary format */ - bool stream; /* Allow streaming in-progress transactions. */ + char stream; /* Allow streaming in-progress transactions. + * See SUBSTREAM_xxx constants. */ char twophasestate; /* Allow streaming two-phase transactions */ bool disableonerr; /* Indicates if the subscription should be * automatically disabled if a worker error @@ -137,6 +139,21 @@ typedef struct Subscription * specified origin */ } Subscription; +/* Disallow streaming in-progress transactions. */ +#define SUBSTREAM_OFF 'f' + +/* + * Streaming in-progress transactions are written to a temporary file and + * applied only after the transaction is committed on upstream. + */ +#define SUBSTREAM_ON 't' + +/* + * Streaming in-progress transactions are applied immediately via a background + * worker. + */ +#define SUBSTREAM_PARALLEL 'p' + extern Subscription *GetSubscription(Oid subid, bool missing_ok); extern void FreeSubscription(Subscription *sub); extern void DisableSubscription(Oid subid); diff --git a/src/include/replication/logicallauncher.h b/src/include/replication/logicallauncher.h index f1e2821e25..ac8ef94381 100644 --- a/src/include/replication/logicallauncher.h +++ b/src/include/replication/logicallauncher.h @@ -14,6 +14,7 @@ extern PGDLLIMPORT int max_logical_replication_workers; extern PGDLLIMPORT int max_sync_workers_per_subscription; +extern PGDLLIMPORT int max_apply_bgworkers_per_subscription; extern void ApplyLauncherRegister(void); extern void ApplyLauncherMain(Datum main_arg); diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h index a771ab8ff3..eb0fd24fd8 100644 --- a/src/include/replication/logicalproto.h +++ b/src/include/replication/logicalproto.h @@ -32,12 +32,17 @@ * * LOGICALREP_PROTO_TWOPHASE_VERSION_NUM is the minimum protocol version with * support for two-phase commit decoding (at prepare time). Introduced in PG15. + * + * LOGICALREP_PROTO_STREAM_PARALLEL_VERSION_NUM is the minimum protocol version + * with support for streaming large transactions using apply background + * workers. Introduced in PG16. */ #define LOGICALREP_PROTO_MIN_VERSION_NUM 1 #define LOGICALREP_PROTO_VERSION_NUM 1 #define LOGICALREP_PROTO_STREAM_VERSION_NUM 2 #define LOGICALREP_PROTO_TWOPHASE_VERSION_NUM 3 -#define LOGICALREP_PROTO_MAX_VERSION_NUM LOGICALREP_PROTO_TWOPHASE_VERSION_NUM +#define LOGICALREP_PROTO_STREAM_PARALLEL_VERSION_NUM 4 +#define LOGICALREP_PROTO_MAX_VERSION_NUM LOGICALREP_PROTO_STREAM_PARALLEL_VERSION_NUM /* * Logical message types @@ -175,6 +180,17 @@ typedef struct LogicalRepRollbackPreparedTxnData char gid[GIDSIZE]; } LogicalRepRollbackPreparedTxnData; +/* + * Transaction protocol information for stream abort. + */ +typedef struct LogicalRepStreamAbortData +{ + TransactionId xid; + TransactionId subxid; + XLogRecPtr abort_lsn; + TimestampTz abort_time; +} LogicalRepStreamAbortData; + extern void logicalrep_write_begin(StringInfo out, ReorderBufferTXN *txn); extern void logicalrep_read_begin(StringInfo in, LogicalRepBeginData *begin_data); @@ -246,9 +262,12 @@ extern void logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn extern TransactionId logicalrep_read_stream_commit(StringInfo out, LogicalRepCommitData *commit_data); extern void logicalrep_write_stream_abort(StringInfo out, TransactionId xid, - TransactionId subxid); -extern void logicalrep_read_stream_abort(StringInfo in, TransactionId *xid, - TransactionId *subxid); + ReorderBufferTXN *txn, + XLogRecPtr abort_lsn, + bool write_abort_lsn); +extern void logicalrep_read_stream_abort(StringInfo in, + LogicalRepStreamAbortData *abort_data, + bool read_abort_lsn); extern char *logicalrep_message_type(LogicalRepMsgType action); #endif /* LOGICAL_PROTO_H */ diff --git a/src/include/replication/logicalworker.h b/src/include/replication/logicalworker.h index cd1b6e8afc..6a1af7f13c 100644 --- a/src/include/replication/logicalworker.h +++ b/src/include/replication/logicalworker.h @@ -13,6 +13,7 @@ #define LOGICALWORKER_H extern void ApplyWorkerMain(Datum main_arg); +extern void ApplyBgworkerMain(Datum main_arg); extern bool IsLogicalWorker(void); diff --git a/src/include/replication/origin.h b/src/include/replication/origin.h index 2d1b5e5c28..c7389b40a7 100644 --- a/src/include/replication/origin.h +++ b/src/include/replication/origin.h @@ -53,7 +53,7 @@ extern XLogRecPtr replorigin_get_progress(RepOriginId node, bool flush); extern void replorigin_session_advance(XLogRecPtr remote_commit, XLogRecPtr local_commit); -extern void replorigin_session_setup(RepOriginId node); +extern void replorigin_session_setup(RepOriginId node, bool must_acquire); extern void replorigin_session_reset(void); extern XLogRecPtr replorigin_session_get_progress(bool flush); diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h index d109d0baed..d2a80d79e5 100644 --- a/src/include/replication/reorderbuffer.h +++ b/src/include/replication/reorderbuffer.h @@ -301,6 +301,7 @@ typedef struct ReorderBufferTXN { TimestampTz commit_time; TimestampTz prepare_time; + TimestampTz abort_time; } xact_time; /* @@ -647,9 +648,11 @@ extern void ReorderBufferFinishPrepared(ReorderBuffer *rb, TransactionId xid, extern void ReorderBufferAssignChild(ReorderBuffer *, TransactionId, TransactionId, XLogRecPtr commit_lsn); extern void ReorderBufferCommitChild(ReorderBuffer *, TransactionId, TransactionId, XLogRecPtr commit_lsn, XLogRecPtr end_lsn); -extern void ReorderBufferAbort(ReorderBuffer *, TransactionId, XLogRecPtr lsn); +extern void ReorderBufferAbort(ReorderBuffer *, TransactionId, XLogRecPtr lsn, + TimestampTz abort_time); extern void ReorderBufferAbortOld(ReorderBuffer *, TransactionId xid); -extern void ReorderBufferForget(ReorderBuffer *, TransactionId, XLogRecPtr lsn); +extern void ReorderBufferForget(ReorderBuffer *, TransactionId, XLogRecPtr lsn, + TimestampTz abort_time); extern void ReorderBufferInvalidate(ReorderBuffer *, TransactionId, XLogRecPtr lsn); extern void ReorderBufferSetBaseSnapshot(ReorderBuffer *, TransactionId, XLogRecPtr lsn, struct SnapshotData *snap); diff --git a/src/include/replication/worker_internal.h b/src/include/replication/worker_internal.h index 901845abc2..a3560d4904 100644 --- a/src/include/replication/worker_internal.h +++ b/src/include/replication/worker_internal.h @@ -17,8 +17,11 @@ #include "access/xlogdefs.h" #include "catalog/pg_subscription.h" #include "datatype/timestamp.h" +#include "replication/logicalrelation.h" #include "storage/fileset.h" #include "storage/lock.h" +#include "storage/shm_mq.h" +#include "storage/shm_toc.h" #include "storage/spin.h" @@ -60,6 +63,9 @@ typedef struct LogicalRepWorker */ FileSet *stream_fileset; + /* Indicates if this slot is used for an apply background worker. */ + bool subworker; + /* Stats. */ XLogRecPtr last_lsn; TimestampTz last_send_time; @@ -68,8 +74,68 @@ typedef struct LogicalRepWorker TimestampTz reply_time; } LogicalRepWorker; +/* Struct for saving and restoring apply errcontext information */ +typedef struct ApplyErrorCallbackArg +{ + LogicalRepMsgType command; /* 0 if invalid */ + LogicalRepRelMapEntry *rel; + + /* Remote node information */ + int remote_attnum; /* -1 if invalid */ + TransactionId remote_xid; + XLogRecPtr finish_lsn; + char *origin_name; +} ApplyErrorCallbackArg; + +/* + * Status of apply background worker. + */ +typedef enum ApplyBgworkerStatus +{ + APPLY_BGWORKER_BUSY = 0, /* assigned to a transaction */ + APPLY_BGWORKER_FINISHED, /* transaction is completed */ + APPLY_BGWORKER_EXIT /* exit */ +} ApplyBgworkerStatus; + +/* + * Struct for sharing information between apply main and apply background + * workers. + */ +typedef struct ApplyBgworkerShared +{ + slock_t mutex; + + /* Status of apply background worker. */ + ApplyBgworkerStatus status; + + /* server version of publisher. */ + uint32 server_version; + + TransactionId stream_xid; + uint32 n; /* id of apply background worker */ +} ApplyBgworkerShared; + +/* + * Struct for maintaining an apply background worker. + */ +typedef struct ApplyBgworkerState +{ + shm_mq_handle *mq_handle; + dsm_segment *dsm_seg; + ApplyBgworkerShared volatile *shared; +} ApplyBgworkerState; + /* Main memory context for apply worker. Permanent during worker lifetime. */ extern PGDLLIMPORT MemoryContext ApplyContext; +extern PGDLLIMPORT MemoryContext ApplyMessageContext; + +extern PGDLLIMPORT ApplyErrorCallbackArg apply_error_callback_arg; + +extern PGDLLIMPORT bool MySubscriptionValid; + +extern PGDLLIMPORT volatile ApplyBgworkerShared *MyParallelShared; + +extern PGDLLIMPORT List *subxactlist; /* libpqreceiver connection */ extern PGDLLIMPORT struct WalReceiverConn *LogRepWorkerWalRcvConn; @@ -79,18 +145,22 @@ extern PGDLLIMPORT Subscription *MySubscription; extern PGDLLIMPORT LogicalRepWorker *MyLogicalRepWorker; extern PGDLLIMPORT bool in_remote_transaction; +extern PGDLLIMPORT bool in_streamed_transaction; +extern PGDLLIMPORT TransactionId stream_xid; extern void logicalrep_worker_attach(int slot); extern LogicalRepWorker *logicalrep_worker_find(Oid subid, Oid relid, bool only_running); extern List *logicalrep_workers_find(Oid subid, bool only_running); -extern void logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, - Oid userid, Oid relid); +extern bool logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, + Oid userid, Oid relid, + dsm_handle subworker_dsm); extern void logicalrep_worker_stop(Oid subid, Oid relid); extern void logicalrep_worker_wakeup(Oid subid, Oid relid); extern void logicalrep_worker_wakeup_ptr(LogicalRepWorker *worker); extern int logicalrep_sync_worker_count(Oid subid); +extern int logicalrep_apply_bgworker_count(Oid subid); extern void ReplicationOriginNameForTablesync(Oid suboid, Oid relid, char *originname, int szorgname); @@ -103,10 +173,38 @@ extern void process_syncing_tables(XLogRecPtr current_lsn); extern void invalidate_syncing_table_states(Datum arg, int cacheid, uint32 hashvalue); +extern void UpdateWorkerStats(XLogRecPtr last_lsn, TimestampTz send_time, + bool reply); + +extern void apply_dispatch(StringInfo s); + +/* Function for apply error callback */ +extern void apply_error_callback(void *arg); + +extern void subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue); + +/* Apply background worker setup and interactions */ +extern ApplyBgworkerState *apply_bgworker_start(TransactionId xid); +extern ApplyBgworkerState *apply_bgworker_find(TransactionId xid); +extern void apply_bgworker_wait_for(ApplyBgworkerState *wstate, + ApplyBgworkerStatus wait_for_status); +extern void apply_bgworker_send_data(ApplyBgworkerState *wstate, Size nbytes, + const void *data); +extern void apply_bgworker_free(ApplyBgworkerState *wstate); +extern void apply_bgworker_check_status(void); +extern void apply_bgworker_set_status(ApplyBgworkerStatus status); +extern void apply_bgworker_subxact_info_add(TransactionId current_xid); + static inline bool am_tablesync_worker(void) { return OidIsValid(MyLogicalRepWorker->relid); } +static inline bool +am_apply_bgworker(void) +{ + return MyLogicalRepWorker->subworker; +} + #endif /* WORKER_INTERNAL_H */ diff --git a/src/include/utils/wait_event.h b/src/include/utils/wait_event.h index c3ade01120..e35f199fd4 100644 --- a/src/include/utils/wait_event.h +++ b/src/include/utils/wait_event.h @@ -105,6 +105,7 @@ typedef enum WAIT_EVENT_HASH_GROW_BUCKETS_ALLOCATE, WAIT_EVENT_HASH_GROW_BUCKETS_ELECT, WAIT_EVENT_HASH_GROW_BUCKETS_REINSERT, + WAIT_EVENT_LOGICAL_APPLY_WORKER_STATE_CHANGE, WAIT_EVENT_LOGICAL_SYNC_DATA, WAIT_EVENT_LOGICAL_SYNC_STATE_CHANGE, WAIT_EVENT_MQ_INTERNAL, diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out index ef0ebf96b9..4be537ea0b 100644 --- a/src/test/regress/expected/subscription.out +++ b/src/test/regress/expected/subscription.out @@ -219,7 +219,7 @@ ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); DROP SUBSCRIPTION regress_testsub; -- fail - streaming must be boolean CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = foo); -ERROR: streaming requires a Boolean value +ERROR: streaming requires a Boolean value or "parallel" -- now it works CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = true); WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 34a76ceb60..4137dc77b4 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -116,6 +116,10 @@ Append AppendPath AppendRelInfo AppendState +ApplyBgworkerEntry +ApplyBgworkerShared +ApplyBgworkerState +ApplyBgworkerStatus ApplyErrorCallbackArg ApplyExecutionData ApplySubXactData @@ -1485,6 +1489,7 @@ LogicalRepRelId LogicalRepRelMapEntry LogicalRepRelation LogicalRepRollbackPreparedTxnData +LogicalRepStreamAbortData LogicalRepTupleData LogicalRepTyp LogicalRepWorker -- 2.23.0.windows.1 [application/octet-stream] v19-0002-Test-streaming-parallel-option-in-tap-test.patch (69.1K, ../../OS3PR01MB62752B29CEC2592E18236A9A9E909@OS3PR01MB6275.jpnprd01.prod.outlook.com/3-v19-0002-Test-streaming-parallel-option-in-tap-test.patch) download | inline diff: From ef785a3bb9baed1a1f588e7905dac00050448b56 Mon Sep 17 00:00:00 2001 From: "shiy.fnst" <[email protected]> Date: Fri, 13 May 2022 14:50:30 +0800 Subject: [PATCH v19 2/4] Test streaming parallel option in tap test Change all TAP tests using the SUBSCRIPTION "streaming" parameter, so they now test both 'on' and 'parallel' values. --- src/test/subscription/t/015_stream.pl | 199 ++++--- src/test/subscription/t/016_stream_subxact.pl | 119 +++-- src/test/subscription/t/017_stream_ddl.pl | 188 ++++--- .../t/018_stream_subxact_abort.pl | 195 ++++--- .../t/019_stream_subxact_ddl_abort.pl | 110 +++- .../subscription/t/022_twophase_cascade.pl | 363 +++++++------ .../subscription/t/023_twophase_stream.pl | 498 ++++++++++-------- 7 files changed, 1035 insertions(+), 637 deletions(-) diff --git a/src/test/subscription/t/015_stream.pl b/src/test/subscription/t/015_stream.pl index 6561b189de..0bdd234935 100644 --- a/src/test/subscription/t/015_stream.pl +++ b/src/test/subscription/t/015_stream.pl @@ -8,6 +8,116 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" +# parameter so the same code can be run both for the streaming=on and +# streaming=parallel cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_parallel) = @_; + + # Interleave a pair of transactions, each exceeding the 64kB limit. + my $in = ''; + my $out = ''; + + my $timer = IPC::Run::timeout($PostgreSQL::Test::Utils::timeout_default); + + my $h = $node_publisher->background_psql('postgres', \$in, \$out, $timer, + on_error_stop => 0); + + # Check that a background worker starts if "streaming" parameter is + # specified as "parallel". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_parallel) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + $in .= q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + }; + $h->pump_nb; + + if ($is_parallel) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001, 9999) s(i); + DELETE FROM test_tab WHERE a > 5000; + COMMIT; + }); + + $in .= q{ + COMMIT; + \q + }; + $h->finish; # errors make the next test fail, so ignore them here + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), + 'check extra columns contain local defaults'); + + # Test the streaming in binary mode + $node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET (binary = on)"); + + # Insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001, 10000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + COMMIT; + }); + + $node_publisher->wait_for_catchup($appname); + + $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(6667|6667|6667), + 'check extra columns contain local defaults'); + + # Change the local values of the extra columns on the subscriber, + # update publisher, and check that subscriber retains the expected + # values. This is to ensure that non-streaming transactions behave + # properly after a streaming transaction. + $node_subscriber->safe_psql('postgres', + "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'" + ); + $node_publisher->safe_psql('postgres', + "UPDATE test_tab SET b = md5(a::text)"); + + $node_publisher->wait_for_catchup($appname); + + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab" + ); + is($result, qq(6667|6667|6667), + 'check extra columns contain locally changed data'); + + # Cleanup the test data + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE (a > 2)"); + $node_publisher->wait_for_catchup($appname); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -37,6 +147,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -54,82 +168,25 @@ my $result = "SELECT count(*), count(c), count(d = 999) FROM test_tab"); is($result, qq(2|2|2), 'check initial data was copied to subscriber'); -# Interleave a pair of transactions, each exceeding the 64kB limit. -my $in = ''; -my $out = ''; - -my $timer = IPC::Run::timeout($PostgreSQL::Test::Utils::timeout_default); - -my $h = $node_publisher->background_psql('postgres', \$in, \$out, $timer, - on_error_stop => 0); - -$in .= q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -}; -$h->pump_nb; - -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001, 9999) s(i); -DELETE FROM test_tab WHERE a > 5000; -COMMIT; -}); - -$in .= q{ -COMMIT; -\q -}; -$h->finish; # errors make the next test fail, so ignore them here - -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), 'check extra columns contain local defaults'); - -# Test the streaming in binary mode -$node_subscriber->safe_psql('postgres', - "ALTER SUBSCRIPTION tap_sub SET (binary = on)"); - -# Insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001, 10000) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -COMMIT; -}); - -$node_publisher->wait_for_catchup($appname); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(6667|6667|6667), 'check extra columns contain local defaults'); +###################################### +# Test using streaming mode 'parallel' +###################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" +); -# Change the local values of the extra columns on the subscriber, -# update publisher, and check that subscriber retains the expected -# values. This is to ensure that non-streaming transactions behave -# properly after a streaming transaction. $node_subscriber->safe_psql('postgres', - "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'" -); -$node_publisher->safe_psql('postgres', - "UPDATE test_tab SET b = md5(a::text)"); + "ALTER SUBSCRIPTION tap_sub SET(streaming = parallel, binary = off)"); -$node_publisher->wait_for_catchup($appname); +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab" -); -is($result, qq(6667|6667|6667), - 'check extra columns contain locally changed data'); +test_streaming($node_publisher, $node_subscriber, $appname, 1); $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/016_stream_subxact.pl b/src/test/subscription/t/016_stream_subxact.pl index f27f1694f2..45429dddba 100644 --- a/src/test/subscription/t/016_stream_subxact.pl +++ b/src/test/subscription/t/016_stream_subxact.pl @@ -8,6 +8,72 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" +# parameter so the same code can be run both for the streaming=on and +# streaming=parallel cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_parallel) = @_; + + # Check that a background worker starts if "streaming" parameter is + # specified as "parallel". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_parallel) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + # Insert, update and delete enough rows to exceed 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series( 3, 500) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501, 1000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001, 1500) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + SAVEPOINT s3; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501, 2000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + SAVEPOINT s4; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001, 2500) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + COMMIT; + }); + + if ($is_parallel) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(1667|1667|1667), + 'check data was copied to subscriber in streaming mode and extra columns contain local defaults' + ); + + # Cleanup the test data + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE (a > 2)"); + $node_publisher->wait_for_catchup($appname); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -37,6 +103,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -54,41 +124,26 @@ my $result = "SELECT count(*), count(c), count(d = 999) FROM test_tab"); is($result, qq(2|2|2), 'check initial data was copied to subscriber'); -# Insert, update and delete enough rows to exceed 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series( 3, 500) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501, 1000) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001, 1500) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -SAVEPOINT s3; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501, 2000) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -SAVEPOINT s4; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001, 2500) s(i); -UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; -DELETE FROM test_tab WHERE mod(a,3) = 0; -COMMIT; -}); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(1667|1667|1667), - 'check data was copied to subscriber in streaming mode and extra columns contain local defaults' +###################################### +# Test using streaming mode 'parallel' +###################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" ); +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = parallel)"); + +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; + +test_streaming($node_publisher, $node_subscriber, $appname, 1); + $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/017_stream_ddl.pl b/src/test/subscription/t/017_stream_ddl.pl index 0bce63b716..52dfef4780 100644 --- a/src/test/subscription/t/017_stream_ddl.pl +++ b/src/test/subscription/t/017_stream_ddl.pl @@ -8,6 +8,111 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" +# parameter so the same code can be run both for the streaming=on and +# streaming=parallel cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_parallel) = @_; + + # a small (non-streamed) transaction with DDL and DML + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab VALUES (3, md5(3::text)); + ALTER TABLE test_tab ADD COLUMN c INT; + SAVEPOINT s1; + INSERT INTO test_tab VALUES (4, md5(4::text), -4); + COMMIT; + }); + + # Check that a background worker starts if "streaming" parameter is + # specified as "parallel". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_parallel) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + # large (streamed) transaction with DDL and DML + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(5, 1000) s(i); + ALTER TABLE test_tab ADD COLUMN d INT; + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001, 2000) s(i); + COMMIT; + }); + + if ($is_parallel) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + # a small (non-streamed) transaction with DDL and DML + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab VALUES (2001, md5(2001::text), -2001, 2*2001); + ALTER TABLE test_tab ADD COLUMN e INT; + SAVEPOINT s1; + INSERT INTO test_tab VALUES (2002, md5(2002::text), -2002, 2*2002, -3*2002); + COMMIT; + }); + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d), count(e) FROM test_tab"); + is($result, qq(2002|1999|1002|1), + 'check data was copied to subscriber in streaming mode and extra columns contain local defaults' + ); + + # A large (streamed) transaction with DDL and DML. One of the DDL is performed + # after DML to ensure that we invalidate the schema sent for test_tab so that + # the next transaction has to send the schema again. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(2003,5000) s(i); + ALTER TABLE test_tab ADD COLUMN f INT; + COMMIT; + }); + + # A small transaction that won't get streamed. This is just to ensure that we + # send the schema again to reflect the last column added in the previous test. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i, 4*i FROM generate_series(5001,5005) s(i); + COMMIT; + }); + + $node_publisher->wait_for_catchup($appname); + + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d), count(e), count(f) FROM test_tab" + ); + is($result, qq(5005|5002|4005|3004|5), + 'check data was copied to subscriber for both streaming and non-streaming transactions' + ); + + # Cleanup the test data + $node_publisher->safe_psql( + 'postgres', q{ + DELETE FROM test_tab WHERE (a > 2); + ALTER TABLE test_tab DROP COLUMN c, DROP COLUMN d, DROP COLUMN e, DROP COLUMN f; + }); + $node_publisher->wait_for_catchup($appname); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -37,6 +142,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -54,76 +163,25 @@ my $result = "SELECT count(*), count(c), count(d = 999) FROM test_tab"); is($result, qq(2|0|0), 'check initial data was copied to subscriber'); -# a small (non-streamed) transaction with DDL and DML -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab VALUES (3, md5(3::text)); -ALTER TABLE test_tab ADD COLUMN c INT; -SAVEPOINT s1; -INSERT INTO test_tab VALUES (4, md5(4::text), -4); -COMMIT; -}); - -# large (streamed) transaction with DDL and DML -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(5, 1000) s(i); -ALTER TABLE test_tab ADD COLUMN d INT; -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001, 2000) s(i); -COMMIT; -}); - -# a small (non-streamed) transaction with DDL and DML -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab VALUES (2001, md5(2001::text), -2001, 2*2001); -ALTER TABLE test_tab ADD COLUMN e INT; -SAVEPOINT s1; -INSERT INTO test_tab VALUES (2002, md5(2002::text), -2002, 2*2002, -3*2002); -COMMIT; -}); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d), count(e) FROM test_tab"); -is($result, qq(2002|1999|1002|1), - 'check data was copied to subscriber in streaming mode and extra columns contain local defaults' +###################################### +# Test using streaming mode 'parallel' +###################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" ); -# A large (streamed) transaction with DDL and DML. One of the DDL is performed -# after DML to ensure that we invalidate the schema sent for test_tab so that -# the next transaction has to send the schema again. -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(2003,5000) s(i); -ALTER TABLE test_tab ADD COLUMN f INT; -COMMIT; -}); - -# A small transaction that won't get streamed. This is just to ensure that we -# send the schema again to reflect the last column added in the previous test. -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i, 4*i FROM generate_series(5001,5005) s(i); -COMMIT; -}); +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = parallel)"); -$node_publisher->wait_for_catchup($appname); +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d), count(e), count(f) FROM test_tab"); -is($result, qq(5005|5002|4005|3004|5), - 'check data was copied to subscriber for both streaming and non-streaming transactions' -); +test_streaming($node_publisher, $node_subscriber, $appname, 1); $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/018_stream_subxact_abort.pl b/src/test/subscription/t/018_stream_subxact_abort.pl index 7155442e76..68f0e4b0d1 100644 --- a/src/test/subscription/t/018_stream_subxact_abort.pl +++ b/src/test/subscription/t/018_stream_subxact_abort.pl @@ -8,6 +8,113 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" +# parameter so the same code can be run both for the streaming=on and +# streaming=parallel cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_parallel) = @_; + + # Check that a background worker starts if "streaming" parameter is + # specified as "parallel". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_parallel) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + # large (streamed) transaction with DDL, DML and ROLLBACKs + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i); + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501,1000) s(i); + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001,1500) s(i); + SAVEPOINT s3; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501,2000) s(i); + ROLLBACK TO s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001,2500) s(i); + ROLLBACK TO s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2501,3000) s(i); + SAVEPOINT s4; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3001,3500) s(i); + SAVEPOINT s5; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3501,4000) s(i); + COMMIT; + }); + + if ($is_parallel) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c) FROM test_tab"); + is($result, qq(2000|0), + 'check rollback to savepoint was reflected on subscriber and extra columns contain local defaults' + ); + + # large (streamed) transaction with subscriber receiving out of order + # subtransaction ROLLBACKs + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(4001,4500) s(i); + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001,5500) s(i); + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(6001,6500) s(i); + SAVEPOINT s3; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(7001,7500) s(i); + RELEASE s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(8001,8500) s(i); + ROLLBACK TO s1; + COMMIT; + }); + + $node_publisher->wait_for_catchup($appname); + + $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c) FROM test_tab"); + is($result, qq(2500|0), + 'check rollback to savepoint was reflected on subscriber'); + + # large (streamed) transaction with subscriber receiving rollback + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(8501,9000) s(i); + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(9001,9500) s(i); + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(9501,10000) s(i); + ROLLBACK; + }); + + $node_publisher->wait_for_catchup($appname); + + $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c) FROM test_tab"); + is($result, qq(2500|0), 'check rollback was reflected on subscriber'); + + # Cleanup the test data + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE (a > 2)"); + $node_publisher->wait_for_catchup($appname); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -36,6 +143,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -53,81 +164,25 @@ my $result = "SELECT count(*), count(c) FROM test_tab"); is($result, qq(2|0), 'check initial data was copied to subscriber'); -# large (streamed) transaction with DDL, DML and ROLLBACKs -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i); -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501,1000) s(i); -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001,1500) s(i); -SAVEPOINT s3; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501,2000) s(i); -ROLLBACK TO s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001,2500) s(i); -ROLLBACK TO s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2501,3000) s(i); -SAVEPOINT s4; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3001,3500) s(i); -SAVEPOINT s5; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3501,4000) s(i); -COMMIT; -}); - -$node_publisher->wait_for_catchup($appname); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c) FROM test_tab"); -is($result, qq(2000|0), - 'check rollback to savepoint was reflected on subscriber and extra columns contain local defaults' +###################################### +# Test using streaming mode 'parallel' +###################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" ); -# large (streamed) transaction with subscriber receiving out of order -# subtransaction ROLLBACKs -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(4001,4500) s(i); -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001,5500) s(i); -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(6001,6500) s(i); -SAVEPOINT s3; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(7001,7500) s(i); -RELEASE s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(8001,8500) s(i); -ROLLBACK TO s1; -COMMIT; -}); - -$node_publisher->wait_for_catchup($appname); +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = parallel)"); -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c) FROM test_tab"); -is($result, qq(2500|0), - 'check rollback to savepoint was reflected on subscriber'); - -# large (streamed) transaction with subscriber receiving rollback -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(8501,9000) s(i); -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(9001,9500) s(i); -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(9501,10000) s(i); -ROLLBACK; -}); +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c) FROM test_tab"); -is($result, qq(2500|0), 'check rollback was reflected on subscriber'); +test_streaming($node_publisher, $node_subscriber, $appname, 1); $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/019_stream_subxact_ddl_abort.pl b/src/test/subscription/t/019_stream_subxact_ddl_abort.pl index dbd0fca4d1..b276063721 100644 --- a/src/test/subscription/t/019_stream_subxact_ddl_abort.pl +++ b/src/test/subscription/t/019_stream_subxact_ddl_abort.pl @@ -9,6 +9,69 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" +# parameter so the same code can be run both for the streaming=on and +# streaming=parallel cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_parallel) = @_; + + # Check that a background worker starts if "streaming" parameter is + # specified as "parallel". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_parallel) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + # large (streamed) transaction with DDL, DML and ROLLBACKs + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i); + ALTER TABLE test_tab ADD COLUMN c INT; + SAVEPOINT s1; + INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(501,1000) s(i); + ALTER TABLE test_tab ADD COLUMN d INT; + SAVEPOINT s2; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001,1500) s(i); + ALTER TABLE test_tab ADD COLUMN e INT; + SAVEPOINT s3; + INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(1501,2000) s(i); + ALTER TABLE test_tab DROP COLUMN c; + ROLLBACK TO s1; + INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(501,1000) s(i); + COMMIT; + }); + + if ($is_parallel) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + $node_publisher->wait_for_catchup($appname); + + my $result = + $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c) FROM test_tab"); + is($result, qq(1000|500), + 'check rollback to savepoint was reflected on subscriber and extra columns contain local defaults' + ); + + # Cleanup the test data + $node_publisher->safe_psql( + 'postgres', q{ + DELETE FROM test_tab WHERE (a > 2); + ALTER TABLE test_tab DROP COLUMN c; + }); + $node_publisher->wait_for_catchup($appname); +} + # Create publisher node my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); $node_publisher->init(allows_streaming => 'logical'); @@ -37,6 +100,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)" ); @@ -54,35 +121,26 @@ my $result = "SELECT count(*), count(c) FROM test_tab"); is($result, qq(2|0), 'check initial data was copied to subscriber'); -# large (streamed) transaction with DDL, DML and ROLLBACKs -$node_publisher->safe_psql( - 'postgres', q{ -BEGIN; -INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i); -ALTER TABLE test_tab ADD COLUMN c INT; -SAVEPOINT s1; -INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(501,1000) s(i); -ALTER TABLE test_tab ADD COLUMN d INT; -SAVEPOINT s2; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001,1500) s(i); -ALTER TABLE test_tab ADD COLUMN e INT; -SAVEPOINT s3; -INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(1501,2000) s(i); -ALTER TABLE test_tab DROP COLUMN c; -ROLLBACK TO s1; -INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(501,1000) s(i); -COMMIT; -}); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -$node_publisher->wait_for_catchup($appname); - -$result = - $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c) FROM test_tab"); -is($result, qq(1000|500), - 'check rollback to savepoint was reflected on subscriber and extra columns contain local defaults' +###################################### +# Test using streaming mode 'parallel' +###################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" ); +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = parallel)"); + +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; + +test_streaming($node_publisher, $node_subscriber, $appname, 1); + $node_subscriber->stop; $node_publisher->stop; diff --git a/src/test/subscription/t/022_twophase_cascade.pl b/src/test/subscription/t/022_twophase_cascade.pl index 7a797f37ba..0a4152d3be 100644 --- a/src/test/subscription/t/022_twophase_cascade.pl +++ b/src/test/subscription/t/022_twophase_cascade.pl @@ -11,6 +11,208 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" parameter +# so the same code can be run both for the streaming=on and streaming=parallel +# cases. +sub test_streaming +{ + my ($node_A, $node_B, $node_C, $appname_B, $appname_C, $streaming_mode) = + @_; + + my $oldpid_B = $node_A->safe_psql( + 'postgres', " + SELECT pid FROM pg_stat_replication + WHERE application_name = '$appname_B' AND state = 'streaming';"); + my $oldpid_C = $node_B->safe_psql( + 'postgres', " + SELECT pid FROM pg_stat_replication + WHERE application_name = '$appname_C' AND state = 'streaming';"); + + # Setup logical replication streaming mode + + $node_B->safe_psql( + 'postgres', " + ALTER SUBSCRIPTION tap_sub_B + SET (streaming = $streaming_mode);"); + $node_C->safe_psql( + 'postgres', " + ALTER SUBSCRIPTION tap_sub_C + SET (streaming = $streaming_mode)"); + + # Wait for subscribers to finish initialization + + $node_A->poll_query_until( + 'postgres', " + SELECT pid != $oldpid_B FROM pg_stat_replication + WHERE application_name = '$appname_B' AND state = 'streaming';" + ) or die "Timed out while waiting for apply to restart"; + $node_B->poll_query_until( + 'postgres', " + SELECT pid != $oldpid_C FROM pg_stat_replication + WHERE application_name = '$appname_C' AND state = 'streaming';" + ) or die "Timed out while waiting for apply to restart"; + + ############################### + # Test 2PC PREPARE / COMMIT PREPARED. + # 1. Data is streamed as a 2PC transaction. + # 2. Then do commit prepared. + # + # Expect all data is replicated on subscriber(s) after the commit. + ############################### + + # Check that a background worker starts if "streaming" optparameterion is + # specified as "parallel". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($streaming_mode eq 'parallel') + { + $node_B->append_conf('postgresql.conf', "log_min_messages = debug1"); + $node_B->reload; + + $node_C->append_conf('postgresql.conf', "log_min_messages = debug1"); + $node_C->reload; + } + + # Insert, update and delete enough rows to exceed the 64kB limit. + # Then 2PC PREPARE + $node_A->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + if ($streaming_mode eq 'parallel') + { + $node_B->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_B->append_conf('postgresql.conf', "log_min_messages = warning"); + $node_B->reload; + + $node_C->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_C->append_conf('postgresql.conf', "log_min_messages = warning"); + $node_C->reload; + } + + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); + + # check the transaction state is prepared on subscriber(s) + my $result = + $node_B->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber B'); + $result = + $node_C->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber C'); + + # 2PC COMMIT + $node_A->safe_psql('postgres', "COMMIT PREPARED 'test_prepared_tab';"); + + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); + + # check that transaction was committed on subscriber(s) + $result = $node_B->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), + 'Rows inserted by 2PC have committed on subscriber B, and extra columns have local defaults' + ); + $result = $node_C->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), + 'Rows inserted by 2PC have committed on subscriber C, and extra columns have local defaults' + ); + + # check the transaction state is ended on subscriber(s) + $result = + $node_B->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is committed on subscriber B'); + $result = + $node_C->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is committed on subscriber C'); + + ############################### + # Test 2PC PREPARE with a nested ROLLBACK TO SAVEPOINT. + # 0. Cleanup from previous test leaving only 2 rows. + # 1. Insert one more row. + # 2. Record a SAVEPOINT. + # 3. Data is streamed using 2PC. + # 4. Do rollback to SAVEPOINT prior to the streamed inserts. + # 5. Then COMMIT PREPARED. + # + # Expect data after the SAVEPOINT is aborted leaving only 3 rows (= 2 original + 1 from step 1). + ############################### + + # First, delete the data except for 2 rows (delete will be replicated) + $node_A->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); + + # 2PC PREPARE with a nested ROLLBACK TO SAVEPOINT + $node_A->safe_psql( + 'postgres', " + BEGIN; + INSERT INTO test_tab VALUES (9999, 'foobar'); + SAVEPOINT sp_inner; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + ROLLBACK TO SAVEPOINT sp_inner; + PREPARE TRANSACTION 'outer'; + "); + + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); + + # check the transaction state prepared on subscriber(s) + $result = + $node_B->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber B'); + $result = + $node_C->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber C'); + + # 2PC COMMIT + $node_A->safe_psql('postgres', "COMMIT PREPARED 'outer';"); + + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); + + # check the transaction state is ended on subscriber + $result = + $node_B->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is ended on subscriber B'); + $result = + $node_C->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is ended on subscriber C'); + + # check inserts are visible at subscriber(s). + # All the streamed data (prior to the SAVEPOINT) should be rolled back. + # (9999, 'foobar') should be committed. + $result = $node_B->safe_psql('postgres', + "SELECT count(*) FROM test_tab where b = 'foobar';"); + is($result, qq(1), 'Rows committed are present on subscriber B'); + $result = + $node_B->safe_psql('postgres', "SELECT count(*) FROM test_tab;"); + is($result, qq(3), 'Rows committed are present on subscriber B'); + $result = $node_C->safe_psql('postgres', + "SELECT count(*) FROM test_tab where b = 'foobar';"); + is($result, qq(1), 'Rows committed are present on subscriber C'); + $result = + $node_C->safe_psql('postgres', "SELECT count(*) FROM test_tab;"); + is($result, qq(3), 'Rows committed are present on subscriber C'); + + # Cleanup the test data + $node_A->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); + $node_A->wait_for_catchup($appname_B); + $node_B->wait_for_catchup($appname_C); +} + ############################### # Setup a cascade of pub/sub nodes. # node_A -> node_B -> node_C @@ -260,160 +462,15 @@ is($result, qq(21), 'Rows committed are present on subscriber C'); # 2PC + STREAMING TESTS # --------------------- -my $oldpid_B = $node_A->safe_psql( - 'postgres', " - SELECT pid FROM pg_stat_replication - WHERE application_name = '$appname_B' AND state = 'streaming';"); -my $oldpid_C = $node_B->safe_psql( - 'postgres', " - SELECT pid FROM pg_stat_replication - WHERE application_name = '$appname_C' AND state = 'streaming';"); - -# Setup logical replication (streaming = on) - -$node_B->safe_psql( - 'postgres', " - ALTER SUBSCRIPTION tap_sub_B - SET (streaming = on);"); -$node_C->safe_psql( - 'postgres', " - ALTER SUBSCRIPTION tap_sub_C - SET (streaming = on)"); - -# Wait for subscribers to finish initialization - -$node_A->poll_query_until( - 'postgres', " - SELECT pid != $oldpid_B FROM pg_stat_replication - WHERE application_name = '$appname_B' AND state = 'streaming';" -) or die "Timed out while waiting for apply to restart"; -$node_B->poll_query_until( - 'postgres', " - SELECT pid != $oldpid_C FROM pg_stat_replication - WHERE application_name = '$appname_C' AND state = 'streaming';" -) or die "Timed out while waiting for apply to restart"; - -############################### -# Test 2PC PREPARE / COMMIT PREPARED. -# 1. Data is streamed as a 2PC transaction. -# 2. Then do commit prepared. -# -# Expect all data is replicated on subscriber(s) after the commit. -############################### - -# Insert, update and delete enough rows to exceed the 64kB limit. -# Then 2PC PREPARE -$node_A->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_A->wait_for_catchup($appname_B); -$node_B->wait_for_catchup($appname_C); +################################ +# Test using streaming mode 'on' +################################ +test_streaming($node_A, $node_B, $node_C, $appname_B, $appname_C, 'on'); -# check the transaction state is prepared on subscriber(s) -$result = - $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber B'); -$result = - $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber C'); - -# 2PC COMMIT -$node_A->safe_psql('postgres', "COMMIT PREPARED 'test_prepared_tab';"); - -$node_A->wait_for_catchup($appname_B); -$node_B->wait_for_catchup($appname_C); - -# check that transaction was committed on subscriber(s) -$result = $node_B->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), - 'Rows inserted by 2PC have committed on subscriber B, and extra columns have local defaults' -); -$result = $node_C->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), - 'Rows inserted by 2PC have committed on subscriber C, and extra columns have local defaults' -); - -# check the transaction state is ended on subscriber(s) -$result = - $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is committed on subscriber B'); -$result = - $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is committed on subscriber C'); - -############################### -# Test 2PC PREPARE with a nested ROLLBACK TO SAVEPOINT. -# 0. Cleanup from previous test leaving only 2 rows. -# 1. Insert one more row. -# 2. Record a SAVEPOINT. -# 3. Data is streamed using 2PC. -# 4. Do rollback to SAVEPOINT prior to the streamed inserts. -# 5. Then COMMIT PREPARED. -# -# Expect data after the SAVEPOINT is aborted leaving only 3 rows (= 2 original + 1 from step 1). -############################### - -# First, delete the data except for 2 rows (delete will be replicated) -$node_A->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); - -# 2PC PREPARE with a nested ROLLBACK TO SAVEPOINT -$node_A->safe_psql( - 'postgres', " - BEGIN; - INSERT INTO test_tab VALUES (9999, 'foobar'); - SAVEPOINT sp_inner; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - ROLLBACK TO SAVEPOINT sp_inner; - PREPARE TRANSACTION 'outer'; - "); - -$node_A->wait_for_catchup($appname_B); -$node_B->wait_for_catchup($appname_C); - -# check the transaction state prepared on subscriber(s) -$result = - $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber B'); -$result = - $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber C'); - -# 2PC COMMIT -$node_A->safe_psql('postgres', "COMMIT PREPARED 'outer';"); - -$node_A->wait_for_catchup($appname_B); -$node_B->wait_for_catchup($appname_C); - -# check the transaction state is ended on subscriber -$result = - $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is ended on subscriber B'); -$result = - $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is ended on subscriber C'); - -# check inserts are visible at subscriber(s). -# All the streamed data (prior to the SAVEPOINT) should be rolled back. -# (9999, 'foobar') should be committed. -$result = $node_B->safe_psql('postgres', - "SELECT count(*) FROM test_tab where b = 'foobar';"); -is($result, qq(1), 'Rows committed are present on subscriber B'); -$result = $node_B->safe_psql('postgres', "SELECT count(*) FROM test_tab;"); -is($result, qq(3), 'Rows committed are present on subscriber B'); -$result = $node_C->safe_psql('postgres', - "SELECT count(*) FROM test_tab where b = 'foobar';"); -is($result, qq(1), 'Rows committed are present on subscriber C'); -$result = $node_C->safe_psql('postgres', "SELECT count(*) FROM test_tab;"); -is($result, qq(3), 'Rows committed are present on subscriber C'); +###################################### +# Test using streaming mode 'parallel' +###################################### +test_streaming($node_A, $node_B, $node_C, $appname_B, $appname_C, 'parallel'); ############################### # check all the cleanup diff --git a/src/test/subscription/t/023_twophase_stream.pl b/src/test/subscription/t/023_twophase_stream.pl index d8475d25a4..b89414ab74 100644 --- a/src/test/subscription/t/023_twophase_stream.pl +++ b/src/test/subscription/t/023_twophase_stream.pl @@ -8,6 +8,266 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Encapsulate all the common test steps which are related to "streaming" +# parameter so the same code can be run both for the streaming=on and +# streaming=parallel cases. +sub test_streaming +{ + my ($node_publisher, $node_subscriber, $appname, $is_parallel) = @_; + + ############################### + # Test 2PC PREPARE / COMMIT PREPARED. + # 1. Data is streamed as a 2PC transaction. + # 2. Then do commit prepared. + # + # Expect all data is replicated on subscriber side after the commit. + ############################### + + # Check that a background worker starts if "streaming" parameter is + # specified as "parallel". We have to look for the DEBUG1 log messages + # about that, so temporarily bump up the log verbosity. + if ($is_parallel) + { + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = debug1"); + $node_subscriber->reload; + } + + # check that 2PC gets replicated to subscriber + # Insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + if ($is_parallel) + { + $node_subscriber->wait_for_log(qr/\[Apply BGW #\d+\] started/, 0); + $node_subscriber->append_conf('postgresql.conf', + "log_min_messages = warning"); + $node_subscriber->reload; + } + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is in prepared state on subscriber + my $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber'); + + # 2PC transaction gets committed + $node_publisher->safe_psql('postgres', + "COMMIT PREPARED 'test_prepared_tab';"); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is committed on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), + 'Rows inserted by 2PC have committed on subscriber, and extra columns contain local defaults' + ); + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is committed on subscriber'); + + ############################### + # Test 2PC PREPARE / ROLLBACK PREPARED. + # 1. Table is deleted back to 2 rows which are replicated on subscriber. + # 2. Data is streamed using 2PC. + # 3. Do rollback prepared. + # + # Expect data rolls back leaving only the original 2 rows. + ############################### + + # First, delete the data except for 2 rows (will be replicated) + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE a > 2;"); + + # Then insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is in prepared state on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber'); + + # 2PC transaction gets aborted + $node_publisher->safe_psql('postgres', + "ROLLBACK PREPARED 'test_prepared_tab';"); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is aborted on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(2|2|2), + 'Rows inserted by 2PC are rolled back, leaving only the original 2 rows' + ); + + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is aborted on subscriber'); + + ############################### + # Check that 2PC COMMIT PREPARED is decoded properly on crash restart. + # 1. insert, update and delete enough rows to exceed the 64kB limit. + # 2. Then server crashes before the 2PC transaction is committed. + # 3. After servers are restarted the pending transaction is committed. + # + # Expect all data is replicated on subscriber side after the commit. + # Note: both publisher and subscriber do crash/restart. + ############################### + + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + $node_subscriber->stop('immediate'); + $node_publisher->stop('immediate'); + + $node_publisher->start; + $node_subscriber->start; + + # commit post the restart + $node_publisher->safe_psql('postgres', + "COMMIT PREPARED 'test_prepared_tab';"); + $node_publisher->wait_for_catchup($appname); + + # check inserts are visible + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3334|3334|3334), + 'Rows inserted by 2PC have committed on subscriber, and extra columns contain local defaults' + ); + + ############################### + # Do INSERT after the PREPARE but before ROLLBACK PREPARED. + # 1. Table is deleted back to 2 rows which are replicated on subscriber. + # 2. Data is streamed using 2PC. + # 3. A single row INSERT is done which is after the PREPARE. + # 4. Then do a ROLLBACK PREPARED. + # + # Expect the 2PC data rolls back leaving only 3 rows on the subscriber + # (the original 2 + inserted 1). + ############################### + + # First, delete the data except for 2 rows (will be replicated) + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE a > 2;"); + + # Then insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is in prepared state on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber'); + + # Insert a different record (now we are outside of the 2PC transaction) + # Note: the 2PC transaction still holds row locks so make sure this insert is for a separate primary key + $node_publisher->safe_psql('postgres', + "INSERT INTO test_tab VALUES (99999, 'foobar')"); + + # 2PC transaction gets aborted + $node_publisher->safe_psql('postgres', + "ROLLBACK PREPARED 'test_prepared_tab';"); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is aborted on subscriber, + # but the extra INSERT outside of the 2PC still was replicated + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3|3|3), + 'check the outside insert was copied to subscriber'); + + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is aborted on subscriber'); + + ############################### + # Do INSERT after the PREPARE but before COMMIT PREPARED. + # 1. Table is deleted back to 2 rows which are replicated on subscriber. + # 2. Data is streamed using 2PC. + # 3. A single row INSERT is done which is after the PREPARE. + # 4. Then do a COMMIT PREPARED. + # + # Expect 2PC data + the extra row are on the subscriber + # (the 3334 + inserted 1 = 3335). + ############################### + + # First, delete the data except for 2 rows (will be replicated) + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE a > 2;"); + + # Then insert, update and delete enough rows to exceed the 64kB limit. + $node_publisher->safe_psql( + 'postgres', q{ + BEGIN; + INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); + UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; + DELETE FROM test_tab WHERE mod(a,3) = 0; + PREPARE TRANSACTION 'test_prepared_tab';}); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is in prepared state on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(1), 'transaction is prepared on subscriber'); + + # Insert a different record (now we are outside of the 2PC transaction) + # Note: the 2PC transaction still holds row locks so make sure this insert is for a separare primary key + $node_publisher->safe_psql('postgres', + "INSERT INTO test_tab VALUES (99999, 'foobar')"); + + # 2PC transaction gets committed + $node_publisher->safe_psql('postgres', + "COMMIT PREPARED 'test_prepared_tab';"); + + $node_publisher->wait_for_catchup($appname); + + # check that transaction is committed on subscriber + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*), count(c), count(d = 999) FROM test_tab"); + is($result, qq(3335|3335|3335), + 'Rows inserted by 2PC (as well as outside insert) have committed on subscriber, and extra columns contain local defaults' + ); + + $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM pg_prepared_xacts;"); + is($result, qq(0), 'transaction is committed on subscriber'); + + # Cleanup the test data + $node_publisher->safe_psql('postgres', + "DELETE FROM test_tab WHERE a > 2;"); + $node_publisher->wait_for_catchup($appname); +} + ############################### # Setup ############################### @@ -48,6 +308,10 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); my $appname = 'tap_sub'; + +################################ +# Test using streaming mode 'on' +################################ $node_subscriber->safe_psql( 'postgres', " CREATE SUBSCRIPTION tap_sub @@ -70,236 +334,30 @@ my $twophase_query = $node_subscriber->poll_query_until('postgres', $twophase_query) or die "Timed out while waiting for subscriber to enable twophase"; -############################### # Check initial data was copied to subscriber -############################### my $result = $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab"); is($result, qq(2|2|2), 'check initial data was copied to subscriber'); -############################### -# Test 2PC PREPARE / COMMIT PREPARED. -# 1. Data is streamed as a 2PC transaction. -# 2. Then do commit prepared. -# -# Expect all data is replicated on subscriber side after the commit. -############################### - -# check that 2PC gets replicated to subscriber -# Insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is in prepared state on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber'); - -# 2PC transaction gets committed -$node_publisher->safe_psql('postgres', - "COMMIT PREPARED 'test_prepared_tab';"); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is committed on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), - 'Rows inserted by 2PC have committed on subscriber, and extra columns contain local defaults' -); -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is committed on subscriber'); - -############################### -# Test 2PC PREPARE / ROLLBACK PREPARED. -# 1. Table is deleted back to 2 rows which are replicated on subscriber. -# 2. Data is streamed using 2PC. -# 3. Do rollback prepared. -# -# Expect data rolls back leaving only the original 2 rows. -############################### - -# First, delete the data except for 2 rows (will be replicated) -$node_publisher->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); - -# Then insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is in prepared state on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber'); - -# 2PC transaction gets aborted -$node_publisher->safe_psql('postgres', - "ROLLBACK PREPARED 'test_prepared_tab';"); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is aborted on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(2|2|2), - 'Rows inserted by 2PC are rolled back, leaving only the original 2 rows'); - -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is aborted on subscriber'); - -############################### -# Check that 2PC COMMIT PREPARED is decoded properly on crash restart. -# 1. insert, update and delete enough rows to exceed the 64kB limit. -# 2. Then server crashes before the 2PC transaction is committed. -# 3. After servers are restarted the pending transaction is committed. -# -# Expect all data is replicated on subscriber side after the commit. -# Note: both publisher and subscriber do crash/restart. -############################### - -$node_publisher->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_subscriber->stop('immediate'); -$node_publisher->stop('immediate'); - -$node_publisher->start; -$node_subscriber->start; - -# commit post the restart -$node_publisher->safe_psql('postgres', - "COMMIT PREPARED 'test_prepared_tab';"); -$node_publisher->wait_for_catchup($appname); +test_streaming($node_publisher, $node_subscriber, $appname, 0); -# check inserts are visible -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3334|3334|3334), - 'Rows inserted by 2PC have committed on subscriber, and extra columns contain local defaults' +###################################### +# Test using streaming mode 'parallel' +###################################### +my $oldpid = $node_publisher->safe_psql('postgres', + "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" ); -############################### -# Do INSERT after the PREPARE but before ROLLBACK PREPARED. -# 1. Table is deleted back to 2 rows which are replicated on subscriber. -# 2. Data is streamed using 2PC. -# 3. A single row INSERT is done which is after the PREPARE. -# 4. Then do a ROLLBACK PREPARED. -# -# Expect the 2PC data rolls back leaving only 3 rows on the subscriber -# (the original 2 + inserted 1). -############################### - -# First, delete the data except for 2 rows (will be replicated) -$node_publisher->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); - -# Then insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is in prepared state on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber'); - -# Insert a different record (now we are outside of the 2PC transaction) -# Note: the 2PC transaction still holds row locks so make sure this insert is for a separate primary key -$node_publisher->safe_psql('postgres', - "INSERT INTO test_tab VALUES (99999, 'foobar')"); - -# 2PC transaction gets aborted -$node_publisher->safe_psql('postgres', - "ROLLBACK PREPARED 'test_prepared_tab';"); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is aborted on subscriber, -# but the extra INSERT outside of the 2PC still was replicated -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3|3|3), 'check the outside insert was copied to subscriber'); - -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is aborted on subscriber'); - -############################### -# Do INSERT after the PREPARE but before COMMIT PREPARED. -# 1. Table is deleted back to 2 rows which are replicated on subscriber. -# 2. Data is streamed using 2PC. -# 3. A single row INSERT is done which is after the PREPARE. -# 4. Then do a COMMIT PREPARED. -# -# Expect 2PC data + the extra row are on the subscriber -# (the 3334 + inserted 1 = 3335). -############################### - -# First, delete the data except for 2 rows (will be replicated) -$node_publisher->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;"); - -# Then insert, update and delete enough rows to exceed the 64kB limit. -$node_publisher->safe_psql( - 'postgres', q{ - BEGIN; - INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i); - UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0; - DELETE FROM test_tab WHERE mod(a,3) = 0; - PREPARE TRANSACTION 'test_prepared_tab';}); - -$node_publisher->wait_for_catchup($appname); - -# check that transaction is in prepared state on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(1), 'transaction is prepared on subscriber'); - -# Insert a different record (now we are outside of the 2PC transaction) -# Note: the 2PC transaction still holds row locks so make sure this insert is for a separare primary key -$node_publisher->safe_psql('postgres', - "INSERT INTO test_tab VALUES (99999, 'foobar')"); - -# 2PC transaction gets committed -$node_publisher->safe_psql('postgres', - "COMMIT PREPARED 'test_prepared_tab';"); - -$node_publisher->wait_for_catchup($appname); +$node_subscriber->safe_psql('postgres', + "ALTER SUBSCRIPTION tap_sub SET(streaming = parallel)"); -# check that transaction is committed on subscriber -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*), count(c), count(d = 999) FROM test_tab"); -is($result, qq(3335|3335|3335), - 'Rows inserted by 2PC (as well as outside insert) have committed on subscriber, and extra columns contain local defaults' -); +$node_publisher->poll_query_until('postgres', + "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';" + ) + or die + "Timed out while waiting for apply to restart after changing SUBSCRIPTION"; -$result = $node_subscriber->safe_psql('postgres', - "SELECT count(*) FROM pg_prepared_xacts;"); -is($result, qq(0), 'transaction is committed on subscriber'); +test_streaming($node_publisher, $node_subscriber, $appname, 1); ############################### # check all the cleanup -- 2.23.0.windows.1 [application/octet-stream] v19-0003-Add-some-checks-before-using-apply-background-wo.patch (37.8K, ../../OS3PR01MB62752B29CEC2592E18236A9A9E909@OS3PR01MB6275.jpnprd01.prod.outlook.com/4-v19-0003-Add-some-checks-before-using-apply-background-wo.patch) download | inline diff: From 1cb19ad525fc6903f9b55568c16254dca3c6736c Mon Sep 17 00:00:00 2001 From: wangw <[email protected]> Date: Tue, 14 Jun 2022 11:23:52 +0800 Subject: [PATCH v19 3/4] Add some checks before using apply background worker to apply changes. streaming=parallel mode has two requirements: 1) The unique column in the relation on the subscriber-side should also be the unique column on the publisher-side; 2) There cannot be any non-immutable functions used by the subscriber-side replicated table. Look for functions in the following places: * a. Trigger functions * b. Column default value expressions and domain constraints * c. Constraint expressions * d. Foreign keys --- doc/src/sgml/ref/create_subscription.sgml | 4 + .../replication/logical/applybgworker.c | 44 ++ src/backend/replication/logical/proto.c | 88 +++- src/backend/replication/logical/relation.c | 201 +++++++++ src/backend/replication/logical/tablesync.c | 1 + src/backend/replication/logical/worker.c | 23 +- src/backend/utils/cache/typcache.c | 17 + src/include/replication/logicalproto.h | 1 + src/include/replication/logicalrelation.h | 15 + src/include/replication/worker_internal.h | 1 + src/include/utils/typcache.h | 2 + .../subscription/t/022_twophase_cascade.pl | 6 + .../subscription/t/032_streaming_apply.pl | 380 ++++++++++++++++++ src/tools/pgindent/typedefs.list | 1 + 14 files changed, 775 insertions(+), 9 deletions(-) create mode 100644 src/test/subscription/t/032_streaming_apply.pl diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml index b08e4b5580..832899570f 100644 --- a/doc/src/sgml/ref/create_subscription.sgml +++ b/doc/src/sgml/ref/create_subscription.sgml @@ -240,6 +240,10 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl transaction is committed. Note that if an error happens when applying changes in a background worker, the finish LSN of the remote transaction might not be reported in the server log. + <literal>parallel</literal> mode has two requirements: 1) the unique + column in the relation on the subscriber-side should also be the + unique column on the publisher-side; 2) there cannot be any + non-immutable functions used by the subscriber-side replicated table. </para> </listitem> </varlistentry> diff --git a/src/backend/replication/logical/applybgworker.c b/src/backend/replication/logical/applybgworker.c index aa222490a0..89c712f785 100644 --- a/src/backend/replication/logical/applybgworker.c +++ b/src/backend/replication/logical/applybgworker.c @@ -800,3 +800,47 @@ apply_bgworker_subxact_info_add(TransactionId current_xid) MemoryContextSwitchTo(oldctx); } } + +/* + * Check if changes on this relation can be applied by an apply background + * worker. + * + * Although the commit order is maintained only allowing one process to commit + * at a time, the access order to the relation has changed. This could cause + * unexpected problems if the unique column on the replicated table is + * inconsistent with the publisher-side or contains non-immutable functions + * when applying transactions in the apply background worker. + */ +void +apply_bgworker_relation_check(LogicalRepRelMapEntry *rel) +{ + /* Skip check if not an apply background worker. */ + if (!am_apply_bgworker()) + return; + + /* + * Partition table checks are done later in function + * apply_handle_tuple_routing. + */ + if (rel->localrel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + return; + + /* + * Return if changes on this relation can be applied by an apply background + * worker. + */ + if (rel->parallel_apply == PARALLEL_APPLY_SAFE) + return; + + /* We are in error mode and should give user correct error. */ + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("cannot replicate target relation \"%s.%s\" using " + "subscription parameter streaming=parallel", + rel->remoterel.nspname, rel->remoterel.relname), + errdetail("The unique column on subscriber is not the unique " + "column on publisher or there is at least one " + "non-immutable function."), + errhint("Please change to use subscription parameter " + "streaming=on."))); +} diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c index 47bd811fb7..511bd9c052 100644 --- a/src/backend/replication/logical/proto.c +++ b/src/backend/replication/logical/proto.c @@ -23,7 +23,8 @@ /* * Protocol message flags. */ -#define LOGICALREP_IS_REPLICA_IDENTITY 1 +#define ATTR_IS_REPLICA_IDENTITY (1 << 0) +#define ATTR_IS_UNIQUE (1 << 1) #define MESSAGE_TRANSACTIONAL (1<<0) #define TRUNCATE_CASCADE (1<<0) @@ -40,6 +41,68 @@ static void logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple); static void logicalrep_write_namespace(StringInfo out, Oid nspid); static const char *logicalrep_read_namespace(StringInfo in); +static Bitmapset *RelationGetUniqueKeyBitmap(Relation rel); + +/* + * RelationGetUniqueKeyBitmap -- get a bitmap of unique attribute numbers + * + * This is similar to RelationGetIdentityKeyBitmap(), but returns a bitmap of + * index attribute numbers for all unique indexes. + */ +static Bitmapset * +RelationGetUniqueKeyBitmap(Relation rel) +{ + List *indexoidlist = NIL; + ListCell *indexoidscan; + Bitmapset *attunique = NULL; + + if (!rel->rd_rel->relhasindex) + return NULL; + + indexoidlist = RelationGetIndexList(rel); + + foreach(indexoidscan, indexoidlist) + { + Oid indexoid = lfirst_oid(indexoidscan); + Relation indexRel; + int i; + + /* Look up the description for index */ + indexRel = RelationIdGetRelation(indexoid); + + if (!RelationIsValid(indexRel)) + elog(ERROR, "could not open relation with OID %u", indexoid); + + if (!indexRel->rd_index->indisunique) + { + RelationClose(indexRel); + continue; + } + + /* Add referenced attributes to idindexattrs */ + for (i = 0; i < indexRel->rd_index->indnatts; i++) + { + int attrnum = indexRel->rd_index->indkey.values[i]; + + /* + * We don't include non-key columns into idindexattrs + * bitmaps. See RelationGetIndexAttrBitmap. + */ + if (attrnum != 0) + { + if (i < indexRel->rd_index->indnkeyatts && + !bms_is_member(attrnum - FirstLowInvalidHeapAttributeNumber, attunique)) + attunique = bms_add_member(attunique, + attrnum - FirstLowInvalidHeapAttributeNumber); + } + } + RelationClose(indexRel); + } + list_free(indexoidlist); + + return attunique; +} + /* * Check if a column is covered by a column list. * @@ -933,7 +996,8 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns) TupleDesc desc; int i; uint16 nliveatts = 0; - Bitmapset *idattrs = NULL; + Bitmapset *idattrs = NULL, + *attunique = NULL; bool replidentfull; desc = RelationGetDescr(rel); @@ -958,6 +1022,9 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns) if (!replidentfull) idattrs = RelationGetIdentityKeyBitmap(rel); + /* fetch bitmap of UNIQUE attributes */ + attunique = RelationGetUniqueKeyBitmap(rel); + /* send the attributes */ for (i = 0; i < desc->natts; i++) { @@ -974,7 +1041,11 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns) if (replidentfull || bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber, idattrs)) - flags |= LOGICALREP_IS_REPLICA_IDENTITY; + flags |= ATTR_IS_REPLICA_IDENTITY; + + if (bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber, + attunique)) + flags |= ATTR_IS_UNIQUE; pq_sendbyte(out, flags); @@ -989,6 +1060,7 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns) } bms_free(idattrs); + bms_free(attunique); } /* @@ -1001,7 +1073,8 @@ logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel) int natts; char **attnames; Oid *atttyps; - Bitmapset *attkeys = NULL; + Bitmapset *attkeys = NULL, + *attunique = NULL; natts = pq_getmsgint(in, 2); attnames = palloc(natts * sizeof(char *)); @@ -1014,9 +1087,13 @@ logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel) /* Check for replica identity column */ flags = pq_getmsgbyte(in); - if (flags & LOGICALREP_IS_REPLICA_IDENTITY) + if (flags & ATTR_IS_REPLICA_IDENTITY) attkeys = bms_add_member(attkeys, i); + /* Check for unique column */ + if (flags & ATTR_IS_UNIQUE) + attunique = bms_add_member(attunique, i); + /* attribute name */ attnames[i] = pstrdup(pq_getmsgstring(in)); @@ -1030,6 +1107,7 @@ logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel) rel->attnames = attnames; rel->atttyps = atttyps; rel->attkeys = attkeys; + rel->attunique = attunique; rel->natts = natts; } diff --git a/src/backend/replication/logical/relation.c b/src/backend/replication/logical/relation.c index e989047681..37e410b8d0 100644 --- a/src/backend/replication/logical/relation.c +++ b/src/backend/replication/logical/relation.c @@ -19,12 +19,19 @@ #include "access/table.h" #include "catalog/namespace.h" +#include "catalog/pg_proc.h" #include "catalog/pg_subscription_rel.h" +#include "commands/trigger.h" #include "executor/executor.h" #include "nodes/makefuncs.h" +#include "optimizer/optimizer.h" #include "replication/logicalrelation.h" #include "replication/worker_internal.h" +#include "rewrite/rewriteHandler.h" #include "utils/inval.h" +#include "utils/lsyscache.h" +#include "utils/syscache.h" +#include "utils/typcache.h" static MemoryContext LogicalRepRelMapContext = NULL; @@ -91,6 +98,26 @@ logicalrep_relmap_invalidate_cb(Datum arg, Oid reloid) } } +/* + * Relcache invalidation callback to reset parallel flag. + */ +static void +logicalrep_relmap_reset_parallel_cb(Datum arg, int cacheid, uint32 hashvalue) +{ + HASH_SEQ_STATUS hash_seq; + LogicalRepRelMapEntry *entry; + + if (LogicalRepRelMap == NULL) + return; + + hash_seq_init(&hash_seq, LogicalRepRelMap); + while ((entry = hash_seq_search(&hash_seq)) != NULL) + { + entry->parallel_apply = PARALLEL_APPLY_UNKNOWN; + entry->localrelvalid = false; + } +} + /* * Initialize the relation map cache. */ @@ -116,6 +143,9 @@ logicalrep_relmap_init(void) /* Watch for invalidation events. */ CacheRegisterRelcacheCallback(logicalrep_relmap_invalidate_cb, (Datum) 0); + CacheRegisterSyscacheCallback(PROCOID, + logicalrep_relmap_reset_parallel_cb, + (Datum) 0); } /* @@ -142,6 +172,7 @@ logicalrep_relmap_free_entry(LogicalRepRelMapEntry *entry) pfree(remoterel->atttyps); } bms_free(remoterel->attkeys); + bms_free(remoterel->attunique); if (entry->attrmap) free_attrmap(entry->attrmap); @@ -190,6 +221,7 @@ logicalrep_relmap_update(LogicalRepRelation *remoterel) } entry->remoterel.replident = remoterel->replident; entry->remoterel.attkeys = bms_copy(remoterel->attkeys); + entry->remoterel.attunique = bms_copy(remoterel->attunique); MemoryContextSwitchTo(oldctx); } @@ -310,6 +342,168 @@ logicalrep_rel_mark_updatable(LogicalRepRelMapEntry *entry) } } +/* + * Check if changes on one relation can be applied by an apply background + * worker and assign the 'parallel_apply' flag. + * + * There are two requirements for applying changes in an apply background + * worker: 1) The unique column in the relation on the subscriber-side should + * also be the unique column on the publisher-side; 2) There cannot be any + * non-immutable functions used by the subscriber-side. + * + * We just mark the relation entry as 'PARALLEL_APPLY_UNSAFE' here if changes + * on one relation can not be applied by an apply background worker and leave + * it to apply_bgworker_relation_check() to throw the actual error if needed. + */ +static void +logicalrep_rel_mark_parallel_apply(LogicalRepRelMapEntry *entry) +{ + Bitmapset *ukey; + int i; + TupleDesc tupdesc; + int attnum; + List *fkeys = NIL; + + /* Fast path if 'parallel_apply' flag is already known. */ + if (entry->parallel_apply != PARALLEL_APPLY_UNKNOWN) + return; + + /* Initialize the flag. */ + entry->parallel_apply = PARALLEL_APPLY_SAFE; + + /* + * First, check if the unique column in the relation on the subscriber-side + * is also the unique column on the publisher-side. + */ + ukey = RelationGetIndexAttrBitmap(entry->localrel, + INDEX_ATTR_BITMAP_KEY); + + if (ukey) + { + i = -1; + while ((i = bms_next_member(ukey, i)) >= 0) + { + attnum = AttrNumberGetAttrOffset(i + FirstLowInvalidHeapAttributeNumber); + + if (entry->attrmap->attnums[attnum] < 0 || + !bms_is_member(entry->attrmap->attnums[attnum], entry->remoterel.attunique)) + { + entry->parallel_apply = PARALLEL_APPLY_UNSAFE; + return; + } + } + + bms_free(ukey); + } + + /* + * Then, check if there is any non-immutable function used by the local + * table. Look for functions in the following places: + * a. trigger functions; + * b. Column default value expressions and domain constraints; + * c. Constraint expressions; + * d. Foreign keys. + */ + /* Check the trigger functions. */ + if (entry->localrel->trigdesc != NULL) + { + for (i = 0; i < entry->localrel->trigdesc->numtriggers; i++) + { + Trigger *trig = entry->localrel->trigdesc->triggers + i; + + if (trig->tgenabled != TRIGGER_FIRES_ALWAYS && + trig->tgenabled != TRIGGER_FIRES_ON_REPLICA) + continue; + + if (func_volatile(trig->tgfoid) != PROVOLATILE_IMMUTABLE) + { + entry->parallel_apply = PARALLEL_APPLY_UNSAFE; + return; + } + } + } + + /* Check the columns. */ + tupdesc = RelationGetDescr(entry->localrel); + for (attnum = 0; attnum < tupdesc->natts; attnum++) + { + Form_pg_attribute att = TupleDescAttr(tupdesc, attnum); + + /* We don't need info for dropped or generated attributes */ + if (att->attisdropped || att->attgenerated) + continue; + + /* + * We don't need to check columns that only exist on the + * subscriber + */ + if (entry->attrmap->attnums[attnum] < 0) + continue; + + if (att->atthasdef) + { + Node *defaultexpr; + + defaultexpr = build_column_default(entry->localrel, attnum + 1); + if (contain_mutable_functions(defaultexpr)) + { + entry->parallel_apply = PARALLEL_APPLY_UNSAFE; + return; + } + } + + /* + * If the column is of a DOMAIN type, determine whether + * that domain has any CHECK expressions that are not + * immutable. + */ + if (get_typtype(att->atttypid) == TYPTYPE_DOMAIN) + { + List *domain_constraints; + ListCell *lc; + + domain_constraints = GetDomainConstraints(att->atttypid); + + foreach(lc, domain_constraints) + { + DomainConstraintState *con = (DomainConstraintState *) lfirst(lc); + + if (con->check_expr && contain_mutable_functions((Node *) con->check_expr)) + { + entry->parallel_apply = PARALLEL_APPLY_UNSAFE; + return; + } + } + } + } + + /* Check the constraints. */ + if (tupdesc->constr) + { + ConstrCheck *check = tupdesc->constr->check; + + /* + * Determine if there are any CHECK constraints which + * contains non-immutable function. + */ + for (i = 0; i < tupdesc->constr->num_check; i++) + { + Expr *check_expr = stringToNode(check[i].ccbin); + + if (contain_mutable_functions((Node *) check_expr)) + { + entry->parallel_apply = PARALLEL_APPLY_UNSAFE; + return; + } + } + } + + /* Check the foreign keys. */ + fkeys = RelationGetFKeyList(entry->localrel); + if (fkeys) + entry->parallel_apply = PARALLEL_APPLY_UNSAFE; +} + /* * Open the local relation associated with the remote one. * @@ -438,6 +632,9 @@ logicalrep_rel_open(LogicalRepRelId remoteid, LOCKMODE lockmode) */ logicalrep_rel_mark_updatable(entry); + /* Set if changes could be applied in the apply background worker. */ + logicalrep_rel_mark_parallel_apply(entry); + entry->localrelvalid = true; } @@ -653,6 +850,7 @@ logicalrep_partition_open(LogicalRepRelMapEntry *root, } entry->remoterel.replident = remoterel->replident; entry->remoterel.attkeys = bms_copy(remoterel->attkeys); + entry->remoterel.attunique = bms_copy(remoterel->attunique); } entry->localrel = partrel; @@ -696,6 +894,9 @@ logicalrep_partition_open(LogicalRepRelMapEntry *root, /* Set if the table's replica identity is enough to apply update/delete. */ logicalrep_rel_mark_updatable(entry); + /* Set if changes could be applied in the apply background worker. */ + logicalrep_rel_mark_parallel_apply(entry); + entry->localrelvalid = true; /* state and statelsn are left set to 0. */ diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c index 8ffba7e2e5..3cdbf8b457 100644 --- a/src/backend/replication/logical/tablesync.c +++ b/src/backend/replication/logical/tablesync.c @@ -884,6 +884,7 @@ fetch_remote_table_info(char *nspname, char *relname, lrel->attnames = palloc0(MaxTupleAttributeNumber * sizeof(char *)); lrel->atttyps = palloc0(MaxTupleAttributeNumber * sizeof(Oid)); lrel->attkeys = NULL; + lrel->attunique = NULL; /* * Store the columns as a list of names. Ignore those that are not diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c index 2aa7797628..41793d26c2 100644 --- a/src/backend/replication/logical/worker.c +++ b/src/backend/replication/logical/worker.c @@ -1391,6 +1391,14 @@ apply_handle_stream_stop(StringInfo s) { char action = LOGICAL_REP_MSG_STREAM_STOP; + /* + * Unlike stream_commit, we don't need to wait here for stream_stop to + * finish. Allowing the other transaction to be applied before + * stream_stop is finished can lead to failures if the unique + * index/constraint is different between publisher and subscriber. But + * for such cases, we don't allow streamed transactions to be applied + * in parallel. See apply_bgworker_relation_check. + */ apply_bgworker_send_data(stream_apply_worker, 1, &action); elog(DEBUG1, "stopped streaming of xid %u, %u changes streamed", stream_xid, nchanges); @@ -2044,6 +2052,8 @@ apply_handle_insert(StringInfo s) /* Set relation for error callback */ apply_error_callback_arg.rel = rel; + apply_bgworker_relation_check(rel); + /* Initialize the executor state. */ edata = create_edata_for_relation(rel); estate = edata->estate; @@ -2187,6 +2197,8 @@ apply_handle_update(StringInfo s) /* Check if we can do the update. */ check_relation_updatable(rel); + apply_bgworker_relation_check(rel); + /* Initialize the executor state. */ edata = create_edata_for_relation(rel); estate = edata->estate; @@ -2355,6 +2367,8 @@ apply_handle_delete(StringInfo s) /* Check if we can do the delete. */ check_relation_updatable(rel); + apply_bgworker_relation_check(rel); + /* Initialize the executor state. */ edata = create_edata_for_relation(rel); estate = edata->estate; @@ -2540,13 +2554,14 @@ apply_handle_tuple_routing(ApplyExecutionData *edata, } MemoryContextSwitchTo(oldctx); + part_entry = logicalrep_partition_open(relmapentry, partrel, + attrmap); + /* Check if we can do the update or delete on the leaf partition. */ if (operation == CMD_UPDATE || operation == CMD_DELETE) - { - part_entry = logicalrep_partition_open(relmapentry, partrel, - attrmap); check_relation_updatable(part_entry); - } + + apply_bgworker_relation_check(part_entry); switch (operation) { diff --git a/src/backend/utils/cache/typcache.c b/src/backend/utils/cache/typcache.c index 808f9ebd0d..b248899d82 100644 --- a/src/backend/utils/cache/typcache.c +++ b/src/backend/utils/cache/typcache.c @@ -2540,6 +2540,23 @@ compare_values_of_enum(TypeCacheEntry *tcache, Oid arg1, Oid arg2) return 0; } +/* + * GetDomainConstraints --- get DomainConstraintState list of specified domain type + */ +List * +GetDomainConstraints(Oid type_id) +{ + TypeCacheEntry *typentry; + List *constraints = NIL; + + typentry = lookup_type_cache(type_id, TYPECACHE_DOMAIN_CONSTR_INFO); + + if(typentry->domainData != NULL) + constraints = typentry->domainData->constraints; + + return constraints; +} + /* * Load (or re-load) the enumData member of the typcache entry. */ diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h index eb0fd24fd8..4395c11f75 100644 --- a/src/include/replication/logicalproto.h +++ b/src/include/replication/logicalproto.h @@ -113,6 +113,7 @@ typedef struct LogicalRepRelation char replident; /* replica identity */ char relkind; /* remote relation kind */ Bitmapset *attkeys; /* Bitmap of key columns */ + Bitmapset *attunique; /* Bitmap of unique columns */ } LogicalRepRelation; /* Type mapping info */ diff --git a/src/include/replication/logicalrelation.h b/src/include/replication/logicalrelation.h index 78cd7e77f5..8011e648d7 100644 --- a/src/include/replication/logicalrelation.h +++ b/src/include/replication/logicalrelation.h @@ -15,6 +15,19 @@ #include "access/attmap.h" #include "replication/logicalproto.h" +/* + * States to determine if changes on one relation can be applied using an + * apply background worker. + */ +typedef enum ParalleApplySafety +{ + PARALLEL_APPLY_UNKNOWN = 0, /* unknown */ + PARALLEL_APPLY_SAFE, /* Can apply changes in an apply background + worker */ + PARALLEL_APPLY_UNSAFE /* Can not apply changes in an apply background + worker */ +} ParalleApplySafety; + typedef struct LogicalRepRelMapEntry { LogicalRepRelation remoterel; /* key is remoterel.remoteid */ @@ -31,6 +44,8 @@ typedef struct LogicalRepRelMapEntry Relation localrel; /* relcache entry (NULL when closed) */ AttrMap *attrmap; /* map of local attributes to remote ones */ bool updatable; /* Can apply updates/deletes? */ + ParalleApplySafety parallel_apply; /* Can apply changes in an apply + background worker? */ /* Sync state. */ char state; diff --git a/src/include/replication/worker_internal.h b/src/include/replication/worker_internal.h index a3560d4904..1c0db05c8a 100644 --- a/src/include/replication/worker_internal.h +++ b/src/include/replication/worker_internal.h @@ -194,6 +194,7 @@ extern void apply_bgworker_free(ApplyBgworkerState *wstate); extern void apply_bgworker_check_status(void); extern void apply_bgworker_set_status(ApplyBgworkerStatus status); extern void apply_bgworker_subxact_info_add(TransactionId current_xid); +extern void apply_bgworker_relation_check(LogicalRepRelMapEntry *rel); static inline bool am_tablesync_worker(void) diff --git a/src/include/utils/typcache.h b/src/include/utils/typcache.h index 431ad7f1b3..ed7c2e7f48 100644 --- a/src/include/utils/typcache.h +++ b/src/include/utils/typcache.h @@ -199,6 +199,8 @@ extern uint64 assign_record_type_identifier(Oid type_id, int32 typmod); extern int compare_values_of_enum(TypeCacheEntry *tcache, Oid arg1, Oid arg2); +extern List *GetDomainConstraints(Oid type_id); + extern size_t SharedRecordTypmodRegistryEstimate(void); extern void SharedRecordTypmodRegistryInit(SharedRecordTypmodRegistry *, diff --git a/src/test/subscription/t/022_twophase_cascade.pl b/src/test/subscription/t/022_twophase_cascade.pl index 0a4152d3be..30a01f7305 100644 --- a/src/test/subscription/t/022_twophase_cascade.pl +++ b/src/test/subscription/t/022_twophase_cascade.pl @@ -39,6 +39,12 @@ sub test_streaming ALTER SUBSCRIPTION tap_sub_C SET (streaming = $streaming_mode)"); + if ($streaming_mode eq 'parallel') + { + $node_C->safe_psql( + 'postgres', "ALTER TABLE test_tab ALTER c DROP DEFAULT"); + } + # Wait for subscribers to finish initialization $node_A->poll_query_until( diff --git a/src/test/subscription/t/032_streaming_apply.pl b/src/test/subscription/t/032_streaming_apply.pl new file mode 100644 index 0000000000..7f8bfa6745 --- /dev/null +++ b/src/test/subscription/t/032_streaming_apply.pl @@ -0,0 +1,380 @@ +# Copyright (c) 2022, PostgreSQL Global Development Group + +# Test the restrictions of streaming mode "parallel" in logical replication + +use strict; +use warnings; +use PostgreSQL::Test::Cluster; +use PostgreSQL::Test::Utils; +use Test::More; + +my $offset = 0; + +# Create publisher node +my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); +$node_publisher->init(allows_streaming => 'logical'); +$node_publisher->append_conf('postgresql.conf', + 'logical_decoding_work_mem = 64kB'); +$node_publisher->start; + +# Create subscriber node +my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber'); +$node_subscriber->init; +$node_subscriber->start; + +# Setup structure on publisher +$node_publisher->safe_psql('postgres', + "CREATE TABLE test_tab (a int primary key, b varchar)"); +$node_publisher->safe_psql('postgres', + "CREATE TABLE test_tab_partitioned (a int primary key, b varchar)"); + +# Setup structure on subscriber +# We need to test normal table and partition table. +$node_subscriber->safe_psql('postgres', + "CREATE TABLE test_tab (a int primary key, b varchar)"); +$node_subscriber->safe_psql('postgres', + "CREATE TABLE test_tab_partitioned (a int primary key, b varchar) PARTITION BY RANGE(a)"); +$node_subscriber->safe_psql('postgres', + "CREATE TABLE test_tab_partition (LIKE test_tab_partitioned)"); +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partitioned ATTACH PARTITION test_tab_partition DEFAULT" +); + +# Setup logical replication +my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres'; +$node_publisher->safe_psql('postgres', + "CREATE PUBLICATION tap_pub FOR TABLE test_tab"); +$node_publisher->safe_psql('postgres', + "CREATE PUBLICATION tap_pub_partitioned FOR TABLE test_tab_partitioned"); + +my $appname = 'tap_sub'; +$node_subscriber->safe_psql( + 'postgres', " + CREATE SUBSCRIPTION tap_sub + CONNECTION '$publisher_connstr application_name=$appname' + PUBLICATION tap_pub, tap_pub_partitioned + WITH (streaming = parallel, copy_data = false)"); + +$node_publisher->wait_for_catchup($appname); + +# It is not allowed that the unique index on the publisher and the subscriber +# is different. Check the error reported by background worker in this case. +# First we check the unique index on normal table. +$node_subscriber->safe_psql('postgres', + "CREATE UNIQUE INDEX test_tab_b_idx ON test_tab (b)"); + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log( + qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab" using subscription parameter streaming=parallel/, + $offset); + +# Drop the unique index on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', "DROP INDEX test_tab_b_idx"); + +$node_publisher->wait_for_catchup($appname); + +my $result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); +is($result, qq(5000), 'data replicated to subscriber after dropping index'); + +# Then we check the unique index on partition table. +$node_subscriber->safe_psql('postgres', + "CREATE UNIQUE INDEX test_tab_b_partition_idx ON test_tab_partition (b)"); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab_partitioned SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log( + qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab_partitioned" using subscription parameter streaming=parallel/, + $offset); + +# Drop the unique index on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', "DROP INDEX test_tab_b_partition_idx"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); +is($result, qq(5000), 'data replicated to subscriber after dropping index'); + +# Triggers which execute non-immutable function are not allowed on the +# subscriber side. Check the error reported by background worker in this case. +# First we check the trigger function on normal table. +$node_subscriber->safe_psql( + 'postgres', qq{ +CREATE FUNCTION trigger_func() RETURNS TRIGGER AS \$\$ + BEGIN + RETURN NULL; + END +\$\$ language plpgsql; +CREATE TRIGGER insert_trig +BEFORE INSERT ON test_tab +FOR EACH ROW EXECUTE PROCEDURE trigger_func(); +ALTER TABLE test_tab ENABLE REPLICA TRIGGER insert_trig; +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', "DELETE FROM test_tab"); + +$node_subscriber->wait_for_log( + qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab" using subscription parameter streaming=parallel/, + $offset); + +# Drop the trigger on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "DROP TRIGGER insert_trig ON test_tab"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); +is($result, qq(0), 'data replicated to subscriber after dropping trigger'); + +# Then we check the trigger function on partition table. +$node_subscriber->safe_psql( + 'postgres', qq{ +CREATE TRIGGER insert_trig +BEFORE INSERT ON test_tab_partition +FOR EACH ROW EXECUTE PROCEDURE trigger_func(); +ALTER TABLE test_tab_partition ENABLE REPLICA TRIGGER insert_trig; +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', "DELETE FROM test_tab_partitioned"); + +$node_subscriber->wait_for_log( + qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab_partitioned" using subscription parameter streaming=parallel/, + $offset); + +# Drop the trigger on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "DROP TRIGGER insert_trig ON test_tab_partition"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); +is($result, qq(0), 'data replicated to subscriber after dropping trigger'); + +# It is not allowed that column default value expression contains a +# non-immutable function on the subscriber side. Check the error reported by +# background worker in this case. +# First we check the column default value expression on normal table. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab ALTER COLUMN b SET DEFAULT random()"); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log( + qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab" using subscription parameter streaming=parallel/, + $offset); + +# Drop default value on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab ALTER COLUMN b DROP DEFAULT"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); +is($result, qq(5000), + 'data replicated to subscriber after dropping default value expression'); + +# Then we check the column default value expression on partition table. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partition ALTER COLUMN b SET DEFAULT random()"); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab_partitioned SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log( + qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab_partitioned" using subscription parameter streaming=parallel/, + $offset); + +# Drop default value on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partition ALTER COLUMN b DROP DEFAULT"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); +is($result, qq(5000), + 'data replicated to subscriber after dropping default value expression'); + +# It is not allowed that domain constraint expression contains a non-immutable +# function on the subscriber side. Check the error reported by background +# worker in this case. +# Because the column type of the partition table must be the same as its parent +# table, only test normal table here. +$node_subscriber->safe_psql( + 'postgres', qq{ +CREATE DOMAIN test_domain AS int CHECK(VALUE > random()); +ALTER TABLE test_tab ALTER COLUMN a TYPE test_domain; +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', "DELETE FROM test_tab"); + +$node_subscriber->wait_for_log( + qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab" using subscription parameter streaming=parallel/, + $offset); + +# Drop domain constraint expression on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab ALTER COLUMN a TYPE int"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); +is($result, qq(0), + 'data replicated to subscriber after dropping domain constraint expression' +); + +# It is not allowed that constraint expression contains a non-immutable function +# on the subscriber side. Check the error reported by background worker in this +# case. +# First we check the constraint expression on normal table. +$node_subscriber->safe_psql( + 'postgres', qq{ +ALTER TABLE test_tab ADD CONSTRAINT test_tab_con check (a > random()); +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log( + qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab" using subscription parameter streaming=parallel/, + $offset); + +# Drop constraint on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab DROP CONSTRAINT test_tab_con"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); +is($result, qq(5000), + 'data replicated to subscriber after dropping constraint expression'); + +# Then we check the constraint expression on partition table. +$node_subscriber->safe_psql( + 'postgres', qq{ +ALTER TABLE test_tab_partition ADD CONSTRAINT test_tab_con check (a > random()); +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', "DELETE FROM test_tab_partitioned"); + +$node_subscriber->wait_for_log( + qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab_partitioned" using subscription parameter streaming=parallel/, + $offset); + +# Drop constraint on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partition DROP CONSTRAINT test_tab_con"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); +is($result, qq(0), + 'data replicated to subscriber after dropping constraint expression'); + +# It is not allowed that foreign key on the subscriber side. Check the error +# reported by background worker in this case. +# First we check the foreign key on normal table. +$node_publisher->safe_psql('postgres', "DELETE FROM test_tab"); +$node_publisher->wait_for_catchup($appname); +$node_subscriber->safe_psql( + 'postgres', qq{ +CREATE TABLE test_tab_f (a int primary key); +ALTER TABLE test_tab ADD CONSTRAINT test_tabfk FOREIGN KEY(a) REFERENCES test_tab_f(a); +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log( + qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab" using subscription parameter streaming=parallel/, + $offset); + +# Drop the foreign key constraint on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab DROP CONSTRAINT test_tabfk"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); +is($result, qq(5000), + 'data replicated to subscriber after dropping the foreign key'); + +# Then we check the foreign key on partition table. +$node_publisher->wait_for_catchup($appname); +$node_subscriber->safe_psql( + 'postgres', qq{ +CREATE TABLE test_tab_partition_f (a int primary key); +ALTER TABLE test_tab_partition ADD CONSTRAINT test_tab_patition_fk FOREIGN KEY(a) REFERENCES test_tab_partition_f(a); +}); + +# Check the subscriber log from now on. +$offset = -s $node_subscriber->logfile; + +$node_publisher->safe_psql('postgres', + "INSERT INTO test_tab_partitioned SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)" +); + +$node_subscriber->wait_for_log( + qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab_partitioned" using subscription parameter streaming=parallel/, + $offset); + +# Drop the foreign key constraint on the subscriber, now it works. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partition DROP CONSTRAINT test_tab_patition_fk"); + +$node_publisher->wait_for_catchup($appname); + +$result = + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); +is($result, qq(5000), + 'data replicated to subscriber after dropping the foreign key'); + +$node_subscriber->stop; +$node_publisher->stop; + +done_testing(); diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 4137dc77b4..697e6a7ba3 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -1891,6 +1891,7 @@ PageXLogRecPtr PagetableEntry Pairs ParallelAppendState +ParallelApplySafety ParallelBitmapHeapState ParallelBlockTableScanDesc ParallelBlockTableScanWorker -- 2.23.0.windows.1 [application/octet-stream] v19-0004-Retry-to-apply-streaming-xact-only-in-apply-work.patch (29.0K, ../../OS3PR01MB62752B29CEC2592E18236A9A9E909@OS3PR01MB6275.jpnprd01.prod.outlook.com/5-v19-0004-Retry-to-apply-streaming-xact-only-in-apply-work.patch) download | inline diff: From 4037ce8edf0d126f7087fc8f069c0ca12f29f70e Mon Sep 17 00:00:00 2001 From: wangw <[email protected]> Date: Wed, 15 Jun 2022 11:02:08 +0800 Subject: [PATCH v19 4/4] Retry to apply streaming xact only in apply worker When the subscription parameter is set streaming=parallel, the logic tries to apply the streaming transaction using an apply background worker. If this fails the background worker exits with an error. In this case, retry applying the streaming transaction using the normal streaming=on mode. This is done to avoid getting caught in a loop of the same retry errors. A new flag field "subretry" has been introduced to catalog "pg_subscription". If the subscriber exits with an error, this flag will be set true, and whenever the transaction is applied successfully, this flag is reset false. Now, when deciding how to apply a streaming transaction, the logic can know if this transaction has previously failed or not (by checking the "subretry" field). --- doc/src/sgml/catalogs.sgml | 9 + doc/src/sgml/ref/create_subscription.sgml | 5 + src/backend/catalog/pg_subscription.c | 1 + src/backend/catalog/system_views.sql | 2 +- src/backend/commands/subscriptioncmds.c | 1 + .../replication/logical/applybgworker.c | 16 +- src/backend/replication/logical/worker.c | 165 +++++++++++++----- src/bin/pg_dump/pg_dump.c | 5 +- src/include/catalog/pg_subscription.h | 4 + .../subscription/t/032_streaming_apply.pl | 154 +++++++++------- 10 files changed, 253 insertions(+), 109 deletions(-) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index 099b3c9661..1adfed667d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -7907,6 +7907,15 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>subretry</structfield> <type>bool</type> + </para> + <para> + True if the previous apply change failed, necessitating a retry + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>subconninfo</structfield> <type>text</type> diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml index 832899570f..8ba6470e8a 100644 --- a/doc/src/sgml/ref/create_subscription.sgml +++ b/doc/src/sgml/ref/create_subscription.sgml @@ -244,6 +244,11 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl column in the relation on the subscriber-side should also be the unique column on the publisher-side; 2) there cannot be any non-immutable functions used by the subscriber-side replicated table. + When applying a streaming transaction, if either requirement is not + met, the background worker will exit with an error. + <literal>parallel</literal> mode is disregarded when retrying; + instead the transaction will be applied using <literal>on</literal> + mode. </para> </listitem> </varlistentry> diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c index 33ae3da8ae..9faeb2b1c0 100644 --- a/src/backend/catalog/pg_subscription.c +++ b/src/backend/catalog/pg_subscription.c @@ -71,6 +71,7 @@ GetSubscription(Oid subid, bool missing_ok) sub->stream = subform->substream; sub->twophasestate = subform->subtwophasestate; sub->disableonerr = subform->subdisableonerr; + sub->retry = subform->subretry; /* Get conninfo */ datum = SysCacheGetAttr(SUBSCRIPTIONOID, diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql index f369b1fc14..8284cdd00c 100644 --- a/src/backend/catalog/system_views.sql +++ b/src/backend/catalog/system_views.sql @@ -1299,7 +1299,7 @@ REVOKE ALL ON pg_replication_origin_status FROM public; REVOKE ALL ON pg_subscription FROM public; GRANT SELECT (oid, subdbid, subskiplsn, subname, subowner, subenabled, subbinary, substream, subtwophasestate, subdisableonerr, - subslotname, subsynccommit, subpublications, suborigin) + subretry, subslotname, subsynccommit, subpublications, suborigin) ON pg_subscription TO public; CREATE VIEW pg_stat_subscription_stats AS diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c index 9697128414..bd55c6ac4d 100644 --- a/src/backend/commands/subscriptioncmds.c +++ b/src/backend/commands/subscriptioncmds.c @@ -689,6 +689,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt, LOGICALREP_TWOPHASE_STATE_PENDING : LOGICALREP_TWOPHASE_STATE_DISABLED); values[Anum_pg_subscription_subdisableonerr - 1] = BoolGetDatum(opts.disableonerr); + values[Anum_pg_subscription_subretry - 1] = BoolGetDatum(false); values[Anum_pg_subscription_subconninfo - 1] = CStringGetTextDatum(conninfo); if (opts.slot_name) diff --git a/src/backend/replication/logical/applybgworker.c b/src/backend/replication/logical/applybgworker.c index 89c712f785..5d7b55e5ab 100644 --- a/src/backend/replication/logical/applybgworker.c +++ b/src/backend/replication/logical/applybgworker.c @@ -106,6 +106,18 @@ apply_bgworker_can_start(TransactionId xid) if (!XLogRecPtrIsInvalid(MySubscription->skiplsn)) return false; + /* + * Don't use apply background workers for retries, because it is possible + * that the last time we tried to apply a transaction using an apply + * background worker the checks failed (see function + * apply_bgworker_relation_check). + */ + if (MySubscription->retry) + { + elog(DEBUG1, "apply background workers are not used for retries"); + return false; + } + /* * For streaming transactions that are being applied in apply background * worker, we cannot decide whether to apply the change for a relation @@ -840,7 +852,5 @@ apply_bgworker_relation_check(LogicalRepRelMapEntry *rel) rel->remoterel.nspname, rel->remoterel.relname), errdetail("The unique column on subscriber is not the unique " "column on publisher or there is at least one " - "non-immutable function."), - errhint("Please change to use subscription parameter " - "streaming=on."))); + "non-immutable function."))); } diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c index 41793d26c2..c914cece99 100644 --- a/src/backend/replication/logical/worker.c +++ b/src/backend/replication/logical/worker.c @@ -379,6 +379,8 @@ static void clear_subscription_skip_lsn(XLogRecPtr finish_lsn); static inline void set_apply_error_context_xact(TransactionId xid, XLogRecPtr lsn); static inline void reset_apply_error_context_info(void); +static void set_subscription_retry(bool retry); + /* * Should this worker apply changes for given relation. * @@ -905,6 +907,9 @@ apply_handle_commit(StringInfo s) apply_handle_commit_internal(&commit_data); + /* Reset the retry flag. */ + set_subscription_retry(false); + /* Check the status of apply background worker if any. */ apply_bgworker_check_status(); @@ -1016,6 +1021,9 @@ apply_handle_prepare(StringInfo s) in_remote_transaction = false; + /* Reset the retry flag. */ + set_subscription_retry(false); + /* Check the status of apply background worker if any. */ apply_bgworker_check_status(); @@ -1069,6 +1077,9 @@ apply_handle_commit_prepared(StringInfo s) store_flush_position(prepare_data.end_lsn); in_remote_transaction = false; + /* Reset the retry flag. */ + set_subscription_retry(false); + /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(prepare_data.end_lsn); @@ -1124,6 +1135,9 @@ apply_handle_rollback_prepared(StringInfo s) store_flush_position(rollback_data.rollback_end_lsn); in_remote_transaction = false; + /* Reset the retry flag. */ + set_subscription_retry(false); + /* Process any tables that are being synchronized in parallel. */ process_syncing_tables(rollback_data.rollback_end_lsn); @@ -1218,6 +1232,9 @@ apply_handle_stream_prepare(StringInfo s) /* Unlink the files with serialized changes and subxact info. */ stream_cleanup_files(MyLogicalRepWorker->subid, prepare_data.xid); } + + /* Reset the retry flag. */ + set_subscription_retry(false); } in_remote_transaction = false; @@ -1646,6 +1663,9 @@ apply_handle_stream_abort(StringInfo s) */ serialize_stream_abort(xid, subxid); } + + /* Reset the retry flag. */ + set_subscription_retry(false); } reset_apply_error_context_info(); @@ -1858,6 +1878,9 @@ apply_handle_stream_commit(StringInfo s) /* Unlink the files with serialized changes and subxact info. */ stream_cleanup_files(MyLogicalRepWorker->subid, xid); } + + /* Reset the retry flag. */ + set_subscription_retry(false); } /* Check the status of apply background worker if any. */ @@ -3902,20 +3925,28 @@ start_table_sync(XLogRecPtr *origin_startpos, char **myslotname) } PG_CATCH(); { + /* + * Emit the error message, and recover from the error state to an idle + * state + */ + HOLD_INTERRUPTS(); + + EmitErrorReport(); + AbortOutOfAnyTransaction(); + FlushErrorState(); + + RESUME_INTERRUPTS(); + + /* Report the worker failed during table synchronization */ + pgstat_report_subscription_error(MySubscription->oid, false); + + /* Set the retry flag. */ + set_subscription_retry(true); + if (MySubscription->disableonerr) DisableSubscriptionAndExit(); - else - { - /* - * Report the worker failed during table synchronization. Abort - * the current transaction so that the stats message is sent in an - * idle state. - */ - AbortOutOfAnyTransaction(); - pgstat_report_subscription_error(MySubscription->oid, false); - PG_RE_THROW(); - } + proc_exit(0); } PG_END_TRY(); @@ -3940,20 +3971,27 @@ start_apply(XLogRecPtr origin_startpos) } PG_CATCH(); { + /* + * Emit the error message, and recover from the error state to an idle + * state + */ + HOLD_INTERRUPTS(); + + EmitErrorReport(); + AbortOutOfAnyTransaction(); + FlushErrorState(); + + RESUME_INTERRUPTS(); + + /* Report the worker failed while applying changes */ + pgstat_report_subscription_error(MySubscription->oid, + !am_tablesync_worker()); + + /* Set the retry flag. */ + set_subscription_retry(true); + if (MySubscription->disableonerr) DisableSubscriptionAndExit(); - else - { - /* - * Report the worker failed while applying changes. Abort the - * current transaction so that the stats message is sent in an - * idle state. - */ - AbortOutOfAnyTransaction(); - pgstat_report_subscription_error(MySubscription->oid, !am_tablesync_worker()); - - PG_RE_THROW(); - } } PG_END_TRY(); } @@ -4200,28 +4238,11 @@ ApplyWorkerMain(Datum main_arg) } /* - * After error recovery, disable the subscription in a new transaction - * and exit cleanly. + * Disable the subscription in a new transaction. */ static void DisableSubscriptionAndExit(void) { - /* - * Emit the error message, and recover from the error state to an idle - * state - */ - HOLD_INTERRUPTS(); - - EmitErrorReport(); - AbortOutOfAnyTransaction(); - FlushErrorState(); - - RESUME_INTERRUPTS(); - - /* Report the worker failed during either table synchronization or apply */ - pgstat_report_subscription_error(MyLogicalRepWorker->subid, - !am_tablesync_worker()); - /* Disable the subscription */ StartTransactionCommand(); DisableSubscription(MySubscription->oid); @@ -4231,8 +4252,6 @@ DisableSubscriptionAndExit(void) ereport(LOG, errmsg("logical replication subscription \"%s\" has been disabled due to an error", MySubscription->name)); - - proc_exit(0); } /* @@ -4467,3 +4486,63 @@ reset_apply_error_context_info(void) apply_error_callback_arg.remote_attnum = -1; set_apply_error_context_xact(InvalidTransactionId, InvalidXLogRecPtr); } + +/* + * Set subretry of pg_subscription catalog. + * + * If retry is true, subscriber is about to exit with an error. Otherwise, it + * means that the transaction was applied successfully. + */ +static void +set_subscription_retry(bool retry) +{ + Relation rel; + HeapTuple tup; + bool started_tx = false; + bool nulls[Natts_pg_subscription]; + bool replaces[Natts_pg_subscription]; + Datum values[Natts_pg_subscription]; + + if (MySubscription->retry == retry || + am_apply_bgworker()) + return; + + if (!IsTransactionState()) + { + StartTransactionCommand(); + started_tx = true; + } + + /* Look up the subscription in the catalog */ + rel = table_open(SubscriptionRelationId, RowExclusiveLock); + tup = SearchSysCacheCopy1(SUBSCRIPTIONOID, + ObjectIdGetDatum(MySubscription->oid)); + + if (!HeapTupleIsValid(tup)) + elog(ERROR, "subscription \"%s\" does not exist", MySubscription->name); + + LockSharedObject(SubscriptionRelationId, MySubscription->oid, 0, + AccessShareLock); + + /* Form a new tuple. */ + memset(values, 0, sizeof(values)); + memset(nulls, false, sizeof(nulls)); + memset(replaces, false, sizeof(replaces)); + + /* Reset subretry */ + values[Anum_pg_subscription_subretry - 1] = BoolGetDatum(retry); + replaces[Anum_pg_subscription_subretry - 1] = true; + + tup = heap_modify_tuple(tup, RelationGetDescr(rel), values, nulls, + replaces); + + /* Update the catalog. */ + CatalogTupleUpdate(rel, &tup->t_self, tup); + + /* Cleanup. */ + heap_freetuple(tup); + table_close(rel, NoLock); + + if (started_tx) + CommitTransactionCommand(); +} diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index b894cca929..e93959f17d 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -4484,8 +4484,9 @@ getSubscriptions(Archive *fout) ntups = PQntuples(res); /* - * Get subscription fields. We don't include subskiplsn in the dump as - * after restoring the dump this value may no longer be relevant. + * Get subscription fields. We don't include subskiplsn and subretry in + * the dump as after restoring the dump this value may no longer be + * relevant. */ i_tableoid = PQfnumber(res, "tableoid"); i_oid = PQfnumber(res, "oid"); diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h index 71ad03a934..9b72f4b40c 100644 --- a/src/include/catalog/pg_subscription.h +++ b/src/include/catalog/pg_subscription.h @@ -88,6 +88,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW bool subdisableonerr; /* True if a worker error should cause the * subscription to be disabled */ + bool subretry BKI_DEFAULT(f); /* True if the previous apply change failed. */ + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* Connection string to the publisher */ text subconninfo BKI_FORCE_NOT_NULL; @@ -131,6 +133,8 @@ typedef struct Subscription bool disableonerr; /* Indicates if the subscription should be * automatically disabled if a worker error * occurs */ + bool retry; /* Indicates if the previous apply change + * failed. */ char *conninfo; /* Connection string to the publisher */ char *slotname; /* Name of the replication slot */ char *synccommit; /* Synchronous commit setting for worker */ diff --git a/src/test/subscription/t/032_streaming_apply.pl b/src/test/subscription/t/032_streaming_apply.pl index 7f8bfa6745..24c519a870 100644 --- a/src/test/subscription/t/032_streaming_apply.pl +++ b/src/test/subscription/t/032_streaming_apply.pl @@ -57,8 +57,13 @@ $node_subscriber->safe_psql( $node_publisher->wait_for_catchup($appname); +# ============================================================================ # It is not allowed that the unique index on the publisher and the subscriber -# is different. Check the error reported by background worker in this case. +# is different. Check the error reported by background worker in this case. And +# after retrying in apply worker, we check if the data is replicated +# successfully. +# ============================================================================ + # First we check the unique index on normal table. $node_subscriber->safe_psql('postgres', "CREATE UNIQUE INDEX test_tab_b_idx ON test_tab (b)"); @@ -71,14 +76,15 @@ $node_subscriber->wait_for_log( qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab" using subscription parameter streaming=parallel/, $offset); -# Drop the unique index on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', "DROP INDEX test_tab_b_idx"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); my $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); -is($result, qq(5000), 'data replicated to subscriber after dropping index'); +is($result, qq(5000), 'data replicated to subscribers after retrying because of unique index'); + +# Drop the unique index on the subscriber. +$node_subscriber->safe_psql('postgres', "DROP INDEX test_tab_b_idx"); # Then we check the unique index on partition table. $node_subscriber->safe_psql('postgres', @@ -95,17 +101,20 @@ $node_subscriber->wait_for_log( qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab_partitioned" using subscription parameter streaming=parallel/, $offset); -# Drop the unique index on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', "DROP INDEX test_tab_b_partition_idx"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); -is($result, qq(5000), 'data replicated to subscriber after dropping index'); +is($result, qq(5000), 'data replicated to subscribers after retrying because of unique index'); + +# Drop the unique index on the subscriber. +$node_subscriber->safe_psql('postgres', "DROP INDEX test_tab_b_partition_idx"); # Triggers which execute non-immutable function are not allowed on the # subscriber side. Check the error reported by background worker in this case. +# And after retrying in apply worker, we check if the data is replicated +# successfully. # First we check the trigger function on normal table. $node_subscriber->safe_psql( 'postgres', qq{ @@ -129,15 +138,16 @@ $node_subscriber->wait_for_log( qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab" using subscription parameter streaming=parallel/, $offset); -# Drop the trigger on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "DROP TRIGGER insert_trig ON test_tab"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); -is($result, qq(0), 'data replicated to subscriber after dropping trigger'); +is($result, qq(0), 'data replicated to subscribers after retrying because of trigger'); + +# Drop the trigger on the subscriber. +$node_subscriber->safe_psql('postgres', + "DROP TRIGGER insert_trig ON test_tab"); # Then we check the trigger function on partition table. $node_subscriber->safe_psql( @@ -157,19 +167,24 @@ $node_subscriber->wait_for_log( qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab_partitioned" using subscription parameter streaming=parallel/, $offset); -# Drop the trigger on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "DROP TRIGGER insert_trig ON test_tab_partition"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); -is($result, qq(0), 'data replicated to subscriber after dropping trigger'); +is($result, qq(0), 'data replicated to subscribers after retrying because of trigger'); +# Drop the trigger on the subscriber. +$node_subscriber->safe_psql('postgres', + "DROP TRIGGER insert_trig ON test_tab_partition"); + +# ============================================================================ # It is not allowed that column default value expression contains a # non-immutable function on the subscriber side. Check the error reported by -# background worker in this case. +# background worker in this case. And after retrying in apply worker, we check +# if the data is replicated successfully. +# ============================================================================ + # First we check the column default value expression on normal table. $node_subscriber->safe_psql('postgres', "ALTER TABLE test_tab ALTER COLUMN b SET DEFAULT random()"); @@ -185,16 +200,17 @@ $node_subscriber->wait_for_log( qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab" using subscription parameter streaming=parallel/, $offset); -# Drop default value on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "ALTER TABLE test_tab ALTER COLUMN b DROP DEFAULT"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); is($result, qq(5000), - 'data replicated to subscriber after dropping default value expression'); + 'data replicated to subscribers after retrying because of column default value'); + +# Drop default value on the subscriber. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab ALTER COLUMN b DROP DEFAULT"); # Then we check the column default value expression on partition table. $node_subscriber->safe_psql('postgres', @@ -211,20 +227,25 @@ $node_subscriber->wait_for_log( qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab_partitioned" using subscription parameter streaming=parallel/, $offset); -# Drop default value on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "ALTER TABLE test_tab_partition ALTER COLUMN b DROP DEFAULT"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); is($result, qq(5000), - 'data replicated to subscriber after dropping default value expression'); + 'data replicated to subscribers after retrying because of column default value'); + +# Drop default value on the subscriber. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partition ALTER COLUMN b DROP DEFAULT"); +# ============================================================================ # It is not allowed that domain constraint expression contains a non-immutable # function on the subscriber side. Check the error reported by background -# worker in this case. +# worker in this case. And after retrying in apply worker, we check if the data +# is replicated successfully. +# ============================================================================ + # Because the column type of the partition table must be the same as its parent # table, only test normal table here. $node_subscriber->safe_psql( @@ -242,21 +263,26 @@ $node_subscriber->wait_for_log( qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab" using subscription parameter streaming=parallel/, $offset); -# Drop domain constraint expression on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "ALTER TABLE test_tab ALTER COLUMN a TYPE int"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); is($result, qq(0), - 'data replicated to subscriber after dropping domain constraint expression' + 'data replicated to subscribers after retrying because of domain' ); -# It is not allowed that constraint expression contains a non-immutable function -# on the subscriber side. Check the error reported by background worker in this -# case. +# Drop domain constraint expression on the subscriber. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab ALTER COLUMN a TYPE int"); + +# ============================================================================ +# It is not allowed that constraint expression contains a non-immutable +# function on the subscriber side. Check the error reported by background +# worker in this case. And after retrying in apply worker, we check if the data +# is replicated successfully. +# ============================================================================ + # First we check the constraint expression on normal table. $node_subscriber->safe_psql( 'postgres', qq{ @@ -274,16 +300,17 @@ $node_subscriber->wait_for_log( qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab" using subscription parameter streaming=parallel/, $offset); -# Drop constraint on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "ALTER TABLE test_tab DROP CONSTRAINT test_tab_con"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); is($result, qq(5000), - 'data replicated to subscriber after dropping constraint expression'); + 'data replicated to subscribers after retrying because of constraint'); + +# Drop constraint on the subscriber. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab DROP CONSTRAINT test_tab_con"); # Then we check the constraint expression on partition table. $node_subscriber->safe_psql( @@ -300,19 +327,24 @@ $node_subscriber->wait_for_log( qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab_partitioned" using subscription parameter streaming=parallel/, $offset); -# Drop constraint on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "ALTER TABLE test_tab_partition DROP CONSTRAINT test_tab_con"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); is($result, qq(0), - 'data replicated to subscriber after dropping constraint expression'); + 'data replicated to subscribers after retrying because of constraint'); +# Drop constraint on the subscriber. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partition DROP CONSTRAINT test_tab_con"); + +# ============================================================================ # It is not allowed that foreign key on the subscriber side. Check the error -# reported by background worker in this case. +# reported by background worker in this case. And after retrying in apply +# worker, we check if the data is replicated successfully. +# ============================================================================ + # First we check the foreign key on normal table. $node_publisher->safe_psql('postgres', "DELETE FROM test_tab"); $node_publisher->wait_for_catchup($appname); @@ -333,16 +365,17 @@ $node_subscriber->wait_for_log( qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab" using subscription parameter streaming=parallel/, $offset); -# Drop the foreign key constraint on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "ALTER TABLE test_tab DROP CONSTRAINT test_tabfk"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab"); is($result, qq(5000), - 'data replicated to subscriber after dropping the foreign key'); + 'data replicated to subscribers after retrying because of foreign key'); + +# Drop the foreign key constraint on the subscriber. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab DROP CONSTRAINT test_tabfk"); # Then we check the foreign key on partition table. $node_publisher->wait_for_catchup($appname); @@ -363,16 +396,17 @@ $node_subscriber->wait_for_log( qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab_partitioned" using subscription parameter streaming=parallel/, $offset); -# Drop the foreign key constraint on the subscriber, now it works. -$node_subscriber->safe_psql('postgres', - "ALTER TABLE test_tab_partition DROP CONSTRAINT test_tab_patition_fk"); - +# Wait for this streaming transaction to be applied in the apply worker. $node_publisher->wait_for_catchup($appname); $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned"); is($result, qq(5000), - 'data replicated to subscriber after dropping the foreign key'); + 'data replicated to subscribers after retrying because of foreign key'); + +# Drop the foreign key constraint on the subscriber. +$node_subscriber->safe_psql('postgres', + "ALTER TABLE test_tab_partition DROP CONSTRAINT test_tab_patition_fk"); $node_subscriber->stop; $node_publisher->stop; -- 2.23.0.windows.1 ^ permalink raw reply [nested|flat] 66+ messages in thread
* Re: Perform streaming logical transactions by background workers and parallel apply @ 2022-07-25 13:50 Amit Kapila <[email protected]> parent: [email protected] <[email protected]> 5 siblings, 0 replies; 66+ messages in thread From: Amit Kapila @ 2022-07-25 13:50 UTC (permalink / raw) To: [email protected] <[email protected]>; +Cc: Peter Smith <[email protected]>; Masahiko Sawada <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; PostgreSQL Hackers <[email protected]> On Fri, Jul 22, 2022 at 8:26 AM [email protected] <[email protected]> wrote: > > On Tues, Jul 19, 2022 at 10:29 AM I wrote: > > Attach the news patches. > > Not able to apply patches cleanly because the change in HEAD (366283961a). > Therefore, I rebased the patch based on the changes in HEAD. > > Attach the new patches. > Few comments on 0001: ====================== 1. - <structfield>substream</structfield> <type>bool</type> + <structfield>substream</structfield> <type>char</type> </para> <para> - If true, the subscription will allow streaming of in-progress - transactions + Controls how to handle the streaming of in-progress transactions: + <literal>f</literal> = disallow streaming of in-progress transactions, + <literal>t</literal> = spill the changes of in-progress transactions to + disk and apply at once after the transaction is committed on the + publisher, + <literal>p</literal> = apply changes directly using a background worker Shouldn't the description of 'p' be something like: apply changes directly using a background worker, if available, otherwise, it behaves the same as 't' 2. Note that if an error happens when + applying changes in a background worker, the finish LSN of the + remote transaction might not be reported in the server log. Is there any case where finish LSN can be reported when applying via background worker, if not, then we should use 'won't' instead of 'might not'? 3. +#define PG_LOGICAL_APPLY_SHM_MAGIC 0x79fb2447 // TODO Consider change It is better to change this as the same magic number is used by PG_TEST_SHM_MQ_MAGIC 4. + /* Ignore statistics fields that have been updated. */ + s.cursor += IGNORE_SIZE_IN_MESSAGE; Can we change the comment to: "Ignore statistics fields that have been updated by the main apply worker."? Will it be better to name the define as "SIZE_STATS_MESSAGE"? 5. +/* Apply Background Worker main loop */ +static void +LogicalApplyBgwLoop(shm_mq_handle *mqh, volatile ApplyBgworkerShared *shared) { ... ... + apply_dispatch(&s); + + if (ConfigReloadPending) + { + ConfigReloadPending = false; + ProcessConfigFile(PGC_SIGHUP); + } + + MemoryContextSwitchTo(oldctx); + MemoryContextReset(ApplyMessageContext); We should not process the config file under ApplyMessageContext. You should switch context before processing the config file. See other similar usages in the code. 6. +/* Apply Background Worker main loop */ +static void +LogicalApplyBgwLoop(shm_mq_handle *mqh, volatile ApplyBgworkerShared *shared) { ... ... + MemoryContextSwitchTo(oldctx); + MemoryContextReset(ApplyMessageContext); + } + + MemoryContextSwitchTo(TopMemoryContext); + MemoryContextReset(ApplyContext); ... } I don't see the need to reset ApplyContext here as we don't do anything in that context here. -- With Regards, Amit Kapila. ^ permalink raw reply [nested|flat] 66+ messages in thread
* Re: Perform streaming logical transactions by background workers and parallel apply @ 2022-07-26 09:00 Dilip Kumar <[email protected]> parent: [email protected] <[email protected]> 5 siblings, 2 replies; 66+ messages in thread From: Dilip Kumar @ 2022-07-26 09:00 UTC (permalink / raw) To: [email protected] <[email protected]>; +Cc: Peter Smith <[email protected]>; Amit Kapila <[email protected]>; Masahiko Sawada <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; PostgreSQL Hackers <[email protected]> On Fri, Jul 22, 2022 at 8:27 AM [email protected] <[email protected]> wrote: > > On Tues, Jul 19, 2022 at 10:29 AM I wrote: > > Attach the news patches. > > Not able to apply patches cleanly because the change in HEAD (366283961a). > Therefore, I rebased the patch based on the changes in HEAD. > > Attach the new patches. + /* Check the foreign keys. */ + fkeys = RelationGetFKeyList(entry->localrel); + if (fkeys) + entry->parallel_apply = PARALLEL_APPLY_UNSAFE; So if there is a foreign key on any of the tables which are parts of a subscription then we do not allow changes for that subscription to be applied in parallel? I think this is a big limitation because having foreign key on the table is very normal right? I agree that if we allow them then there could be failure due to out of order apply right? but IMHO we should not put the restriction instead let it fail if there is ever such conflict. Because if there is a conflict the transaction will be sent again. Do we see that there could be wrong or inconsistent results if we allow such things to be executed in parallel. If not then IMHO just to avoid some corner case failure we are restricting very normal cases. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 66+ messages in thread
* Re: Perform streaming logical transactions by background workers and parallel apply @ 2022-07-26 09:33 Dilip Kumar <[email protected]> parent: Dilip Kumar <[email protected]> 1 sibling, 1 reply; 66+ messages in thread From: Dilip Kumar @ 2022-07-26 09:33 UTC (permalink / raw) To: [email protected] <[email protected]>; +Cc: Peter Smith <[email protected]>; Amit Kapila <[email protected]>; Masahiko Sawada <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; PostgreSQL Hackers <[email protected]> On Tue, Jul 26, 2022 at 2:30 PM Dilip Kumar <[email protected]> wrote: > > On Fri, Jul 22, 2022 at 8:27 AM [email protected] > <[email protected]> wrote: > > > > On Tues, Jul 19, 2022 at 10:29 AM I wrote: > > > Attach the news patches. > > > > Not able to apply patches cleanly because the change in HEAD (366283961a). > > Therefore, I rebased the patch based on the changes in HEAD. > > > > Attach the new patches. > > + /* Check the foreign keys. */ > + fkeys = RelationGetFKeyList(entry->localrel); > + if (fkeys) > + entry->parallel_apply = PARALLEL_APPLY_UNSAFE; > > So if there is a foreign key on any of the tables which are parts of a > subscription then we do not allow changes for that subscription to be > applied in parallel? I think this is a big limitation because having > foreign key on the table is very normal right? I agree that if we > allow them then there could be failure due to out of order apply > right? but IMHO we should not put the restriction instead let it fail > if there is ever such conflict. Because if there is a conflict the > transaction will be sent again. Do we see that there could be wrong > or inconsistent results if we allow such things to be executed in > parallel. If not then IMHO just to avoid some corner case failure we > are restricting very normal cases. some more comments.. 1. + /* + * If we have found a free worker or if we are already applying this + * transaction in an apply background worker, then we pass the data to + * that worker. + */ + if (first_segment) + apply_bgworker_send_data(stream_apply_worker, s->len, s->data); Comment says that if we have found a free worker or we are already applying in the worker then pass the changes to the worker but actually as per the code here we are only passing in case of first_segment? I think what you are trying to say is that if it is first segment then send the 2. + /* + * This is the main apply worker. Check if there is any free apply + * background worker we can use to process this transaction. + */ + if (first_segment) + stream_apply_worker = apply_bgworker_start(stream_xid); + else + stream_apply_worker = apply_bgworker_find(stream_xid); So currently, whenever we get a new streamed transaction we try to start a new background worker for that. Why do we need to start/close the background apply worker every time we get a new streamed transaction. I mean we can keep the worker in the pool for time being and if there is a new transaction looking for a worker then we can find from that. Starting a worker is costly operation and since we are using parallelism for this mean we are expecting that there would be frequent streamed transaction needing parallel apply worker so why not to let it wait for a certain amount of time so that if load is low it will anyway stop and if the load is high it will be reused for next streamed transaction. 3. Why are we restricting parallel apply workers only for the streamed transactions, because streaming depends upon the size of the logical decoding work mem so making steaming and parallel apply tightly coupled seems too restrictive to me. Do we see some obvious problems in applying other transactions in parallel? -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 66+ messages in thread
* Re: Perform streaming logical transactions by background workers and parallel apply @ 2022-07-26 09:56 Peter Smith <[email protected]> parent: [email protected] <[email protected]> 5 siblings, 0 replies; 66+ messages in thread From: Peter Smith @ 2022-07-26 09:56 UTC (permalink / raw) To: [email protected] <[email protected]>; +Cc: Amit Kapila <[email protected]>; Masahiko Sawada <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; PostgreSQL Hackers <[email protected]> Here are some review comment for patch v19-0001: ====== 1.1 Missing docs for protocol version Since you bumped the logical replication protocol version for this patch, now there is some missing documentation to describe this new protocol version. e.g. See here [1] ====== 1.2 doc/src/sgml/config.sgml + <para> + Maximum number of apply background workers per subscription. This + parameter controls the amount of parallelism of the streaming of + in-progress transactions when subscription parameter + <literal>streaming = parallel</literal>. + </para> SUGGESTION Maximum number of apply background workers per subscription. This parameter controls the amount of parallelism for streaming of in-progress transactions with subscription parameter <literal>streaming = parallel</literal>. ====== 1.3 src/sgml/protocol.sgml @@ -6809,6 +6809,25 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;" </listitem> </varlistentry> + <varlistentry> + <term>Int64 (XLogRecPtr)</term> + <listitem> + <para> + The LSN of the abort. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Int64 (TimestampTz)</term> + <listitem> + <para> + Abort timestamp of the transaction. The value is in number + of microseconds since PostgreSQL epoch (2000-01-01). + </para> + </listitem> + </varlistentry> There are missing notes on these new fields. They both should says something like "This field is available since protocol version 4." (See similar examples on the same docs page) ====== 1.4 src/backend/replication/logical/applybgworker.c - apply_bgworker_start Previously [1] I wrote: > The Assert that the entries in the free-list are FINISHED seems like > unnecessary checking. IIUC, code is already doing the Assert that > entries are FINISHED before allowing them into the free-list in the > first place. IMO this Assert just causes unnecessary doubts, but if you really want to keep it then I think it belongs logically *above* the list_delete_last. ~~~ 1.5 src/backend/replication/logical/applybgworker.c - apply_bgworker_start + server_version = walrcv_server_version(LogRepWorkerWalRcvConn); + wstate->shared->server_version = + server_version >= 160000 ? LOGICALREP_PROTO_STREAM_PARALLEL_VERSION_NUM : + server_version >= 150000 ? LOGICALREP_PROTO_TWOPHASE_VERSION_NUM : + server_version >= 140000 ? LOGICALREP_PROTO_STREAM_VERSION_NUM : + LOGICALREP_PROTO_VERSION_NUM; It makes no sense to assign a protocol version to a server_version. Perhaps it is just a simple matter of a poorly named struct member. e.g Maybe everything is OK if it said something like wstate->shared->proto_version. ~~~ 1.6 src/backend/replication/logical/applybgworker.c - LogicalApplyBgwLoop +/* Apply Background Worker main loop */ +static void +LogicalApplyBgwLoop(shm_mq_handle *mqh, volatile ApplyBgworkerShared *shared) 'shared' seems a very vague param name. Maybe can be 'bgw_shared' or 'parallel_shared' or something better? ~~~ 1.7 src/backend/replication/logical/applybgworker.c - ApplyBgworkerMain +/* + * Apply Background Worker entry point + */ +void +ApplyBgworkerMain(Datum main_arg) +{ + volatile ApplyBgworkerShared *shared; 'shared' seems a very vague var name. Maybe can be 'bgw_shared' or 'parallel_shared' or something better? ~~~ 1.8 src/backend/replication/logical/applybgworker.c - apply_bgworker_setup_dsm +static void +apply_bgworker_setup_dsm(ApplyBgworkerState *wstate) +{ + shm_toc_estimator e; + Size segsize; + dsm_segment *seg; + shm_toc *toc; + ApplyBgworkerShared *shared; + shm_mq *mq; 'shared' seems a very vague var name. Maybe can be 'bgw_shared' or 'parallel_shared' or something better? ~~~ 1.9 src/backend/replication/logical/applybgworker.c - apply_bgworker_setup_dsm + server_version = walrcv_server_version(LogRepWorkerWalRcvConn); + shared->server_version = + server_version >= 160000 ? LOGICALREP_PROTO_STREAM_PARALLEL_VERSION_NUM : + server_version >= 150000 ? LOGICALREP_PROTO_TWOPHASE_VERSION_NUM : + server_version >= 140000 ? LOGICALREP_PROTO_STREAM_VERSION_NUM : + LOGICALREP_PROTO_VERSION_NUM; Same as earlier review comment #1.5 ====== 1.10 src/backend/replication/logical/worker.c @@ -22,8 +22,28 @@ * STREAMED TRANSACTIONS * --------------------- * Streamed transactions (large transactions exceeding a memory limit on the - * upstream) are not applied immediately, but instead, the data is written - * to temporary files and then applied at once when the final commit arrives. + * upstream) are applied using one of two approaches. + * + * 1) Separate background workers "two approaches." --> "two approaches:" ~~~ 1.11 src/backend/replication/logical/worker.c - apply_handle_stream_abort + /* Check whether the publisher sends abort_lsn and abort_time. */ + if (am_apply_bgworker()) + read_abort_lsn = MyParallelShared->server_version >= + LOGICALREP_PROTO_STREAM_PARALLEL_VERSION_NUM; IMO makes no sense to compare a server version with a protocol version. Same as review comment #1.5 ====== 1.12 src/include/replication/worker_internal.h +typedef struct ApplyBgworkerShared +{ + slock_t mutex; + + /* Status of apply background worker. */ + ApplyBgworkerStatus status; + + /* server version of publisher. */ + uint32 server_version; + + TransactionId stream_xid; + uint32 n; /* id of apply background worker */ +} ApplyBgworkerShared; AFAICT you only ever used 'server_version' for storing the *protocol* version, so really this member should be called something like 'proto_version'. Please see earlier review comment #1.5 and others. ------ [1] https://www.postgresql.org/message-id/CAHut%2BPvN7fwtUE%3DbidzrsOUXSt%2BJpnkJztZ-Jn5t86moofaZ6g%40ma... [2] https://www.postgresql.org/docs/devel/protocol-logical-replication.html Kind Reagrds, Peter Smith. Fujitsu Australia. ^ permalink raw reply [nested|flat] 66+ messages in thread
* Re: Perform streaming logical transactions by background workers and parallel apply @ 2022-07-27 03:37 Peter Smith <[email protected]> parent: [email protected] <[email protected]> 5 siblings, 0 replies; 66+ messages in thread From: Peter Smith @ 2022-07-27 03:37 UTC (permalink / raw) To: [email protected] <[email protected]>; +Cc: Amit Kapila <[email protected]>; Masahiko Sawada <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; PostgreSQL Hackers <[email protected]> Here are some review comments for patch v19-0003: ====== 3.1 doc/src/sgml/ref/create_subscription.sgml @@ -240,6 +240,10 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl transaction is committed. Note that if an error happens when applying changes in a background worker, the finish LSN of the remote transaction might not be reported in the server log. + <literal>parallel</literal> mode has two requirements: 1) the unique + column in the relation on the subscriber-side should also be the + unique column on the publisher-side; 2) there cannot be any + non-immutable functions used by the subscriber-side replicated table. </para> 3.1a. It looked a bit strange starting the sentence with the enum "<literal>parallel</literal> mode". Maybe reword it something like: "This mode has two requirements: ..." or "There are two requirements for using <literal>parallel</literal> mode: ..." 3.1b. Point 1) says "relation", but point 2) says "table". I think the consistent term should be used. ====== 3.2 <general> For consistency, please search all this patch and replace every: "... applied by an apply background worker" -> "... applied using an apply background worker" And also search/replace every: "... in the apply background worker: -> "... using an apply background worker" ====== 3.3 .../replication/logical/applybgworker.c @@ -800,3 +800,47 @@ apply_bgworker_subxact_info_add(TransactionId current_xid) MemoryContextSwitchTo(oldctx); } } + +/* + * Check if changes on this relation can be applied by an apply background + * worker. + * + * Although the commit order is maintained only allowing one process to commit + * at a time, the access order to the relation has changed. This could cause + * unexpected problems if the unique column on the replicated table is + * inconsistent with the publisher-side or contains non-immutable functions + * when applying transactions in the apply background worker. + */ +void +apply_bgworker_relation_check(LogicalRepRelMapEntry *rel) "only allowing" -> "by only allowing" (I think you mean this, right?) ~~~ 3.4 + /* + * Return if changes on this relation can be applied by an apply background + * worker. + */ + if (rel->parallel_apply == PARALLEL_APPLY_SAFE) + return; + + /* We are in error mode and should give user correct error. */ + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("cannot replicate target relation \"%s.%s\" using " + "subscription parameter streaming=parallel", + rel->remoterel.nspname, rel->remoterel.relname), + errdetail("The unique column on subscriber is not the unique " + "column on publisher or there is at least one " + "non-immutable function."), + errhint("Please change to use subscription parameter " + "streaming=on."))); 3.4a. Of course, the code should give the user the "correct error" if there is an error (!), but having a comment explicitly saying so does not serve any purpose. 3.4b. The logic might be simplified if it was written differently like: + if (rel->parallel_apply != PARALLEL_APPLY_SAFE) + ereport(ERROR, ... ====== 3.5 src/backend/replication/logical/proto.c @@ -40,6 +41,68 @@ static void logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple); static void logicalrep_write_namespace(StringInfo out, Oid nspid); static const char *logicalrep_read_namespace(StringInfo in); +static Bitmapset *RelationGetUniqueKeyBitmap(Relation rel); + +/* + * RelationGetUniqueKeyBitmap -- get a bitmap of unique attribute numbers + * + * This is similar to RelationGetIdentityKeyBitmap(), but returns a bitmap of + * index attribute numbers for all unique indexes. + */ +static Bitmapset * +RelationGetUniqueKeyBitmap(Relation rel) Why is the forward declaration needed when the static function immediately follows it? ====== 3.6 src/backend/replication/logical/relation.c - logicalrep_relmap_reset_parallel_cb @@ -91,6 +98,26 @@ logicalrep_relmap_invalidate_cb(Datum arg, Oid reloid) } } +/* + * Relcache invalidation callback to reset parallel flag. + */ +static void +logicalrep_relmap_reset_parallel_cb(Datum arg, int cacheid, uint32 hashvalue) "reset parallel flag" -> "reset parallel_apply flag" ~~~ 3.7 src/backend/replication/logical/relation.c - logicalrep_rel_mark_parallel_apply + * There are two requirements for applying changes in an apply background + * worker: 1) The unique column in the relation on the subscriber-side should + * also be the unique column on the publisher-side; 2) There cannot be any + * non-immutable functions used by the subscriber-side. This comment should exactly match the help text. See review comment #3.1 ~~~ 3.8 + /* Initialize the flag. */ + entry->parallel_apply = PARALLEL_APPLY_SAFE; I previously suggested [1] (#3.6b) to move this. Consider, that you cannot logically flag the entry as "safe" until you are certain that it is safe. And you cannot be sure of that until you've passed all the checks this function is doing. Therefore IMO the assignment to PARALLEL_APPLY_SAFE should be the last line of the function. ~~~ 3.9 + /* + * Then, check if there is any non-immutable function used by the local + * table. Look for functions in the following places: + * a. trigger functions; + * b. Column default value expressions and domain constraints; + * c. Constraint expressions; + * d. Foreign keys. + */ "used by the local table" -> "used by the subscriber-side relation" (reworded so that it is consistent with the First comment) ~~~ 3.10 I previously suggested [1] (#3.7) to use goto in this function to avoid the excessive number of returns. IMO there is nothing inherently evil about gotos, so long as they are used with care - sometimes they are the best option. Anyway, I attached some BEFORE/AFTER example code to this post - others can judge which way is preferable. ====== 3.11 src/backend/utils/cache/typcache.c - GetDomainConstraints @@ -2540,6 +2540,23 @@ compare_values_of_enum(TypeCacheEntry *tcache, Oid arg1, Oid arg2) return 0; } +/* + * GetDomainConstraints --- get DomainConstraintState list of specified domain type + */ +List * +GetDomainConstraints(Oid type_id) +{ + TypeCacheEntry *typentry; + List *constraints = NIL; + + typentry = lookup_type_cache(type_id, TYPECACHE_DOMAIN_CONSTR_INFO); + + if(typentry->domainData != NULL) + constraints = typentry->domainData->constraints; + + return constraints; +} This function can be simplified (if you want). e.g. List * GetDomainConstraints(Oid type_id) { TypeCacheEntry *typentry; typentry = lookup_type_cache(type_id, TYPECACHE_DOMAIN_CONSTR_INFO); return typentry->domainData ? typentry->domainData->constraints : NIL; } ====== 3.12 src/include/replication/logicalrelation.h @@ -15,6 +15,19 @@ #include "access/attmap.h" #include "replication/logicalproto.h" +/* + * States to determine if changes on one relation can be applied using an + * apply background worker. + */ +typedef enum ParalleApplySafety +{ + PARALLEL_APPLY_UNKNOWN = 0, /* unknown */ + PARALLEL_APPLY_SAFE, /* Can apply changes in an apply background + worker */ + PARALLEL_APPLY_UNSAFE /* Can not apply changes in an apply background + worker */ +} ParalleApplySafety; + 3.12a Typo in enum and typedef names: "ParalleApplySafety" -> "ParallelApplySafety" 3.12b I think the values are quite self-explanatory now. Commenting on each of them separately is not really adding anything useful. 3.12c. New enum missing from typedefs.list? ====== 3.13 typdefs.list Should include the new typedef. See comment #3.12c. ------ [1] https://www.postgresql.org/message-id/OS3PR01MB62758A6AAED27B3A848CEB7A9E8F9%40OS3PR01MB6275.jpnprd0... Kind Regards, Peter Smith. Fujitsu Australia /* * Check if changes on one relation can be applied by an apply background * worker and assign the 'parallel_apply' flag. * * There are two requirements for applying changes in an apply background * worker: 1) The unique column in the relation on the subscriber-side should * also be the unique column on the publisher-side; 2) There cannot be any * non-immutable functions used by the subscriber-side. * * We just mark the relation entry as 'PARALLEL_APPLY_UNSAFE' here if changes * on one relation can not be applied by an apply background worker and leave * it to apply_bgworker_relation_check() to throw the actual error if needed. */ static void logicalrep_rel_mark_parallel_apply(LogicalRepRelMapEntry *entry) { Bitmapset *ukey; int i; TupleDesc tupdesc; int attnum; List *fkeys = NIL; /* Fast path if 'parallel_apply' flag is already known. */ if (entry->parallel_apply != PARALLEL_APPLY_UNKNOWN) return; /* * First, check if the unique column in the relation on the subscriber-side * is also the unique column on the publisher-side. */ ukey = RelationGetIndexAttrBitmap(entry->localrel, INDEX_ATTR_BITMAP_KEY); if (ukey) { i = -1; while ((i = bms_next_member(ukey, i)) >= 0) { attnum = AttrNumberGetAttrOffset(i + FirstLowInvalidHeapAttributeNumber); if (entry->attrmap->attnums[attnum] < 0 || !bms_is_member(entry->attrmap->attnums[attnum], entry->remoterel.attunique)) goto parallel_apply_unsafe; } bms_free(ukey); } /* * Then, check if there is any non-immutable function used by the local * table. Look for functions in the following places: * a. trigger functions; * b. Column default value expressions and domain constraints; * c. Constraint expressions; * d. Foreign keys. */ /* Check the trigger functions. */ if (entry->localrel->trigdesc != NULL) { for (i = 0; i < entry->localrel->trigdesc->numtriggers; i++) { Trigger *trig = entry->localrel->trigdesc->triggers + i; if (trig->tgenabled != TRIGGER_FIRES_ALWAYS && trig->tgenabled != TRIGGER_FIRES_ON_REPLICA) continue; if (func_volatile(trig->tgfoid) != PROVOLATILE_IMMUTABLE) goto parallel_apply_unsafe; } } /* Check the columns. */ tupdesc = RelationGetDescr(entry->localrel); for (attnum = 0; attnum < tupdesc->natts; attnum++) { Form_pg_attribute att = TupleDescAttr(tupdesc, attnum); /* We don't need info for dropped or generated attributes */ if (att->attisdropped || att->attgenerated) continue; /* * We don't need to check columns that only exist on the * subscriber */ if (entry->attrmap->attnums[attnum] < 0) continue; if (att->atthasdef) { Node *defaultexpr; defaultexpr = build_column_default(entry->localrel, attnum + 1); if (contain_mutable_functions(defaultexpr)) goto parallel_apply_unsafe; } /* * If the column is of a DOMAIN type, determine whether * that domain has any CHECK expressions that are not * immutable. */ if (get_typtype(att->atttypid) == TYPTYPE_DOMAIN) { List *domain_constraints; ListCell *lc; domain_constraints = GetDomainConstraints(att->atttypid); foreach(lc, domain_constraints) { DomainConstraintState *con = (DomainConstraintState *) lfirst(lc); if (con->check_expr && contain_mutable_functions((Node *) con->check_expr)) goto parallel_apply_unsafe; } } } /* Check the constraints. */ if (tupdesc->constr) { ConstrCheck *check = tupdesc->constr->check; /* * Determine if there are any CHECK constraints which * contains non-immutable function. */ for (i = 0; i < tupdesc->constr->num_check; i++) { Expr *check_expr = stringToNode(check[i].ccbin); if (contain_mutable_functions((Node *) check_expr)) goto parallel_apply_unsafe; } } /* Check the foreign keys. */ fkeys = RelationGetFKeyList(entry->localrel); if (fkeys) goto parallel_apply_unsafe; parallel_apply_safe: entry->parallel_apply = PARALLEL_APPLY_SAFE; return; parallel_apply_unsafe: entry->parallel_apply = PARALLEL_APPLY_UNSAFE; return; } /* * Check if changes on one relation can be applied by an apply background * worker and assign the 'parallel_apply' flag. * * There are two requirements for applying changes in an apply background * worker: 1) The unique column in the relation on the subscriber-side should * also be the unique column on the publisher-side; 2) There cannot be any * non-immutable functions used by the subscriber-side. * * We just mark the relation entry as 'PARALLEL_APPLY_UNSAFE' here if changes * on one relation can not be applied by an apply background worker and leave * it to apply_bgworker_relation_check() to throw the actual error if needed. */ static void logicalrep_rel_mark_parallel_apply(LogicalRepRelMapEntry *entry) { Bitmapset *ukey; int i; TupleDesc tupdesc; int attnum; List *fkeys = NIL; /* Fast path if 'parallel_apply' flag is already known. */ if (entry->parallel_apply != PARALLEL_APPLY_UNKNOWN) return; /* Initialize the flag. */ entry->parallel_apply = PARALLEL_APPLY_SAFE; /* * First, check if the unique column in the relation on the subscriber-side * is also the unique column on the publisher-side. */ ukey = RelationGetIndexAttrBitmap(entry->localrel, INDEX_ATTR_BITMAP_KEY); if (ukey) { i = -1; while ((i = bms_next_member(ukey, i)) >= 0) { attnum = AttrNumberGetAttrOffset(i + FirstLowInvalidHeapAttributeNumber); if (entry->attrmap->attnums[attnum] < 0 || !bms_is_member(entry->attrmap->attnums[attnum], entry->remoterel.attunique)) { entry->parallel_apply = PARALLEL_APPLY_UNSAFE; return; } } bms_free(ukey); } /* * Then, check if there is any non-immutable function used by the local * table. Look for functions in the following places: * a. trigger functions; * b. Column default value expressions and domain constraints; * c. Constraint expressions; * d. Foreign keys. */ /* Check the trigger functions. */ if (entry->localrel->trigdesc != NULL) { for (i = 0; i < entry->localrel->trigdesc->numtriggers; i++) { Trigger *trig = entry->localrel->trigdesc->triggers + i; if (trig->tgenabled != TRIGGER_FIRES_ALWAYS && trig->tgenabled != TRIGGER_FIRES_ON_REPLICA) continue; if (func_volatile(trig->tgfoid) != PROVOLATILE_IMMUTABLE) { entry->parallel_apply = PARALLEL_APPLY_UNSAFE; return; } } } /* Check the columns. */ tupdesc = RelationGetDescr(entry->localrel); for (attnum = 0; attnum < tupdesc->natts; attnum++) { Form_pg_attribute att = TupleDescAttr(tupdesc, attnum); /* We don't need info for dropped or generated attributes */ if (att->attisdropped || att->attgenerated) continue; /* * We don't need to check columns that only exist on the * subscriber */ if (entry->attrmap->attnums[attnum] < 0) continue; if (att->atthasdef) { Node *defaultexpr; defaultexpr = build_column_default(entry->localrel, attnum + 1); if (contain_mutable_functions(defaultexpr)) { entry->parallel_apply = PARALLEL_APPLY_UNSAFE; return; } } /* * If the column is of a DOMAIN type, determine whether * that domain has any CHECK expressions that are not * immutable. */ if (get_typtype(att->atttypid) == TYPTYPE_DOMAIN) { List *domain_constraints; ListCell *lc; domain_constraints = GetDomainConstraints(att->atttypid); foreach(lc, domain_constraints) { DomainConstraintState *con = (DomainConstraintState *) lfirst(lc); if (con->check_expr && contain_mutable_functions((Node *) con->check_expr)) { entry->parallel_apply = PARALLEL_APPLY_UNSAFE; return; } } } } /* Check the constraints. */ if (tupdesc->constr) { ConstrCheck *check = tupdesc->constr->check; /* * Determine if there are any CHECK constraints which * contains non-immutable function. */ for (i = 0; i < tupdesc->constr->num_check; i++) { Expr *check_expr = stringToNode(check[i].ccbin); if (contain_mutable_functions((Node *) check_expr)) { entry->parallel_apply = PARALLEL_APPLY_UNSAFE; return; } } } /* Check the foreign keys. */ fkeys = RelationGetFKeyList(entry->localrel); if (fkeys) entry->parallel_apply = PARALLEL_APPLY_UNSAFE; } Attachments: [text/plain] logicalrep_rel_mark_parallel_apply-with-goto.txt (4.2K, ../../CAHut+Pv9cKurDQHtk-ygYp45-8LYdE=4sMZY-8UmbeDTGgECVg@mail.gmail.com/2-logicalrep_rel_mark_parallel_apply-with-goto.txt) download | inline: /* * Check if changes on one relation can be applied by an apply background * worker and assign the 'parallel_apply' flag. * * There are two requirements for applying changes in an apply background * worker: 1) The unique column in the relation on the subscriber-side should * also be the unique column on the publisher-side; 2) There cannot be any * non-immutable functions used by the subscriber-side. * * We just mark the relation entry as 'PARALLEL_APPLY_UNSAFE' here if changes * on one relation can not be applied by an apply background worker and leave * it to apply_bgworker_relation_check() to throw the actual error if needed. */ static void logicalrep_rel_mark_parallel_apply(LogicalRepRelMapEntry *entry) { Bitmapset *ukey; int i; TupleDesc tupdesc; int attnum; List *fkeys = NIL; /* Fast path if 'parallel_apply' flag is already known. */ if (entry->parallel_apply != PARALLEL_APPLY_UNKNOWN) return; /* * First, check if the unique column in the relation on the subscriber-side * is also the unique column on the publisher-side. */ ukey = RelationGetIndexAttrBitmap(entry->localrel, INDEX_ATTR_BITMAP_KEY); if (ukey) { i = -1; while ((i = bms_next_member(ukey, i)) >= 0) { attnum = AttrNumberGetAttrOffset(i + FirstLowInvalidHeapAttributeNumber); if (entry->attrmap->attnums[attnum] < 0 || !bms_is_member(entry->attrmap->attnums[attnum], entry->remoterel.attunique)) goto parallel_apply_unsafe; } bms_free(ukey); } /* * Then, check if there is any non-immutable function used by the local * table. Look for functions in the following places: * a. trigger functions; * b. Column default value expressions and domain constraints; * c. Constraint expressions; * d. Foreign keys. */ /* Check the trigger functions. */ if (entry->localrel->trigdesc != NULL) { for (i = 0; i < entry->localrel->trigdesc->numtriggers; i++) { Trigger *trig = entry->localrel->trigdesc->triggers + i; if (trig->tgenabled != TRIGGER_FIRES_ALWAYS && trig->tgenabled != TRIGGER_FIRES_ON_REPLICA) continue; if (func_volatile(trig->tgfoid) != PROVOLATILE_IMMUTABLE) goto parallel_apply_unsafe; } } /* Check the columns. */ tupdesc = RelationGetDescr(entry->localrel); for (attnum = 0; attnum < tupdesc->natts; attnum++) { Form_pg_attribute att = TupleDescAttr(tupdesc, attnum); /* We don't need info for dropped or generated attributes */ if (att->attisdropped || att->attgenerated) continue; /* * We don't need to check columns that only exist on the * subscriber */ if (entry->attrmap->attnums[attnum] < 0) continue; if (att->atthasdef) { Node *defaultexpr; defaultexpr = build_column_default(entry->localrel, attnum + 1); if (contain_mutable_functions(defaultexpr)) goto parallel_apply_unsafe; } /* * If the column is of a DOMAIN type, determine whether * that domain has any CHECK expressions that are not * immutable. */ if (get_typtype(att->atttypid) == TYPTYPE_DOMAIN) { List *domain_constraints; ListCell *lc; domain_constraints = GetDomainConstraints(att->atttypid); foreach(lc, domain_constraints) { DomainConstraintState *con = (DomainConstraintState *) lfirst(lc); if (con->check_expr && contain_mutable_functions((Node *) con->check_expr)) goto parallel_apply_unsafe; } } } /* Check the constraints. */ if (tupdesc->constr) { ConstrCheck *check = tupdesc->constr->check; /* * Determine if there are any CHECK constraints which * contains non-immutable function. */ for (i = 0; i < tupdesc->constr->num_check; i++) { Expr *check_expr = stringToNode(check[i].ccbin); if (contain_mutable_functions((Node *) check_expr)) goto parallel_apply_unsafe; } } /* Check the foreign keys. */ fkeys = RelationGetFKeyList(entry->localrel); if (fkeys) goto parallel_apply_unsafe; parallel_apply_safe: entry->parallel_apply = PARALLEL_APPLY_SAFE; return; parallel_apply_unsafe: entry->parallel_apply = PARALLEL_APPLY_UNSAFE; return; } [text/plain] logicalrep_rel_mark_parallel_apply-without-goto.txt (4.3K, ../../CAHut+Pv9cKurDQHtk-ygYp45-8LYdE=4sMZY-8UmbeDTGgECVg@mail.gmail.com/3-logicalrep_rel_mark_parallel_apply-without-goto.txt) download | inline: /* * Check if changes on one relation can be applied by an apply background * worker and assign the 'parallel_apply' flag. * * There are two requirements for applying changes in an apply background * worker: 1) The unique column in the relation on the subscriber-side should * also be the unique column on the publisher-side; 2) There cannot be any * non-immutable functions used by the subscriber-side. * * We just mark the relation entry as 'PARALLEL_APPLY_UNSAFE' here if changes * on one relation can not be applied by an apply background worker and leave * it to apply_bgworker_relation_check() to throw the actual error if needed. */ static void logicalrep_rel_mark_parallel_apply(LogicalRepRelMapEntry *entry) { Bitmapset *ukey; int i; TupleDesc tupdesc; int attnum; List *fkeys = NIL; /* Fast path if 'parallel_apply' flag is already known. */ if (entry->parallel_apply != PARALLEL_APPLY_UNKNOWN) return; /* Initialize the flag. */ entry->parallel_apply = PARALLEL_APPLY_SAFE; /* * First, check if the unique column in the relation on the subscriber-side * is also the unique column on the publisher-side. */ ukey = RelationGetIndexAttrBitmap(entry->localrel, INDEX_ATTR_BITMAP_KEY); if (ukey) { i = -1; while ((i = bms_next_member(ukey, i)) >= 0) { attnum = AttrNumberGetAttrOffset(i + FirstLowInvalidHeapAttributeNumber); if (entry->attrmap->attnums[attnum] < 0 || !bms_is_member(entry->attrmap->attnums[attnum], entry->remoterel.attunique)) { entry->parallel_apply = PARALLEL_APPLY_UNSAFE; return; } } bms_free(ukey); } /* * Then, check if there is any non-immutable function used by the local * table. Look for functions in the following places: * a. trigger functions; * b. Column default value expressions and domain constraints; * c. Constraint expressions; * d. Foreign keys. */ /* Check the trigger functions. */ if (entry->localrel->trigdesc != NULL) { for (i = 0; i < entry->localrel->trigdesc->numtriggers; i++) { Trigger *trig = entry->localrel->trigdesc->triggers + i; if (trig->tgenabled != TRIGGER_FIRES_ALWAYS && trig->tgenabled != TRIGGER_FIRES_ON_REPLICA) continue; if (func_volatile(trig->tgfoid) != PROVOLATILE_IMMUTABLE) { entry->parallel_apply = PARALLEL_APPLY_UNSAFE; return; } } } /* Check the columns. */ tupdesc = RelationGetDescr(entry->localrel); for (attnum = 0; attnum < tupdesc->natts; attnum++) { Form_pg_attribute att = TupleDescAttr(tupdesc, attnum); /* We don't need info for dropped or generated attributes */ if (att->attisdropped || att->attgenerated) continue; /* * We don't need to check columns that only exist on the * subscriber */ if (entry->attrmap->attnums[attnum] < 0) continue; if (att->atthasdef) { Node *defaultexpr; defaultexpr = build_column_default(entry->localrel, attnum + 1); if (contain_mutable_functions(defaultexpr)) { entry->parallel_apply = PARALLEL_APPLY_UNSAFE; return; } } /* * If the column is of a DOMAIN type, determine whether * that domain has any CHECK expressions that are not * immutable. */ if (get_typtype(att->atttypid) == TYPTYPE_DOMAIN) { List *domain_constraints; ListCell *lc; domain_constraints = GetDomainConstraints(att->atttypid); foreach(lc, domain_constraints) { DomainConstraintState *con = (DomainConstraintState *) lfirst(lc); if (con->check_expr && contain_mutable_functions((Node *) con->check_expr)) { entry->parallel_apply = PARALLEL_APPLY_UNSAFE; return; } } } } /* Check the constraints. */ if (tupdesc->constr) { ConstrCheck *check = tupdesc->constr->check; /* * Determine if there are any CHECK constraints which * contains non-immutable function. */ for (i = 0; i < tupdesc->constr->num_check; i++) { Expr *check_expr = stringToNode(check[i].ccbin); if (contain_mutable_functions((Node *) check_expr)) { entry->parallel_apply = PARALLEL_APPLY_UNSAFE; return; } } } /* Check the foreign keys. */ fkeys = RelationGetFKeyList(entry->localrel); if (fkeys) entry->parallel_apply = PARALLEL_APPLY_UNSAFE; } ^ permalink raw reply [nested|flat] 66+ messages in thread
* Re: Perform streaming logical transactions by background workers and parallel apply @ 2022-07-27 04:36 Amit Kapila <[email protected]> parent: Dilip Kumar <[email protected]> 1 sibling, 1 reply; 66+ messages in thread From: Amit Kapila @ 2022-07-27 04:36 UTC (permalink / raw) To: Dilip Kumar <[email protected]>; +Cc: [email protected] <[email protected]>; Peter Smith <[email protected]>; Masahiko Sawada <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; PostgreSQL Hackers <[email protected]> On Tue, Jul 26, 2022 at 2:30 PM Dilip Kumar <[email protected]> wrote: > > On Fri, Jul 22, 2022 at 8:27 AM [email protected] > <[email protected]> wrote: > > > > On Tues, Jul 19, 2022 at 10:29 AM I wrote: > > > Attach the news patches. > > > > Not able to apply patches cleanly because the change in HEAD (366283961a). > > Therefore, I rebased the patch based on the changes in HEAD. > > > > Attach the new patches. > > + /* Check the foreign keys. */ > + fkeys = RelationGetFKeyList(entry->localrel); > + if (fkeys) > + entry->parallel_apply = PARALLEL_APPLY_UNSAFE; > > So if there is a foreign key on any of the tables which are parts of a > subscription then we do not allow changes for that subscription to be > applied in parallel? > I think the above check will just prevent the parallelism for a xact operating on the corresponding relation not the relations of the entire subscription. Your statement sounds like you are saying that it will prevent parallelism for all the other tables in the subscription which has a table with FK. > I think this is a big limitation because having > foreign key on the table is very normal right? I agree that if we > allow them then there could be failure due to out of order apply > right? > What kind of failure do you have in mind and how it can occur? The one way it can fail is if the publisher doesn't have a corresponding foreign key on the table because then the publisher could have allowed an insert into a table (insert into FK table without having the corresponding key in PK table) which may not be allowed on the subscriber. However, I don't see any check that could prevent this because for this we need to compare the FK list for a table from the publisher with the corresponding one on the subscriber. I am not really sure if due to the risk of such conflicts we should block parallelism of transactions operating on tables with FK because those conflicts can occur even without parallelism, it is just a matter of timing. But, I could be missing something due to which the above check can be useful? -- With Regards, Amit Kapila. ^ permalink raw reply [nested|flat] 66+ messages in thread
* Re: Perform streaming logical transactions by background workers and parallel apply @ 2022-07-27 05:28 Dilip Kumar <[email protected]> parent: Amit Kapila <[email protected]> 0 siblings, 1 reply; 66+ messages in thread From: Dilip Kumar @ 2022-07-27 05:28 UTC (permalink / raw) To: Amit Kapila <[email protected]>; +Cc: [email protected] <[email protected]>; Peter Smith <[email protected]>; Masahiko Sawada <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; PostgreSQL Hackers <[email protected]> On Wed, Jul 27, 2022 at 10:06 AM Amit Kapila <[email protected]> wrote: > > On Tue, Jul 26, 2022 at 2:30 PM Dilip Kumar <[email protected]> wrote: > > > > On Fri, Jul 22, 2022 at 8:27 AM [email protected] > > <[email protected]> wrote: > > > > > > On Tues, Jul 19, 2022 at 10:29 AM I wrote: > > > > Attach the news patches. > > > > > > Not able to apply patches cleanly because the change in HEAD (366283961a). > > > Therefore, I rebased the patch based on the changes in HEAD. > > > > > > Attach the new patches. > > > > + /* Check the foreign keys. */ > > + fkeys = RelationGetFKeyList(entry->localrel); > > + if (fkeys) > > + entry->parallel_apply = PARALLEL_APPLY_UNSAFE; > > > > So if there is a foreign key on any of the tables which are parts of a > > subscription then we do not allow changes for that subscription to be > > applied in parallel? > > > > I think the above check will just prevent the parallelism for a xact > operating on the corresponding relation not the relations of the > entire subscription. Your statement sounds like you are saying that it > will prevent parallelism for all the other tables in the subscription > which has a table with FK. Okay, got it. I thought we are disallowing parallelism for the entire subscription. > > I think this is a big limitation because having > > foreign key on the table is very normal right? I agree that if we > > allow them then there could be failure due to out of order apply > > right? > > > > What kind of failure do you have in mind and how it can occur? The one > way it can fail is if the publisher doesn't have a corresponding > foreign key on the table because then the publisher could have allowed > an insert into a table (insert into FK table without having the > corresponding key in PK table) which may not be allowed on the > subscriber. However, I don't see any check that could prevent this > because for this we need to compare the FK list for a table from the > publisher with the corresponding one on the subscriber. I am not > really sure if due to the risk of such conflicts we should block > parallelism of transactions operating on tables with FK because those > conflicts can occur even without parallelism, it is just a matter of > timing. But, I could be missing something due to which the above check > can be useful? Actually, my question starts with this check[1][2], from this it appears that if this relation is having a foreign key then we are marking it parallel unsafe[2] and later in [1] while the worker is applying changes for that relation and if it was marked parallel unsafe then we are throwing error. So my question was why we are putting this restriction? Although this error is only talking about unique and non-immutable functions this is also giving an error if the target table had a foreign key. So my question was do we really need to restrict this? I mean why we are restricting this case? [1] +apply_bgworker_relation_check(LogicalRepRelMapEntry *rel) +{ + /* Skip check if not an apply background worker. */ + if (!am_apply_bgworker()) + return; + + /* + * Partition table checks are done later in function + * apply_handle_tuple_routing. + */ + if (rel->localrel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + return; + + /* + * Return if changes on this relation can be applied by an apply background + * worker. + */ + if (rel->parallel_apply == PARALLEL_APPLY_SAFE) + return; + + /* We are in error mode and should give user correct error. */ + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("cannot replicate target relation \"%s.%s\" using " + "subscription parameter streaming=parallel", + rel->remoterel.nspname, rel->remoterel.relname), + errdetail("The unique column on subscriber is not the unique " + "column on publisher or there is at least one " + "non-immutable function."), + errhint("Please change to use subscription parameter " + "streaming=on."))); +} [2] > > + /* Check the foreign keys. */ > > + fkeys = RelationGetFKeyList(entry->localrel); > > + if (fkeys) > > + entry->parallel_apply = PARALLEL_APPLY_UNSAFE; -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 66+ messages in thread
* RE: Perform streaming logical transactions by background workers and parallel apply @ 2022-07-27 07:57 [email protected] <[email protected]> parent: Dilip Kumar <[email protected]> 0 siblings, 1 reply; 66+ messages in thread From: [email protected] @ 2022-07-27 07:57 UTC (permalink / raw) To: Dilip Kumar <[email protected]>; Amit Kapila <[email protected]>; +Cc: [email protected] <[email protected]>; Peter Smith <[email protected]>; Masahiko Sawada <[email protected]>; [email protected] <[email protected]>; PostgreSQL Hackers <[email protected]> On Wednesday, July 27, 2022 1:29 PM Dilip Kumar <[email protected]> wrote: > > On Wed, Jul 27, 2022 at 10:06 AM Amit Kapila <[email protected]> > wrote: > > > > On Tue, Jul 26, 2022 at 2:30 PM Dilip Kumar <[email protected]> wrote: > > > > > > On Fri, Jul 22, 2022 at 8:27 AM [email protected] > > > <[email protected]> wrote: > > > > > > > > On Tues, Jul 19, 2022 at 10:29 AM I wrote: > > > > > Attach the news patches. > > > > > > > > Not able to apply patches cleanly because the change in HEAD > (366283961a). > > > > Therefore, I rebased the patch based on the changes in HEAD. > > > > > > > > Attach the new patches. > > > > > > + /* Check the foreign keys. */ > > > + fkeys = RelationGetFKeyList(entry->localrel); > > > + if (fkeys) > > > + entry->parallel_apply = PARALLEL_APPLY_UNSAFE; > > > > > > So if there is a foreign key on any of the tables which are parts of > > > a subscription then we do not allow changes for that subscription to > > > be applied in parallel? > > > > > > > I think the above check will just prevent the parallelism for a xact > > operating on the corresponding relation not the relations of the > > entire subscription. Your statement sounds like you are saying that it > > will prevent parallelism for all the other tables in the subscription > > which has a table with FK. > > Okay, got it. I thought we are disallowing parallelism for the entire subscription. > > > > I think this is a big limitation because having foreign key on the > > > table is very normal right? I agree that if we allow them then > > > there could be failure due to out of order apply right? > > > > > > > What kind of failure do you have in mind and how it can occur? The one > > way it can fail is if the publisher doesn't have a corresponding > > foreign key on the table because then the publisher could have allowed > > an insert into a table (insert into FK table without having the > > corresponding key in PK table) which may not be allowed on the > > subscriber. However, I don't see any check that could prevent this > > because for this we need to compare the FK list for a table from the > > publisher with the corresponding one on the subscriber. I am not > > really sure if due to the risk of such conflicts we should block > > parallelism of transactions operating on tables with FK because those > > conflicts can occur even without parallelism, it is just a matter of > > timing. But, I could be missing something due to which the above check > > can be useful? > > Actually, my question starts with this check[1][2], from this it > appears that if this relation is having a foreign key then we are > marking it parallel unsafe[2] and later in [1] while the worker is > applying changes for that relation and if it was marked parallel > unsafe then we are throwing error. So my question was why we are > putting this restriction? Although this error is only talking about > unique and non-immutable functions this is also giving an error if the > target table had a foreign key. So my question was do we really need > to restrict this? I mean why we are restricting this case? > Hi, I think the foreign key check is used to prevent the apply worker from waiting indefinitely which is caused by foreign key difference between publisher and subscriber, Like the following example: ------------------------------------- Publisher: -- both table are published CREATE TABLE PKTABLE ( ptest1 int); CREATE TABLE FKTABLE ( ftest1 int); -- initial data INSERT INTO PKTABLE VALUES(1); Subcriber: CREATE TABLE PKTABLE ( ptest1 int PRIMARY KEY); CREATE TABLE FKTABLE ( ftest1 int REFERENCES PKTABLE); -- Execute the following transactions on publisher Tx1: INSERT ... -- make enough changes to start streaming mode DELETE FROM PKTABLE; Tx2: INSERT ITNO FKTABLE VALUES(1); COMMIT; COMMIT; ------------------------------------- The subcriber's apply worker will wait indefinitely, because the main apply worker is waiting for the streaming transaction to finish which is in another apply bgworker. BTW, I think the foreign key won't take effect in subscriber's apply worker by default. Because we set session_replication_role to 'replica' in apply worker which prevent the FK trigger function to be executed(only the trigger with FIRES_ON_REPLICA flag will be executed in this mode). User can only alter the trigger to enable it on replica mode to make the foreign key work. So, ISTM, we won't hit this ERROR frequently. And based on this, another comment about the patch is that it seems unnecessary to directly check the FK returned by RelationGetFKeyList. Checking the actual FK trigger function seems enough. Best regards, Hou zj ^ permalink raw reply [nested|flat] 66+ messages in thread
* Re: Perform streaming logical transactions by background workers and parallel apply @ 2022-07-27 08:03 Peter Smith <[email protected]> parent: [email protected] <[email protected]> 5 siblings, 0 replies; 66+ messages in thread From: Peter Smith @ 2022-07-27 08:03 UTC (permalink / raw) To: [email protected] <[email protected]>; +Cc: Amit Kapila <[email protected]>; Masahiko Sawada <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; PostgreSQL Hackers <[email protected]> Here are some review comments for the patch v19-0004: ====== 1. doc/src/sgml/ref/create_subscription.sgml @@ -244,6 +244,11 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl column in the relation on the subscriber-side should also be the unique column on the publisher-side; 2) there cannot be any non-immutable functions used by the subscriber-side replicated table. + When applying a streaming transaction, if either requirement is not + met, the background worker will exit with an error. + <literal>parallel</literal> mode is disregarded when retrying; + instead the transaction will be applied using <literal>on</literal> + mode. </para> That last sentence starting with lowercase seems odd - that's why I thought saying "The parallel mode..." might be better. IMO "on mode" seems strange too. Hence my previous [1] (#4.3) suggestion for this SUGGESTION The <literal>parallel</literal> mode is disregarded when retrying; instead the transaction will be applied using <literal>streaming = on</literal>. ====== 2. src/backend/replication/logical/worker.c - start_table_sync @@ -3902,20 +3925,28 @@ start_table_sync(XLogRecPtr *origin_startpos, char **myslotname) } PG_CATCH(); { + /* + * Emit the error message, and recover from the error state to an idle + * state + */ + HOLD_INTERRUPTS(); + + EmitErrorReport(); + AbortOutOfAnyTransaction(); + FlushErrorState(); + + RESUME_INTERRUPTS(); + + /* Report the worker failed during table synchronization */ + pgstat_report_subscription_error(MySubscription->oid, false); + + /* Set the retry flag. */ + set_subscription_retry(true); + if (MySubscription->disableonerr) DisableSubscriptionAndExit(); - else - { - /* - * Report the worker failed during table synchronization. Abort - * the current transaction so that the stats message is sent in an - * idle state. - */ - AbortOutOfAnyTransaction(); - pgstat_report_subscription_error(MySubscription->oid, false); - PG_RE_THROW(); - } + proc_exit(0); } But is it correct to set the 'retry' flag even if the MySubscription->disableonerr is true? Won’t that mean even after the user corrects the problem and then re-enabled the subscription it still won't let the streaming=parallel work, because that retry flag is set? Also, Something seems wrong to me here - IIUC the patch changed this code because of the potential risk of an error within the set_subscription_retry function, but now if such an error happens the current code will bypass even getting to DisableSubscriptionAndExit, so the subscription won't have a chance to get disabled as the user might have wanted. ~~~ 3. src/backend/replication/logical/worker.c - start_apply @@ -3940,20 +3971,27 @@ start_apply(XLogRecPtr origin_startpos) } PG_CATCH(); { + /* + * Emit the error message, and recover from the error state to an idle + * state + */ + HOLD_INTERRUPTS(); + + EmitErrorReport(); + AbortOutOfAnyTransaction(); + FlushErrorState(); + + RESUME_INTERRUPTS(); + + /* Report the worker failed while applying changes */ + pgstat_report_subscription_error(MySubscription->oid, + !am_tablesync_worker()); + + /* Set the retry flag. */ + set_subscription_retry(true); + if (MySubscription->disableonerr) DisableSubscriptionAndExit(); - else - { - /* - * Report the worker failed while applying changes. Abort the - * current transaction so that the stats message is sent in an - * idle state. - */ - AbortOutOfAnyTransaction(); - pgstat_report_subscription_error(MySubscription->oid, !am_tablesync_worker()); - - PG_RE_THROW(); - } } (Same as previous review comment #2) But is it correct to set the 'retry' flag even if the MySubscription->disableonerr is true? Won’t that mean even after the user corrects the problem and then re-enabled the subscription it still won't let the streaming=parallel work, because that retry flag is set? Also, Something seems wrong to me here - IIUC the patch changed this code because of the potential risk of an error within the set_subscription_retry function, but now if such an error happens the current code will bypass even getting to DisableSubscriptionAndExit, so the subscription won't have a chance to get disabled as the user might have wanted. ~~~ 4. src/backend/replication/logical/worker.c - DisableSubscriptionAndExit /* - * After error recovery, disable the subscription in a new transaction - * and exit cleanly. + * Disable the subscription in a new transaction. */ static void DisableSubscriptionAndExit(void) { - /* - * Emit the error message, and recover from the error state to an idle - * state - */ - HOLD_INTERRUPTS(); - - EmitErrorReport(); - AbortOutOfAnyTransaction(); - FlushErrorState(); - - RESUME_INTERRUPTS(); - - /* Report the worker failed during either table synchronization or apply */ - pgstat_report_subscription_error(MyLogicalRepWorker->subid, - !am_tablesync_worker()); - /* Disable the subscription */ StartTransactionCommand(); DisableSubscription(MySubscription->oid); @@ -4231,8 +4252,6 @@ DisableSubscriptionAndExit(void) ereport(LOG, errmsg("logical replication subscription \"%s\" has been disabled due to an error", MySubscription->name)); - - proc_exit(0); } 4a. Hmm, I think it is a bad idea to remove the "exiting" code from the function but still leave the function name the same as before saying "AndExit". 4b. Also, now the patch is unconditionally doing proc_exit(0) in the calling code where previously it would do PG_RE_THROW. So it's a subtle difference from the path the code used to take for worker errors.. ~~~ 5. src/backend/replication/logical/worker.c - set_subscription_retry @@ -4467,3 +4486,63 @@ reset_apply_error_context_info(void) apply_error_callback_arg.remote_attnum = -1; set_apply_error_context_xact(InvalidTransactionId, InvalidXLogRecPtr); } + +/* + * Set subretry of pg_subscription catalog. + * + * If retry is true, subscriber is about to exit with an error. Otherwise, it + * means that the transaction was applied successfully. + */ +static void +set_subscription_retry(bool retry) +{ + Relation rel; + HeapTuple tup; + bool started_tx = false; + bool nulls[Natts_pg_subscription]; + bool replaces[Natts_pg_subscription]; + Datum values[Natts_pg_subscription]; + + if (MySubscription->retry == retry || + am_apply_bgworker()) + return; Currently, I think this new 'subretry' field is only used to decide whether a retry can use an apply background worker or not. I think all this logic is *only* used when streaming=parallel. But AFAICT the logic for setting/clearing the retry flag is executed *always* regardless of the streaming mode. So for all the times when the user did not ask for streaming=parallel doesn't this just cause unnecessary overhead for every transaction? ------ [1] https://www.postgresql.org/message-id/OS3PR01MB62758A6AAED27B3A848CEB7A9E8F9%40OS3PR01MB6275.jpnprd0... Kind Regards, Peter Smith. Fujitsu Australia ^ permalink raw reply [nested|flat] 66+ messages in thread
* RE: Perform streaming logical transactions by background workers and parallel apply @ 2022-07-27 08:21 [email protected] <[email protected]> parent: Dilip Kumar <[email protected]> 0 siblings, 0 replies; 66+ messages in thread From: [email protected] @ 2022-07-27 08:21 UTC (permalink / raw) To: Dilip Kumar <[email protected]>; [email protected] <[email protected]>; +Cc: Peter Smith <[email protected]>; Amit Kapila <[email protected]>; Masahiko Sawada <[email protected]>; [email protected] <[email protected]>; PostgreSQL Hackers <[email protected]> On Tuesday, July 26, 2022 5:34 PM Dilip Kumar <[email protected]> wrote: > On Tue, Jul 26, 2022 at 2:30 PM Dilip Kumar <[email protected]> wrote: > > > > On Fri, Jul 22, 2022 at 8:27 AM [email protected] > > <[email protected]> wrote: > > > > > > On Tues, Jul 19, 2022 at 10:29 AM I wrote: > > > > Attach the news patches. > > > > > > Not able to apply patches cleanly because the change in HEAD > (366283961a). > > > Therefore, I rebased the patch based on the changes in HEAD. > > > > > > Attach the new patches. > > > > + /* Check the foreign keys. */ > > + fkeys = RelationGetFKeyList(entry->localrel); > > + if (fkeys) > > + entry->parallel_apply = PARALLEL_APPLY_UNSAFE; > > > > So if there is a foreign key on any of the tables which are parts of a > > subscription then we do not allow changes for that subscription to be > > applied in parallel? I think this is a big limitation because having > > foreign key on the table is very normal right? I agree that if we > > allow them then there could be failure due to out of order apply > > right? but IMHO we should not put the restriction instead let it fail > > if there is ever such conflict. Because if there is a conflict the > > transaction will be sent again. Do we see that there could be wrong > > or inconsistent results if we allow such things to be executed in > > parallel. If not then IMHO just to avoid some corner case failure we > > are restricting very normal cases. > > some more comments.. > 1. > + /* > + * If we have found a free worker or if we are already > applying this > + * transaction in an apply background worker, then we > pass the data to > + * that worker. > + */ > + if (first_segment) > + apply_bgworker_send_data(stream_apply_worker, s->len, > + s->data); > > Comment says that if we have found a free worker or we are already applying in > the worker then pass the changes to the worker but actually as per the code > here we are only passing in case of first_segment? > > I think what you are trying to say is that if it is first segment then send the > > 2. > + /* > + * This is the main apply worker. Check if there is any free apply > + * background worker we can use to process this transaction. > + */ > + if (first_segment) > + stream_apply_worker = apply_bgworker_start(stream_xid); > + else > + stream_apply_worker = apply_bgworker_find(stream_xid); > > So currently, whenever we get a new streamed transaction we try to start a new > background worker for that. Why do we need to start/close the background > apply worker every time we get a new streamed transaction. I mean we can > keep the worker in the pool for time being and if there is a new transaction > looking for a worker then we can find from that. Starting a worker is costly > operation and since we are using parallelism for this mean we are expecting > that there would be frequent streamed transaction needing parallel apply > worker so why not to let it wait for a certain amount of time so that if load is low > it will anyway stop and if the load is high it will be reused for next streamed > transaction. It seems the function name was a bit mislead. Currently, the started apply bgworker won't exit after applying the transaction. And the apply_bgworker_start will first try to choose a free worker. It will start a new worker only if no free worker is available. > 3. > Why are we restricting parallel apply workers only for the streamed > transactions, because streaming depends upon the size of the logical decoding > work mem so making steaming and parallel apply tightly coupled seems too > restrictive to me. Do we see some obvious problems in applying other > transactions in parallel? We thought there could be some conflict failure and deadlock if we parallel apply normal transaction which need transaction dependency check[1]. But I will do some more research for this and share the result soon. [1] https://www.postgresql.org/message-id/CAA4eK1%2BwyN6zpaHUkCLorEWNx75MG0xhMwcFhvjqm2KURZEAGw%40mail.g... Best regards, Hou zj ^ permalink raw reply [nested|flat] 66+ messages in thread
* RE: Perform streaming logical transactions by background workers and parallel apply @ 2022-07-27 12:41 [email protected] <[email protected]> parent: [email protected] <[email protected]> 5 siblings, 1 reply; 66+ messages in thread From: [email protected] @ 2022-07-27 12:41 UTC (permalink / raw) To: [email protected] <[email protected]>; +Cc: Amit Kapila <[email protected]>; Masahiko Sawada <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; PostgreSQL Hackers <[email protected]>; Peter Smith <[email protected]> Dear Wang-san, Hi, I'm also interested in the patch and I started to review this. Followings are comments about 0001. 1. terminology In your patch a new worker "apply background worker" has been introduced, but I thought it might be confused because PostgreSQL has already the worker "background worker". Both of apply worker and apply bworker are categolized as bgworker. Do you have any reasons not to use "apply parallel worker" or "apply streaming worker"? (Note that I'm not native English speaker) 2. logicalrep_worker_stop() ``` - /* No worker, nothing to do. */ - if (!worker) - { - LWLockRelease(LogicalRepWorkerLock); - return; - } + if (worker) + logicalrep_worker_stop_internal(worker); + + LWLockRelease(LogicalRepWorkerLock); +} ``` I thought you could add a comment the meaning of if-statement, like "No main apply worker, nothing to do" 3. logicalrep_workers_find() I thought you could add a description about difference between this and logicalrep_worker_find() at the top of the function. IIUC logicalrep_workers_find() counts subworker, but logicalrep_worker_find() does not focus such type of workers. 4. logicalrep_worker_detach() ``` static void logicalrep_worker_detach(void) { + /* + * If we are the main apply worker, stop all the apply background workers + * we started before. + * ``` I thought "we are" should be "This is", based on other comments. 5. applybgworker.c ``` +/* Apply background workers hash table (initialized on first use) */ +static HTAB *ApplyWorkersHash = NULL; +static List *ApplyWorkersFreeList = NIL; +static List *ApplyWorkersList = NIL; ``` I thought they should be ApplyBgWorkersXXX, because they stores information only related with apply bgworkers. 6. ApplyBgworkerShared ``` + TransactionId stream_xid; + uint32 n; /* id of apply background worker */ +} ApplyBgworkerShared; ``` I thought the field "n" is too general, how about "proc_id" or "worker_id"? 7. apply_bgworker_wait_for() ``` + /* If any workers (or the postmaster) have died, we have failed. */ + if (status == APPLY_BGWORKER_EXIT) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("background worker %u failed to apply transaction %u", + wstate->shared->n, wstate->shared->stream_xid))) ``` 7.a I thought we should not mention about PM death here, because in this case apply worker will exit at WaitLatch(). 7.b The error message should be "apply background worker %u...". 8. apply_bgworker_check_status() ``` + errmsg("background worker %u exited unexpectedly", + wstate->shared->n))); ``` The error message should be "apply background worker %u...". 9. apply_bgworker_send_data() ``` + if (result != SHM_MQ_SUCCESS) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("could not send tuples to shared-memory queue"))); ``` I thought the error message should be "could not send data to..." because sent data might not be tuples. For example, in case of STEAM PREPARE, I thit does not contain tuple. 10. wait_event.h ``` WAIT_EVENT_HASH_GROW_BUCKETS_REINSERT, + WAIT_EVENT_LOGICAL_APPLY_WORKER_STATE_CHANGE, WAIT_EVENT_LOGICAL_SYNC_DATA, ``` I thought the event should be WAIT_EVENT_LOGICAL_APPLY_BG_WORKER_STATE_CHANGE, because this is used when apply worker waits until the status of bgworker changes. Best Regards, Hayato Kuroda FUJITSU LIMITED ^ permalink raw reply [nested|flat] 66+ messages in thread
* RE: Perform streaming logical transactions by background workers and parallel apply @ 2022-07-28 05:20 [email protected] <[email protected]> parent: [email protected] <[email protected]> 0 siblings, 0 replies; 66+ messages in thread From: [email protected] @ 2022-07-28 05:20 UTC (permalink / raw) To: [email protected] <[email protected]>; +Cc: Amit Kapila <[email protected]>; Masahiko Sawada <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; PostgreSQL Hackers <[email protected]>; Peter Smith <[email protected]> Dear Wang, I found further comments about the test code. 11. src/test/regress/sql/subscription.sql ``` -- fail - streaming must be boolean CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = foo); ``` The comment is no longer correct: should be "streaming must be boolean or 'parallel'" 12. src/test/regress/sql/subscription.sql ``` -- now it works CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = true); ``` I think we should test the case of streaming = 'parallel'. 13. 015_stream.pl I could not find test about TRUNCATE. IIUC apply bgworker works well even if it gets LOGICAL_REP_MSG_TRUNCATE message from main worker. Can you add the case? Best Regards, Hayato Kuroda FUJITSU LIMITED ^ permalink raw reply [nested|flat] 66+ messages in thread
* Re: Perform streaming logical transactions by background workers and parallel apply @ 2022-07-28 13:32 Amit Kapila <[email protected]> parent: [email protected] <[email protected]> 0 siblings, 0 replies; 66+ messages in thread From: Amit Kapila @ 2022-07-28 13:32 UTC (permalink / raw) To: [email protected] <[email protected]>; +Cc: Dilip Kumar <[email protected]>; [email protected] <[email protected]>; Peter Smith <[email protected]>; Masahiko Sawada <[email protected]>; [email protected] <[email protected]>; PostgreSQL Hackers <[email protected]> On Wed, Jul 27, 2022 at 1:27 PM [email protected] <[email protected]> wrote: > > On Wednesday, July 27, 2022 1:29 PM Dilip Kumar <[email protected]> wrote: > > > > On Wed, Jul 27, 2022 at 10:06 AM Amit Kapila <[email protected]> > > > > > > What kind of failure do you have in mind and how it can occur? The one > > > way it can fail is if the publisher doesn't have a corresponding > > > foreign key on the table because then the publisher could have allowed > > > an insert into a table (insert into FK table without having the > > > corresponding key in PK table) which may not be allowed on the > > > subscriber. However, I don't see any check that could prevent this > > > because for this we need to compare the FK list for a table from the > > > publisher with the corresponding one on the subscriber. I am not > > > really sure if due to the risk of such conflicts we should block > > > parallelism of transactions operating on tables with FK because those > > > conflicts can occur even without parallelism, it is just a matter of > > > timing. But, I could be missing something due to which the above check > > > can be useful? > > > > Actually, my question starts with this check[1][2], from this it > > appears that if this relation is having a foreign key then we are > > marking it parallel unsafe[2] and later in [1] while the worker is > > applying changes for that relation and if it was marked parallel > > unsafe then we are throwing error. So my question was why we are > > putting this restriction? Although this error is only talking about > > unique and non-immutable functions this is also giving an error if the > > target table had a foreign key. So my question was do we really need > > to restrict this? I mean why we are restricting this case? > > > > Hi, > > I think the foreign key check is used to prevent the apply worker from waiting > indefinitely which is caused by foreign key difference between publisher and > subscriber, Like the following example: > > ------------------------------------- > Publisher: > -- both table are published > CREATE TABLE PKTABLE ( ptest1 int); > CREATE TABLE FKTABLE ( ftest1 int); > > -- initial data > INSERT INTO PKTABLE VALUES(1); > > Subcriber: > CREATE TABLE PKTABLE ( ptest1 int PRIMARY KEY); > CREATE TABLE FKTABLE ( ftest1 int REFERENCES PKTABLE); > > -- Execute the following transactions on publisher > > Tx1: > INSERT ... -- make enough changes to start streaming mode > DELETE FROM PKTABLE; > Tx2: > INSERT ITNO FKTABLE VALUES(1); > COMMIT; > COMMIT; > ------------------------------------- > > The subcriber's apply worker will wait indefinitely, because the main apply worker is > waiting for the streaming transaction to finish which is in another apply > bgworker. > IIUC, here the problem will be that TX2 (Insert in FK table) performed by the apply worker will wait for a parallel worker doing streaming transaction TX1 which has performed Delete from PK table. This wait is required because we can't decide if Insert will be successful or not till TX1 is either committed or Rollback. This is similar to the problem related to primary/unique keys mentioned earlier [1]. If so, then, we should try to forbid this in some way to avoid subscribers from being stuck. Dilip, does this reason sounds sufficient to you for such a check, or do you still think we don't need any check for FK's? > > BTW, I think the foreign key won't take effect in subscriber's apply worker by > default. Because we set session_replication_role to 'replica' in apply worker > which prevent the FK trigger function to be executed(only the trigger with > FIRES_ON_REPLICA flag will be executed in this mode). User can only alter the > trigger to enable it on replica mode to make the foreign key work. So, ISTM, we > won't hit this ERROR frequently. > > And based on this, another comment about the patch is that it seems unnecessary > to directly check the FK returned by RelationGetFKeyList. Checking the actual FK > trigger function seems enough. > That is correct. I think it would have been better if we can detect that publisher doesn't have FK but the subscriber has FK as it can occur only in that scenario. If that requires us to send more information from the publisher, we can leave it for now (as this doesn't seem to be a frequent scenario) and keep a simpler check based on subscriber schema. I think we should add a test as mentioned by you above so that if tomorrow one tries to remove the FK check, we have a way to know. Also, please add comments and tests for additional checks related to constraints in the patch. [1] - https://www.postgresql.org/message-id/CAA4eK1JwahU_WuP3S%2B7POqta%3DPhm_3gxZeVmJuuoUq1NV%3DkrXA%40ma... -- With Regards, Amit Kapila. ^ permalink raw reply [nested|flat] 66+ messages in thread
end of thread, other threads:[~2022-07-28 13:32 UTC | newest] Thread overview: 66+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2020-08-27 05:49 [PATCH v2] Fix NaN handling of some geometric operators and functions Kyotaro Horiguchi <[email protected]> 2022-04-19 06:57 Re: Perform streaming logical transactions by background workers and parallel apply Amit Kapila <[email protected]> 2022-04-20 08:57 ` RE: Perform streaming logical transactions by background workers and parallel apply [email protected] <[email protected]> 2022-04-20 12:22 ` RE: Perform streaming logical transactions by background workers and parallel apply [email protected] <[email protected]> 2022-04-22 04:12 ` Re: Perform streaming logical transactions by background workers and parallel apply Peter Smith <[email protected]> 2022-04-25 08:35 ` RE: Perform streaming logical transactions by background workers and parallel apply [email protected] <[email protected]> 2022-04-29 02:06 ` RE: Perform streaming logical transactions by background workers and parallel apply [email protected] <[email protected]> 2022-04-29 05:22 ` RE: Perform streaming logical transactions by background workers and parallel apply [email protected] <[email protected]> 2022-05-06 08:56 ` Re: Perform streaming logical transactions by background workers and parallel apply Peter Smith <[email protected]> 2022-05-13 09:57 ` RE: Perform streaming logical transactions by background workers and parallel apply [email protected] <[email protected]> 2022-05-18 07:11 ` Re: Perform streaming logical transactions by background workers and parallel apply Peter Smith <[email protected]> 2022-05-30 08:51 ` RE: Perform streaming logical transactions by background workers and parallel apply [email protected] <[email protected]> 2022-05-30 11:38 ` Re: Perform streaming logical transactions by background workers and parallel apply Amit Kapila <[email protected]> 2022-05-31 08:52 ` Re: Perform streaming logical transactions by background workers and parallel apply Amit Kapila <[email protected]> 2022-06-01 02:00 ` Re: Perform streaming logical transactions by background workers and parallel apply Masahiko Sawada <[email protected]> 2022-06-01 05:19 ` Re: Perform streaming logical transactions by background workers and parallel apply Amit Kapila <[email protected]> 2022-06-02 10:01 ` RE: Perform streaming logical transactions by background workers and parallel apply [email protected] <[email protected]> 2022-06-08 07:12 ` RE: Perform streaming logical transactions by background workers and parallel apply [email protected] <[email protected]> 2022-06-14 03:37 ` RE: Perform streaming logical transactions by background workers and parallel apply [email protected] <[email protected]> 2022-06-15 08:26 ` RE: Perform streaming logical transactions by background workers and parallel apply [email protected] <[email protected]> 2022-06-21 01:41 ` Re: Perform streaming logical transactions by background workers and parallel apply Peter Smith <[email protected]> 2022-06-21 04:24 ` Re: Perform streaming logical transactions by background workers and parallel apply Amit Kapila <[email protected]> 2022-06-28 03:23 ` RE: Perform streaming logical transactions by background workers and parallel apply [email protected] <[email protected]> 2022-06-15 12:12 ` Re: Perform streaming logical transactions by background workers and parallel apply Amit Kapila <[email protected]> 2022-06-17 07:17 ` RE: Perform streaming logical transactions by background workers and parallel apply [email protected] <[email protected]> 2022-06-17 08:57 ` Re: Perform streaming logical transactions by background workers and parallel apply Amit Kapila <[email protected]> 2022-06-20 02:59 ` Re: Perform streaming logical transactions by background workers and parallel apply Amit Kapila <[email protected]> 2022-06-23 07:21 ` RE: Perform streaming logical transactions by background workers and parallel apply [email protected] <[email protected]> 2022-06-23 08:43 ` Re: Perform streaming logical transactions by background workers and parallel apply Amit Kapila <[email protected]> 2022-06-28 03:21 ` RE: Perform streaming logical transactions by background workers and parallel apply [email protected] <[email protected]> 2022-06-28 04:15 ` Re: Perform streaming logical transactions by background workers and parallel apply Amit Kapila <[email protected]> 2022-06-28 07:20 ` RE: Perform streaming logical transactions by background workers and parallel apply [email protected] <[email protected]> 2022-07-01 06:43 ` Re: Perform streaming logical transactions by background workers and parallel apply Peter Smith <[email protected]> 2022-07-01 09:43 ` Re: Perform streaming logical transactions by background workers and parallel apply Amit Kapila <[email protected]> 2022-07-07 03:45 ` RE: Perform streaming logical transactions by background workers and parallel apply [email protected] <[email protected]> 2022-07-07 03:44 ` RE: Perform streaming logical transactions by background workers and parallel apply [email protected] <[email protected]> 2022-07-07 10:20 ` RE: Perform streaming logical transactions by background workers and parallel apply [email protected] <[email protected]> 2022-07-13 04:33 ` Re: Perform streaming logical transactions by background workers and parallel apply Peter Smith <[email protected]> 2022-07-19 02:28 ` RE: Perform streaming logical transactions by background workers and parallel apply [email protected] <[email protected]> 2022-07-22 02:56 ` RE: Perform streaming logical transactions by background workers and parallel apply [email protected] <[email protected]> 2022-07-25 13:50 ` Re: Perform streaming logical transactions by background workers and parallel apply Amit Kapila <[email protected]> 2022-07-26 09:00 ` Re: Perform streaming logical transactions by background workers and parallel apply Dilip Kumar <[email protected]> 2022-07-26 09:33 ` Re: Perform streaming logical transactions by background workers and parallel apply Dilip Kumar <[email protected]> 2022-07-27 08:21 ` RE: Perform streaming logical transactions by background workers and parallel apply [email protected] <[email protected]> 2022-07-27 04:36 ` Re: Perform streaming logical transactions by background workers and parallel apply Amit Kapila <[email protected]> 2022-07-27 05:28 ` Re: Perform streaming logical transactions by background workers and parallel apply Dilip Kumar <[email protected]> 2022-07-27 07:57 ` RE: Perform streaming logical transactions by background workers and parallel apply [email protected] <[email protected]> 2022-07-28 13:32 ` Re: Perform streaming logical transactions by background workers and parallel apply Amit Kapila <[email protected]> 2022-07-26 09:56 ` Re: Perform streaming logical transactions by background workers and parallel apply Peter Smith <[email protected]> 2022-07-27 03:37 ` Re: Perform streaming logical transactions by background workers and parallel apply Peter Smith <[email protected]> 2022-07-27 08:03 ` Re: Perform streaming logical transactions by background workers and parallel apply Peter Smith <[email protected]> 2022-07-27 12:41 ` RE: Perform streaming logical transactions by background workers and parallel apply [email protected] <[email protected]> 2022-07-28 05:20 ` RE: Perform streaming logical transactions by background workers and parallel apply [email protected] <[email protected]> 2022-07-04 04:11 ` Re: Perform streaming logical transactions by background workers and parallel apply Peter Smith <[email protected]> 2022-07-07 03:46 ` RE: Perform streaming logical transactions by background workers and parallel apply [email protected] <[email protected]> 2022-07-04 06:47 ` Re: Perform streaming logical transactions by background workers and parallel apply Peter Smith <[email protected]> 2022-07-07 03:47 ` RE: Perform streaming logical transactions by background workers and parallel apply [email protected] <[email protected]> 2022-07-07 03:31 ` RE: Perform streaming logical transactions by background workers and parallel apply [email protected] <[email protected]> 2022-07-13 05:48 ` RE: Perform streaming logical transactions by background workers and parallel apply [email protected] <[email protected]> 2022-06-23 02:47 ` Re: Perform streaming logical transactions by background workers and parallel apply Peter Smith <[email protected]> 2022-06-23 06:50 ` Re: Perform streaming logical transactions by background workers and parallel apply Peter Smith <[email protected]> 2022-06-28 03:23 ` RE: Perform streaming logical transactions by background workers and parallel apply [email protected] <[email protected]> 2022-06-02 10:03 ` RE: Perform streaming logical transactions by background workers and parallel apply [email protected] <[email protected]> 2022-05-19 06:22 ` Re: Perform streaming logical transactions by background workers and parallel apply Peter Smith <[email protected]> 2022-05-05 05:45 ` Re: Perform streaming logical transactions by background workers and parallel apply Peter Smith <[email protected]> 2022-05-13 08:48 ` RE: Perform streaming logical transactions by background workers and parallel apply [email protected] <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox